Query: See this method to bring together concept bottleneck models and LLMs to produce interpretable hypotheses that distinguish text distributions.

\section{Motivation}

Assume we have $N$ text instances $x_1, \cdots, x_N$ from a text distribution $\mathcal{X}$ with corresponding labels $y_1, \cdots, y_N$ from a label distribution $\mathcal{Y}$. 
We want to understand which features of $x_i$ affect $y_i$. 
For examples:
\begin{enumerate}
    \item $\mathcal{X}$ consists of multiple choice medical exam questions used to benchmark LLMs, and $\mathcal{Y} = \{0, 1\}$ denotes whether the LLM answered the question correctly or not. 
    We want to understand what features of medical questions affect whether the LLM will answer them correctly or not.
    
    \item $\mathcal{X}$ consists of ICLR rebuttal texts, and $\mathcal{Y}$ consists of integer values corresponding to how much a reviewer raised their score.
    We want to understand what features of rebuttals explain whether a reviewer raised their score. 
    
    \item $\mathcal{X}$ consists of ER discharge summaries written by doctors, and $\mathcal{Y}$ consists of doctor fatigue scores (e.g.~based on how much they worked in the past week). 
    We want to come up with hypotheses for how clinical notes change when doctors are more fatigued \citep{hsu_clinical_2023}.
    This type of task might also necessitate controls for things like the type of case, since doctors may be disproportionately assigned to certain types of cases when they are more/less fatigued. 
    So we'd want to know which features of notes explain doctor fatigue with controls (and their coefficients) for case type, and perhaps also other known features like word variety (lower word variety correlates with higher fatigue).
\end{enumerate}

\section{Method}
\label{sec:method}

We focus on a class of methods which contain four main steps. 
\begin{enumerate}
    \item \textbf{Prediction}: We use a standard NLP model to predict the label from text embeddings. 
    Text embeddings are quite expressive, so they can learn many functions, but they are not easily interpretable.
    On the other hand, if this model already fails to achieve substantial signal at predicting the label, it's unlikely that the the following steps will work.
    
    The twist in this model architecture is that instead of a standard embeddings $\rightarrow$ label architecture, our architecture is: embeddings $\rightarrow$ intermediate layer with a small number of neurons $\rightarrow$ label.
    We'll use the intermediate layer to interpret the model.
    
    \item \textbf{Hypothesis Proposal}: Given a predictive model, we want to draw from what the model has learned to propose explanatory hypotheses for our dataset.
    We interpret one neuron at a time in the intermediate layer.
    For a single neuron, we will compute its activation value across lots of text samples (e.g., $n = 20-100$), and then prompt a language model with a sorted list of texts and their activations and ask it to propose a hypothesis that explains what features lead to higher scores. 
    Hypotheses are things like ``the text contains spelling mistakes''.

    \item \textbf{Hypothesis Validation}: We validate hypotheses by using a separately-prompted \textit{annotator LLM} to apply the hypothesis to a bunch of text samples.
    First, we verify that the hypothesis annotations actually correlate with the concept neuron activations (otherwise, the hypothesis is just a bad representation of when the neuron fires).
    This winnows the list significantly.
    Second, we validate that the hypothesis actually correlates with the output label on a larger set of data. 
    We keep a running list of hypotheses ranked by their correlation with the output label.
    
    Hypothesis Validation is the most expensive step, because we need to compute hundreds of annotations for every hypothesis that is proposed, and many of the proposed hypotheses are bad (so improving the precision of the proposal step is high-value).

    \item \textbf{Iteration}: One round of the above steps usually does not produce good hypotheses. 
    So we follow a process similar to boosting, where we fix the values of the annotations provided by hypotheses discovered in prior iterations, and then re-iterate the above steps.
    This forces the model to predict the residual signal in the labels on top of the existing hypotheses. 
    
\end{enumerate}

One issue is that: there's no particular reason that a low-dimensional embedding has to have interpretable dimensions, right? even if they're orthogonal. you can just rotate them by some rotation matrix and mush the interpretable dimensions together. 

How does the CBM literature deal with this?
Subject: computer science
Intent: Paper Understanding