SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Handling interference between concepts. During the operationalization of concept  $i$ , we maintain  $S_0^{obv}$  unchanged. However, in handling interference (Section 3.4), we add data to other concepts and the original data, potentially leading to new conflicts with concept  $i$ . Therefore, in addition to considering the projection of  $S_i^{uno}$  on observed subspaces we need to consider the unobserved subspaces as well. Similar to above, if an unobserved direction in one concept has been observed in another concept we do not need to query user. With notation of  $S_0^{obv}$  denoting sum of all the  $S_i^{obv}$ , and  $S_{-i}$  denotes sum of all subspaces except  $i$ , the following proposition bounds number of times users need to add data to their concepts due to interference.
Proposition 2. If for all  $i$ ,  $\mathrm{proj}_{S_i^{uno}}(\theta^\star) = 0$  then the maximum number of times that we need to handle interference is  $\sum_{i=1}^k \dim \left(\mathrm{proj}_{S_{-i}}(S_i^{uno} \cap (S_i^{uno} \cap S_{0:k}^{obv})^\perp)\right)$ .
3Recall that only some of the data points are valid thus we need the generator to find a data points with variation in  $[0,0,1]$  direction.
Concept	Examples	Examples of bugs found by CoAlign
X person = not antonym (X) person	How can I become a positive person?
How can I become a person who is not negative?	predicts duplicate
shortcut bugs	{How can I become a mysterious person?
How can I become someone with no mystery?
predicts non-duplicate
overfit bugs	{How can I become a blind person?
How can I become someone who has lost his (physical) vision?
Modifiers changes question intent	Is Mark Wright a photographer?
Is Mark Wright an accredited photographer?	predicts not-duplicate
shortcut bugs	{Is he an artist?
Is he an artist among other people?
predicts duplicate
overfit bugs	{Is Joe Bennett a famous court case?
Is Joe Bennett a famous American court case?
Table 1: Examples of bugs found by CoAlign in the concepts introduced by CheckList, which were subsequently "debugged" using AdaTest, demonstrating that AdaTest had not yet fully operationalized these concepts.
Corig: “X = not antonym (X)”, Cnew: “Modifiers changes question intent”	Corig: “X = synonym (X)”, Cnew: “less X = more antonym (X)”
CoAlign	AdaTest	CoAlign	AdaTest
broken by new concept	7/50	24/50	9/50	18/50
fixed by new concept	5/50	2/50	20/50	18/50
Table 2: Comparison of CoAlign and AdaTest in terms of handling interference. For both methods, we labeled 50 sentences in the disagreement region of a model that only learned the original concept and a model that learned both original and the new concepts. CoAlign outperforms AdaTest when the new concept conflicts (top) or is similar (bottom) to the original concept.
We first show CoAlign outperforms AdaTest [5], a leading NLP debugging tool that also uses GPT-3, by revealing more bugs and resolving them without interference. We then show CoAlign can operationalize a concept even with biased seed data. We then show the selection mechanism of CoAlign outperforms baselines such as random sampling and uncertainty sampling. We conclude with a small pilot study of using CoAlign.<sup>4</sup>
CoAlign outperforms AdaTest[5]. We consider a scenario where a user finds multiple bugs and wants to fix them. Following Ribeiro and Lundberg [5], we use Quora Question Pairs (QQP) dataset where the goal is to predict if two questions are duplicate or not. We finetune RoBERTa-Large model on the QQP dataset, despite high accuracy of the model  $(92.2\%)$ , Ribeiro et al. [15] identified multiple concepts where the model has low performance. We use the 6 concepts with highest failure (see Table 1 for two of the concepts example).
For each of the 6 concepts, AdaTest iteratively adds data with GPT-3 and adaptively find failures "until finding failures becomes qualitatively difficult". For each concept, we initialize CoAlign with the AdaTest generated data as  $D_{i}$ . Even though the model has been 'debugged' with AdaTest, CoAlign quickly reveals  $\approx 5$  semantically meaningful sub-categories of bugs for each concept (with many failures within each sub-category). We show a few examples from different sub-categories in Table 1, which illustrate that AdaTest had not yet operationalized various regions of concepts where the model was still failing.
We now compare CoAlign with AdaTest in terms of handling interference. To do so, we pick pairs of concepts  $C_{orig}$ ,  $C_{new}$  that might cause interference with one another, but that were debugged and reported as "fixed" by Ribeiro and Lundberg [5]. We then run CoAlign on these pairs, noting that the output of both AdaTest and CoAlign are small concept datasets  $D_{orig}$  and  $D_{new}$ .
We then train two models for each method, one finetuned on  $D_{orig}$  and one finetuned on the union of  $D_{orig}$  and  $D_{new}$ . Finally, we generate new data from  $P_{orig}$  and manually label 50 instances where there is disagreement between the models, to check if adding  $D_{new}$  caused interference on  $P_{orig}$ . We present the proportion of sentences that were broken (right to wrong) or fixed (wrong to right) when the new concept is added in Table 2, disregarding instances that are invalid, out of domain, or for which the label is unclear. The top pair seems more liable to interference between concepts, but we note that AdaTest data results in much more interference than CoAlign data. In the bottom pair,
$^{4}$ Code and data will be released in https://github.com/fereshte-khani/CoAlign.
adding a concept with CoAlign actually improves the original concept more often than interferes with it, while AdaTest data has a neutral effect.
biased-SB	SB
Base	86.7 ± 2.5	82.6 ± 1.7
Data collection	98.6 ± 0.9	80.7 ± 1.6
CoAlign	94.9 ± 1.7	94.5 ± 1.1
Table 3: When we have access to a biased dataset (biased-SB) with consistently negative skin reviews and positive battery reviews. CoAlign outperforms naive data collection across the entire skin and battery reviews (SB) dataset by efficiently conceptualizing concepts and avoiding shortcuts.
CoAlign works even with biased seed data. To start working with CoAlign, a user should first provide some seed data (the first data points to train the local model). One natural question is that how much the model is dependent on the this seed data? This question is especially important when data is very biased and naive data collection might lead to very skewed data. Our goal is to understand if CoAlign can cover a concept even when the initial data points are very biased. For evaluating this question, we simulate a concept consists of reviews containing the phrases "battery life" or "my skin" (from now on, we refer to this concept as SB). We then simulate a very extreme scenario where instances with "battery life" are always positive, and those with "my skin" are always negative, from now on we refer to this distribution biased-SB. We train a weak base model by finetuning BERT-base-uncased on reviews that contain the word "install". Our goal is to see if CoAlign can lead to good accuracy in SB while only having data points from biased-SB.
We initialize CoAlign with 10 instances from biased-SB (5 positive and 5 negative sentence). We then run CoAlign for 5 rounds adding 10 data points in each round. To avoid the need of user labels, we train a high accuracy model as an oracle to simulate user labels. As an oracle, we train RoBERTa-Large on the whole Amazon Review dataset where we only keep reviews with rating 5 (positive sentiment) and rating 1 (negative sentiment). The accuracy of oracle on validation dataset is  $98.6\%$ . As shown in Table 3, despite starting with biased samples, CoAlign achieves high accuracy across all of SB. Conversely, the same number of random instances from biased-SB (simulating naive data collection) only increases accuracy for biased-SB while decreasing accuracy on the whole concept (SB). Qualitatively, GPT-3 starts generating in-concept instances without the bias, as that is where the local concept model disagrees with the base model. This controlled experiment illustrates how a generator focused on disagreements can be helpful in exploring the boundaries of concepts, even when we start from a biased sample.
Figure 4: CoAlign outperforms other data selection baselines when learning downward-monotone concept in MNLI task.
MNLI	Amazon
Overall average	Worst case	Overall average	Worst case
Random	89.6	87.7	87.6	85.3
Uncertainty	91.6	90.6	87.9	85.9
CoAlign	91.9	90.7	88.2	86.0
Table 4: CoAlign outperforms data selection baselines when simultaneously learning two concepts for MNLI (upward and downward monotone) and 4 concepts (product categories) for sentiment analysis. Results shown are an average of 10 runs.
CoAlign outperforms random and uncertainty sampling. We adapt CoAlign so that it selects instances from a pool of unlabeled data based on the disagreement score rather than using GPT-3 as a generator (lines 5-6 in Algorithm 1). We compare CoAlign with two data selection baselines: random selection and uncertainty sampling [16, 17]. Each method selects a batch of 50 examples per iteration (CoAlign selects examples randomly in the first batch), after which models are retrained.
We use RoBERTa-Large [18] finetuned on MNLI (binary) [19] as a base model, and use the downward-monotone concept from an NLI monotonicity dataset [2] as a pool of unlabeled data. The base model starts with high accuracy on the base validation dataset  $(93.5\%)$ , and low accuracy on the concept  $(23.5\%)$ . We present accuracy on the concept over iterations in Figure 4. While accuracy on the base
data remains constant throughout iterations, CoAlign's disagreement-based sampling is more efficient than uncertainty sampling or random selection.
In order to evaluate interference between concepts, we try to learn various concepts simultaneously. We use the same NLI model and concept as the previous paragraph, and add upward monotone [2] as an additional concept. We also use RoBERTa-LARGE finetuned on the "digital ebook" category of Amazon Reviews as a base model, and use 4 additional product categories as concepts to be learned (pet-product, home, home-improvement, and wireless). We run 10 iterations of each method, with random sampling and uncertainty sampling selecting an equal number of instances from each concept per round. Table 4 shows the per-concept average and worst case accuracy of each method, where we note that CoAlign produces better models on average.
Task & Topic	Seed data	Initial rounds	Final rounds
Sentiment & Islam	Lewis is a terrorist ← negative	Alice practices radical Islam	Alice prays for ISIS
Alice practices Islam ← neutral	Alice is a fundamentalist Muslim	Lewis is Muslim extremist
Marco goes to the mosque ← neutral	Alice wants to convert everyone to Islam	Lara, president of ISIS, said ...
Toxicity & Gender	I'm a man ← non-toxic	I'm feminine	Men have caused most of the wars
men are superior to women ← toxic	women like to use emotion to manipulate
Toxicity & Islam	ISIS is violent ← non-toxic	Dealing with Muslim countries can be tricky	Quron is a political manifesto
Muslims are violent ← toxic	Taliban is violent
Table 5: Some examples of a pilot study that qualitatively examines the effectiveness of the CoAlign in assisting users in conceptualizing their ideas and debugging the model. Users typically begin with simple sentences in their concept as seed data along with their labels. As users continue to use CoAlign, the CoAlign suggestions (disagreements between locally fitted model and the global model) become increasingly complex showing the diverse coverage of GPT-3.
Pilot Study (qualitative). We conducted a very small pilot study to show humans need assistance to operationalize their concept (i.e., we show that users might not even know the exact boundaries of their concept beforehand). Four users (one computer scientist and three with no prior knowledge of AI) used CoAlign to align a model within their chosen concept in either sentiment analysis or toxicity task. Each participant interacted with CoAlign for 5-7 rounds, and reported an improved alignment in their concept and an increase in sentence complexity over time. For example, for the Sentiment & Islam, the user-provided seed data matched the global model perfectly, indicating no initial bugs. However, in the first round of using CoAlign, some disagreement were found between the fitted local model and the global model. The user identified some of these as bugs in the global model (e.g. "Alice practices Islam daily" where the global model predicted negative) and some as bugs in the local model (e.g. "Alice is a radical Islamist" where the local model predicted neutral). As the user made repeated adjustments to both models, the disagreements gradually centered around the concept boundaries, reaching a point where users could no longer determine the correct behavior. This pattern of initial misalignment correction followed by uncertainty in label determination was consistent across all users, suggesting successful concept operationalization in clear label regions. Table 5 provides examples of user interactions with CoAlign<sup>5</sup>.
Our work relates to three areas of research: debugging, alignment, and interference management. We briefly review the relevant literature and highlight the differences and contributions of our work.
Debugging. Numerous efforts aim to find and fix models failures. Checklist [15] uses templates to test and improve models in different areas; however, these templates have low coverage for finding bugs. Dynabench [6] iteratively discovers bugs in a model by using human-generated adversarial examples; however, this approach requires human creativity. The closest work to us is AdaTest [5], which uses an LLM (and a few prompts) as a proxy for the user's behavior. Thus, unlike CoAlign AdaTest proxies do not adapt to the user's feedback and is susceptible to the biases and limitations of the LLM, resulting in lower performance than CoAlign. Moreover, AdaTest do not consider the interference among different concepts.
<sup>5</sup>We acknowledge that in sentiment and toxicity tasks a single instance can have multiple acceptable labels according to different users [20, 21], here we show that even a single user needs assistance to understand the boundary of their concept.
Alignment. The objective is to align a model with human intentions, a complex task since humans often cannot articulate their intentions. A prevalent method to tackle this is Reinforcement Learning with Human in the Loop (RLHF) [22-25], where the model learns a reward function based on human feedback on various outputs. The key distinction between our work and RLHF lies in our use of a local function for each concept, rather than a universal function for all concepts, and our generation of inputs in the model's disagreement region to assist users in better operationalizing their concept. Constitutional AI [26] works with multiple "expert" models in different domains, but they aggregate all the data generated by these models without addressing potential interference between them.
Interference management. Interference is a common problem in ML since it is hard to change a model's behavior locally without affecting other areas. The trade-off between accuracy and robust-accuracy where robust-training led to decrease in accuracy shown to be mitigated with self-training with the original model and using unlabeled data [11, 27]. Unlike their work, we do not have access to a reliable model for self-training and we need to improve the models while handling interference. Another type of interference is catastrophic forgetting [28, 29], in which learning on a new task may degrade the performance on the previously learned tasks. Some possible mitigation is multi-task learning [30, 31], or weight averaging between the original and the fine-tuned model [32]. Unlike these works, we are interested in exploiting the interference between models, as they can help the user operationalize their concept in the context of the model better. Lastly, this work primarily addresses interference arising from shortcut learning, there may be literal disagreements among users[20, 21]. Although a crucial issue, it falls outside the scope of this paper. However, we like to note that our method can surface such disagreements. This could potentially facilitate resolution through discussions, voting, or modifying the model to reflect a range of perspectives, especially in situations where there isn't a consensus among users.
Specifying model behavior on specific concepts is crucial in the development of NLP models, as it allows users to encode business rules, fix undesirable behavior, and force alignment with user values. Operationalizing concepts in a way that avoids shortcuts, overfitting, and interference with prior data is challenging. In this paper we presented CoAlign, a framework that leverages local concept models and large language models to help users operationalize concepts effectively while avoiding interference. We showed that CoAlign is more effective than prior work at exploring problematic concept regions, even prior work that uses the same language model and relies on interactive user feedback. We envision a future where NLP models are developed in a collaborative fashion, similar to open source software or Wikipedia, and speculate that harnessing the perspectives and expertise of a large and diverse set of users would lead to better models, both in terms of overall quality and in various fairness dimensions. For this scenario to materialize, we need ways to help users express their knowledge, and verify the impact of their proposed changes to models (the equivalent of "diffs" or "regression tests"). We believe CoAlign is a step in this direction.
8 Broader Impact and Limitations
CoAlign aids in operationalizing concepts without filtering the values a user wishes the model to align with, which might inadvertently allow a malicious user to encode harmful behavior into the NLP model, a risk for which we currently have no safeguards. Next, we only handled interference that arises from machine learning shortcomings and can be addressed by adding more data. However, there might be literal disagreements between users (i.e., two users prefer different labels for the same sentence). Although our method can surface such disagreements, we lack a definitive solution to resolve disagreements between users.
Furthermore, our efforts were primarily on classification tasks, leaving out generative tasks (e.g., next word prediction). A possible workaround is to use a local classifier where a user only indicates whether an output aligns with their concept or not, then train the global model accordingly (similar to Ouyang et al. [23] but with multiple reward models instead of just one). Lastly, our theoretical framework is limited but our goal was to gain some initial insights into why interference occurs and estimates the number of instances required to address it. Handling malicious users, resolving literal disagreements, studying CoAlign for generative tasks and more general theoretical analysis of alignment are compelling further research directions.
9 Acknowledgment
We gratefully acknowledge the contribution of Scott Lundberg, whose insightful discussions and assistance in utilizing the AdaTest repository greatly enhanced our research. Further, we thank Brent Hecht and Zexue He for providing invaluable early feedback on this work.
[30] Yu Zhang and Qiang Yang. An overview of multi-task learning. National Science Review, 5(1): 30-43, 2018.
[31] Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv:1511.06342, 2015.
[32] Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. arXiv preprint arXiv:2208.05592, 2022.
[33] Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in Neural Information Processing Systems, 30, 2017.
A Stationary distribution of the generator
Let  $\mathcal{P}_{\mathcal{G}}$  denote the transition probability of the generator, where  $\mathcal{P}_{\mathcal{G}}(x \mid r = x')$  denote probability of generating  $x$  condition on the prompt being  $x'$ . We want to create a Markov chain to simulate a random walk within the user's distribution  $(\mathcal{P}_i)$ . Specifically, we begin with a user-provided seed data point  $x$  and use it as a prompt for  $\mathcal{G}$  to generate a new data point  $x'$ . If  $\mathcal{P}_i(x') > 0$ , we accept  $x'$ , otherwise we remain at  $x$  and repeat the process. We are interested in conditions under which the stationary distribution of this markov chain is  $\mathcal{P}_i$
Let's assume that support of  $\mathcal{P}_i$  is finite and denote it by  $S$ , let  $\mathcal{P}_{\mathcal{G}}'(x'|x)$  be the transition function over  $S \times S$  where  $\mathcal{P}_{\mathcal{G}}'(x|x) = \mathcal{P}_{\mathcal{G}}(x|x) + \sum \mathcal{P}_{\mathcal{G}}(x'|x)\mathbb{I}[\mathcal{P}_i(x') = 0]$  and for  $x, x' \in S$  where  $x \neq x'$  we have  $\mathcal{P}_{\mathcal{G}}'(x|x') = \mathcal{P}_{\mathcal{G}}(x|x')$ .
If the transition graph generated by  $\mathcal{P}'$  is irreducible (any state can be reached from any other state) and all its states are positive recurrent (the expected time to return to a state is finite), then the unique stationary distribution using  $\mathcal{G}$  is  $\mathcal{P}_i$  if the following equality holds:
$$
\mathbb {E} _ {x \sim \mathcal {P} _ {i}} \left[ \mathcal {P} ^ {\prime} \left(x ^ {\prime} \mid x\right) \right] = \mathcal {P} _ {i} \left(x ^ {\prime}\right) \tag {1}
$$
The above statement states that the probability of a data point  $(\mathcal{P}_i(x))$  should be proportional to the probability of reaching to that point with the transition function of  $\mathcal{P}'$ .
If instead of only one data point we use  $m$  data points for prompt, we can create a graph where each node is  $m$  data points, and then analyse the stationary distribution of Markov chain on such graph. In this case, when we start from a node with  $m$  examples and prompt the language model with them in this case the probability of going to  $(x', x_m, \ldots, x_2)$  from  $(x_m, \ldots, x_1)$  is equal to  $\mathcal{P}_G'(x' | r = (x_m, \ldots, x_1))$ .
B Linear regression analysis
In this section, we examine the performance of CoAlign in a simple linear regression scenario. Specifically, we aim to investigate following aspects: (1) the number of data points required to teach a local concept to a global model, and (2) the reasons behind interference among concepts and the number of steps necessary to resolve it.
B.1 Setup
We consider each input  $x \in \mathbb{R}^d$  where only some of the data points are valid. There exist a true function  $\theta^{\star} \in \mathbb{R}^{d}$  such that  $y = \theta^{\star}^{\top}x$ . The support of each concept  $(\mathcal{P}_i)$  lays on a subspace  $(S_i)$  and all valid data points on that subspace belong to  $C_i$ . Given  $k$  examples in  $C_i$ , let  $S_i^{obv}$  denote the subspace observed by the training data,  $S_i^{uno}$  denote the unobserved subspace, thus  $S_i = S_i^{obv} + S_i^{uno}$  is the smallest subspace containing all data points in  $C_i$ . Finally  $S_i^{inv}$  denote the subspace that concept  $i$  does not have any variation in it.
As a running example, let  $x \in \mathbb{R}^3$  and consider a concept where data points belonging to that concept satisfies  $x_1 = x_2$ . Recall that only some of the data points in this subspace are valid e.g., a point is valid if  $x_1$  is odd thus [1, 1, 0] is valid while [2, 2, 1] is not. Let's assume we observed  $x = [1, 1, 0]$  in that subspace with label  $y = 2$ . In this case we have:  $S_1^{obv} = [1, 1, 0]$ ,  $S_1^{uno} = [0, 0, 1]$  and  $S_1^{inv} = [1, -1, 0]$ .