Collaborative Alignment of NLP Models
Authors: Fereshte Khani
Marco Tulio Ribeiro
Venue: NeurIPS 2023 poster (2023)
Keywords: alignment
collaborative alignment
debugging
nlp
interference
multi-user interaction
TL;DR: We enable experts/users to interact with the model and align it with their values without interfering with each other.

Abstract: Despite substantial advancements, Natural Language Processing (NLP) models often require post-training adjustments to enforce business rules, rectify undesired behavior, and align with user values.  
These adjustments involve operationalizing "concepts"—dictating desired model responses to certain inputs. 
However, it's difficult for a single entity to enumerate and define all possible concepts, indicating a need for a multi-user, collaborative model alignment framework. 
Moreover, the exhaustive delineation of a concept is challenging, and an improper approach can create shortcuts or interfere with original data or other concepts.

To address these challenges, we introduce CoAlign, a framework that enables multi-user interaction with the model, thereby mitigating individual limitations. 
CoAlign aids users in operationalizing their concepts using Large Language Models, and relying on the principle that NLP models exhibit simpler behaviors in local regions. 
Our main insight is learning a \emph{local} model for each concept, and a \emph{global} model to integrate the original data with all concepts.
We then steer a large language model to generate instances within concept boundaries where local and global disagree.
Our experiments show CoAlign is effective at helping multiple users operationalize concepts and avoid interference for a variety of scenarios, tasks, and models.

Collaborative Alignment of NLP models
Fereshte Khani Microsoft fkhani @mi crosoft.......o com
Marco Tulio Ribeiro Google DeepMind marcotcr@gmail. com
Abstract
Despite substantial advancements, Natural Language Processing (NLP) models often require cost-training adjustments to enforce business rules, rectify undesired behavior, and align with user values. These adjustments involve operationalizing "concepts" dictating desired model responses to certain inputs. However, it's difficult for single entity enumerate and define all possible concepts, indicating a need for multi-user, collaborative model alignment framework. Moreover, the exhaustive delineation of concept is challenging, and improper approach create shortcuts or interfere with original data or other concepts.
To address these challenges, we introduce CoAlign, a framework that enables multi- user interaction with the model, thereby mitigating individual limitations. CoAlign aids users in operationalizing their concepts using Large Language Models, and relying on the principle that NLP models exhibit simpler behaviors in local regions. Our main insight learning local model for each concept, and global model integrate the original data with all concepts. We then steer a large language model generate instances within concept boundaries where local and global disagree. Our experiments show CoAlign is effective at helping multiple users operationalize concepts and avoid interference for variety of scenarios, tasks, and models.
Introduction
NLP models have showcased remarkable capabilities, yet they not exempt from imperfections. Unacceptable values embedded their training data, persistent errors, or violations of business rules highlight the need teach certain concepts these models. A concept relates desired behaviors, e.g. in sentiment analysis, a concept may dictate that "religion does not connote sentiment" (e.g., "I'm Muslim" is Neutral). Similarly, in natural language inference (NLI), the broader concept of "downward monotonicity" specifies entailment relations when certain parts of expressions are made more specific (e.g., "All cats like tuna" entails "All small cats like tuna").
of inputs
The standard way of teaching concepts to models adding new training data that exemplifies the concept, e.g. adding neutral sentences containing religious words [.. or adding entailment pairs that illustrate downward monotonicity (2). However, it is hard to ensure that the data provided does not lead to shortcuts, i.c. spurious correlations or heuristics that allow models to make predictions without capturing the true underlying concept, such as "all sentences with religious terms are neutral", "going from general specific leads to entailment".. Further, the model may overfit fail to generalize from the instances provided to the actual concept, e.g. only recognizing pairs in the form ("all X...", "all ADJECTIVE X..."), and not pairs like ("all animals..." "all cats...)). Finally, both shortcuts and overfitting can lead to interference with the original data or other concepts, e.g. leading failures "I love Islam" pairs like ("Some cats like tuna". "Some small cats like tuna"). In sum, operationalizing concepts is challenging, since users typically cannot think of all concept boundaries and interactions in advance.
Work done while at Microsoft
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

ppl vesiss C boad
Lovr lobells iistanues where loic and giotaal sisggnee
Triking Intil local model
Updane global
User scedi
GPT-3 generstes data and findh dangrrements
Repeat Convergance
Figure 1: CoAlign loop for operationalizing a single concept. (a) The user starts by providing some seed data from the concept and their labels, (b) they are used to learn local concept model. (e) GPT-3 is then prompted to generate new examples, prioritizing examples where the local model disagrees with the global model. (d) Actual disagreements are shown the user for labeling, and (e) each label improves either the local or the global model. The loop c-d-e is repeated until convergence, i.e. until the user has operationalized the concept and the global model has learned it.
One possible solution is to ask domain experts to create data that covers the concept as compre- hensively and precisely as possible, e.g. the GLUE diagnostics dataset (3) or the FraCaS test suite 4]. However, these datasets are often expensive to create, small (and thus not suitable for training), and not exhaustive, even experts still fail to think about all aspects and nuances of concept [3]. Another solution is to have users provide data incrementally while they receive feedback from the model, e.g. adversarial training (6) or adaptive testing [5]. These do not require users think about everything in advance, and can expose and correct model weaknesses. However, adversarial training does not include the notion of concepts explicitly, and adaptive testing does not address the interaction between different concepts or between a concept and the original data. As result, users may not explore concept boundaries efficiently, and thus do not know when they have sufficiently covered a concept or whether they have introduced interference with other concepts.
In this paper, we introduce Collaborative Alignment of NLP Models (CoAlign). CoAlign leverages the collective knowledge of multiple users to cover many concepts instead of relying on a single user. CoAlign allows users to collaborate with AI systems and other users in specifying concepts and teaching them to models. Our key insight is making concepts explicit by learning a local model for each concept. in addition to a global model that integrates the original data and all additional concepts. When operationalizing a new concept, we rely on a large language model (LLM) to generate new instances where the local and global model disagree, and ask users to label such examples (Figure ??) Intuitively, these examples are either cases where the local model is not yet fully specified, or where the global model still makes errors on the concept (due to overfitting or shortcut reliance). As users label these examples, both models are updated until convergence, i.e. until the concept has been learned in a way that does not conflict with prior data or prior concepts. In a sense, each local model is ever-improving cheap expert in its respective concept. The speed of prediction of local models and diversity of examples generated by the LLM enable users to explore the boundaries between concepts and existing data, an exploration that could be challenging for users to do without aid.
Our experimental results demonstrate the effectiveness of CoAlign in operationalizing concepts and handling interference. We first show CoAlign outperforms AdaTest (5), a SOTA tool for debugging NLP models that also uses GPT-3 by revealing more bugs and fixing them without interference. We then demonstrate that CoAlign operatinalize concepts even when user starts with biased data, outperforming a model that relies solely on data collection. We compare the data selection mechanism of CoAlign to random selection and uncertainty sampling by running simplified version of CoAlign where instead of using GPT-3 we iteratively select examples from an unlabeled pool. We show CoAlign outperforms both baselines in teaching NLI model about downward- and upward- monotone concepts (3.. as well as teaching a sentiment analysis model about Amazon products reviews. Finally, in pilot study, we demonstrated that CoAlign helped users clarify their concepts.
2 Setup
Let x be text string, and y a categorical label, e.g. sentiment (positive, negative, neutral). We assume there is a true function f() = y that aim to approximate with a model f(x). Assume

---

we have access to a "base" dataset Do {[211 Vi)..... (Zn, Un)), e.g. of movie reviews, from base distribution Po- We refer to the model trained on Do as the base model fo-
concept C₁ is associated with a distribution P. over the input space, e.g. a concept might place probability mass exclusively on sentences containing religious words. We say I E C, if (z > 0. Since it's hard for users to be exhaustive, we do not assume users can generate from P. but that they can label any with (z), and as to whether it is in C2 or not. We further assume users can provide very small number samples in the support of P..
Without loss of generality, we assume that we have k users developing a model collaboratively, each with their own concept. Our goal is to train model ff that does well on both the base distribution and all concepts, i.c., minimizing E P.- [f(x)= * f(x)].
3 Collaborative Alignment of NLP models
In this section we describe how a single user operationalizes their concept by producing dataset D, in the context of an existing global model f trained the base dataset Do and previous concepts D1s-1- If fo is already "aligned" within the concept, i.e. fa(x;) f()) for all
C₁, would be done and there would be no need for D; Thus, what we really want is for D; to specify the boundary around "failures", cases not currently handled by fo- We abstract away the choice model and learning procedure, assuming that the model can learn the concept given D, (in practice, we use neural networks that are expressive enough).
3.1 Sampling from the concept
Since by assumption we cannot sample from the concept distribution P,, it is challenge to find regions of P. where fo fails. To address this, we use GPT-3 7 as a generator G to simulate random walk within the concept. To do so, we construct prompt with m in-concept examples, and use this prompt as input G to generate more samples. Then, we ask the user in the loop accept or reject each generated sample x' (z' is accepted if x' E C₃), and also to label the accepted x' with fx²). The value of controls the tradeoff between precision and recall, with high m generating more in-concept examples and low m exploring the space more broadly.
Under some conditions it can be shown that G simulates Markov chain with stationary distribution P; (Appendix [A). but the weaker condition of connectivity suffices for finding the concept failures, i.e. there must be path between any x', x" C. with nonzero transition probabilities according to G and the prompt. That is, if the concept is connected, with enough time we should be able to find regions of P. that are not already learned by fo.
While sampling from G eventually leads to the yet-unlearned concept regions, it is an inefficient use of human effort, as it does not use the user labels to guide generation (i.e. the user has to label many examples that the current model already handles conrectly). A better approach would be ask the user to label in a way that maximizes the expected information gain for the concept, to which we now turn.
3.2 Local Concept Models
Complex functions can be approximated by simpler functions in a local neighborhood, evidenced by theoretical results (e.g. Taylor cxpansion) and empirical applications [8. 9]. Since a concept is a natural local neighborhood, we use this insight and learn a local model f. to approximate f(x) in C,-
We present toy example for intuition in Figure 2 where we show toy distributions Po (Figure 2h) and Po with an additional concept P1 (Figure 2). In 2p. fo learned on samples from Po (dashed line) predicts most of P1 correctly, except for a small region in the bottom left quadrant. However, we would need many random samples from P; in order to learn that region, and reach the best model (solid line in Figure 20). In contrast, we can learn a good local model ft for Pt with a trivial number of samples (Figure 2b. This local model can be used to produce a disagreement region between fi and fo (Figure 2) and sampling from that region would lead to failure discovery much faster.
3

---

prompted such as generate in-concept samples that are likely be in the disagreement region. (a) The initial data is is (b) The user concept (P)) (c) local function for the (d) disagreement regions be- elliptical Gaussian
spherical Gaussian
data
tween local and global models
Figure 2: (a) A model is trained on two elliptical Gaussian (b) The spherical Gaussian are showing user concept, who wants find bugs in the model and teach the model about the new concept. However, since the model (dashed line) have high accuracy on the user concept is hard to find bugs and teaching the model requires data from the low probability region. (c) We fit a classifier the Spherical Gaussian which can be done with only a few data points (i.e., local model), (d) we then focus on disagreements between these two models to find bugs in the user concept.
More generally, we define a score function as the disagreement between the local and global function. This score function used steer generation such maximize the score of generated samples v', by adding instances to the prompt for G with probability proportional to their score (similar to Ribeiro and Lundberg [5], who different score function). We note that models may present false agreement on some samples, i.e. (z() = fo(x') * f(=)). To account for this, we also sample from the agreement region sometimes, with a probability that decays over time as we gain more confidence in the local model.
3.3
Operationalizing concept: from disagreement to convergence
The local and global models disagree on regions where the concept has not yet been learned (global is wrong) the local model does not fit the user's concept correctly (local is wrong). Thus, every label from the disagreement region results in an improvement in whichever model is incorrect. As labeling progresses, we update both the concept model f; and the global model f until we reach convergence (i.e. f has learned the concept C:). Note that while fü is trained on data from P., f is trained to do well on the base distribution Po. as well as on all concepts Pi.k
(a) seed
(b) local fit
(c) generation
(d) selection
(e) update
Figure 3: This figure illustrates the main steps in CoAlign: (a) The user starts by providing small number of data points and their labels within their domain (represented by the black circle), (b) we fit simple model (shown as linear model) to represent the user's concept, (c) we use the generator (g) to generate data points towards the region where the local model disagrees with the global model, (d) a diverse set of data points are selected for the user to label, (e) based on the user's feedback, the local and global models are updated until convergence.
We present pseudo-code for operationalizing concept in Algorithm [ and illustrate it in Figure The local and global models are initialized in lines 1-2 (Figure 3p and b). In line 5 (Figure 3c.) g
Then, in line 6 (Figure 3d), we select a batch of examples for labeling according to the disagreement score of the generated instances. This generation-labelnn both the local and global models get updated in lines 7-8 (Figure Be). For the sake of interactivity. practice we do train from scratch when updating the models, and instead finetune them from a previous checkpoint. After a few iterations, ʃ and f; converge (i.e. the user has operationalized the concept and the global model has learned it). If the number of generated samples b L is large
loop repeated for L iterations, after which

---

enough, we can assume that no disagreement on generated sample between f and fi means they have converged, and thus we stop and output the generated data Di (line 9).
Algorithm 1: Operationalizing a new concept i
input :Bss dataset Do. Concept datasets Dia-1, A small set of samples from concept D; Init local and global: Train f; on D;, and train f on Doa-1;;
2 do
for L iiterations do
Generation: Prompt 9 with subset from D, chosen with probability lf(x) f((x);
Labeling: Select b samples with prob. x 1f(x²) - -((x')) Users reject x' if out of
concept, or add to D, with label f(x'):
Update local and global: Train ft on D;, and train f on Doci 7 while D; was updated this round; output AA dataset D;
3.4 Handling interference between concepts
When aligning a model to multiple users (or training model on different distributions), two types of conflicts can happen: (I) Literal disagreements: where two users disagree in labeling the same input. (2) Interference: where over-generaliiztion in one concept interfere with another concept (e.g.. if a user adds lot of neutral examples for the "religion does not connote sentiment" concept, then training the model on these new examples can cause the model to predict neutral for sentences like "I love Islam" which interfere with concept that says "love X" is positive). As stated in Section (2] we assume there is a single true function f (thus no literal disagreement) and we only focus on handling interference. Handling interference is crucial since any local change in ML models can interfere with other parts of the model 12]
Having local functions (cheap experts) enable us to check interference efficiently. Every time that operationalizes concept according to Algorithm we check the resulting global model f against the local models for all previous concepts. To do so, we re-run Algorithm | only asking the user for labels if there is newfound disagreement region. In practice, this means that user adding new concept needs to make sure it does not break any concepts from other users, a process similar to regression testing in software engineering.
While we want to handle interference, having multiplicity of concepts can be beneficial in refining a concept. In particular, training data) and generalize well; however generalizing well is dependent on other concepts and the previous data. Take, for instance, world where only bananas are yellow. A user might guide an ML model to recognize bananas solely based on their yellow color. Now if new user introduces another yellow object, like corn, the model must discern other distinguishing features. Merely combining training data for two such concepts doesn't 't suffice (as shown in [TT);; the boundaries must be distinguished. Furthermore, interference can be beneficial by exposing false agreement regions between the global model and any individual concept model. In other words, while both ff and fü may be relying on the same shortcut to solve some regions of P. it is unlikely that this shortcut does not cause interference between f and all local models f, for j # i. In this case, the interference caused by adding Cj is actually beneficial, as further refines concept Cz.
want the global model to not overfit to concept (i.e., only memorize the
In practice the original dataset (D₂) is often very large and we cannot fine-tune the model on Dock- Instead of choosing the whole Do, every time we sample data points with highest disagreement between fo and ff from Dy. In other word, we treat fo as a user with concept distribution Po. this enable us to deal with interference with original model as well (as an example in Figure we might only choose the data points from elliptical Gaussian in disagreement region).
We like to note that while we haven't 't introduced specific mechanism to resolve literal disagreements, our method can surface such disagreements. This can pave the way for resolution through discussions, voting. or even tweaking the model to reflect multiple perspectives, especially in cases where there isn' 't a consensus among
5

---

4 Linear regression analysis
We now study CoAlign in a noiseless overparametrized linear regression scenario. This particular setup is used in recent literature to gain some insights into the behaviors of deep networks [10] 14
13
For more details and proofs see Appendix B
Setup. Let x E IR where only subset the data points are valid. We assume 0" E Rd such that = 0* T x. Let St denote the smallest subspace containing all valid data points of C.- Given examples in C, let S: denote the subspace observed by the training data, S.mo denote the unobserved subspace (thus S. = Sp + Sy and S: is the subspace that concept i does not have any variation in it. We consider overparametrized noiseless linear regression, where the number of features (d) exceeds the number of training data, enabling us to always interpolate all training data. We assume local and global models infer the min L2 norm interpolant.
As running example, let E R33 and consider concept where data points belonging to that concept satisfies 11 = Z2. Let's assume we observed æ = [1, 1, 0J with label 2. In this case we have: Sp = {[1, 1, 0]} Sym = {[o, 0, 1}} and Sime = {[1, -1, 0]}. The min-norm solution interpolating
the concept is 0. = [1,
Operationalizing. a concept. The min-nom solution can also be constructed by introducing constraints that projection of 0. on Sumo and S!n" has to be zero. In the above example, the min- norm solution is the unique answer of the following linear equations ((0, 0, 1]6 = 0, 1, 0, 1, = 2). When we teach the global model about a concept, the naive combination of data points could violate these constraints, leading to disagreements between local and global models. For our running example, suppose the original data point is [0, 1, 1] y 2. Combining the concept data point with the original data point for inferring global model results in gglobull = 1 31 3 which leads to a disagreement between local and global model for data points varying in the 0, direction.
In case of disagreement, two scenarios could arise: (1) the direction is non-zero and the local model needs more specification (happens a lot at the beginning), (2) the direction is zero, but explicit constraint is needed to prevent the global model from assuming other values. In our running example, the generator can help us to find data point that the two model disagree Let's assume the data point is x = [0, 0, 1] where local model predicts 0 but global model predicts 3. We show this data point to the user and either the user confirms (0, 0, 1] as non-zero makes the zero explicit, resulting
in global prediction of gglebu 2,.
Once we learn the local concept (i.e., all unobserved directions are indeed zero), how many of Sumo directions need to be added explicit constraints? Intuitively, we do not need query user for any direction in 5uuo that has already been observed in S or are orthogonal S since there is no interference. The following proposition indicates the maximum number of disagreements for teaching global model about the local concept.
Proposition 1. If proje (0*) O, then the maximum number of disagreement between local and global models dimppprjssed (S n (Sumn n (ibh))
Handling interference between concepts. During the operationalization of concept i, we maintain Sabe unchanged. However, in handling interference (Section 3.4). we add data to other concepts and the original data, potentially leading to new conflicts with concept i. Therefore, in addition to considering the projection of Sumo on observed subspaces we need to consider the unobserved subspaces as well. Similar to above, if an unobserved direction in one concept has been observed in another concept we do not need to query user. With notation of Sobe denoting sum of all the Sobe and S denotes sum of all subspaces except i, the following proposition bounds number of times users need to add data to their concepts due to interference.
Proposition 2. If for all i, projs (0*) = then the maximum number of times that we need to handle interference is Σk_ - dim (prois.. (S n (Suno n S0.k )))
Recall that only some of the data points are valid thus we need the generator to find a data points with variation in [0, 0, 1] direction.

---

Concept
Examples
Examples of bugs found by CoAlign
person not antonym (X) person
posttic perwot
predecs depiictte
shakel
How
Bow becoree amrorn with myster??
mysrrdiat
grediets rox daplicat
Iiw bosme penso??
Hiw cas bearms someane who has lost his (pbysiza)) visian
Modifiers changes question intent
Ie Mark Wright a planommphest
predicts tat-daplicate
he on anti??
Is Mark Wiight iceecited p phonorgphur?
Is
odler people?
predicts duplicste onurta bigs
Ise fanous Amterican sase?
Jee Bernet Semooe coumt caes?
Table Examples of bugs found by CoAlign in the concepts introduced by CheckList, which were subsequently "debugged" using AdaTest, demonstrating that AdaTest had not yet fully operationalized these concepts.
Cavig" "X re seteyy (XT: Cuersi whw.://rrr.tnngg....sooooo
Carty 'K qutoyy (XT". Cuess"" "Tro x mor amrnym (k)"
CoAlign
AdaTest
CoAlign
AdaTest
broken by new concept fixed by new concept
7/50
24/50
9/50
18/50
5/50
2/50
20/50
18/50
Table 2: Comparison of CoAlign and AdaTest in terms of handling interference. For both methods, we labeled 50 sentences in the disagreement region of a model that only learned the original concept and model that learned both original and the new concepts. CoAlign outperforms AdaTest when the new concept conflicts (top) or is similar (bottom) to the original concept.
5 Experiments
We first show CoAlign outperforms AdaTest [5], a leading NLP debugging tool that also uses GPT- 3, by revealing more bugs and resolving them without interference. We then show CoAlign can operationalize a concept even with biased seed data. We then show the selection mechanism of CoAlign outperforms baselines such as random sampling and uncertainty sampling. We conclude with small pilot study of using CoAlign
CoAlign outperforms [5 We consider a scenario where a user finds multiple bugs and wants to fix them. Following Ribeiro and Lundberg [3], we use Quora Question Pairs (QQP) dataset where the goal is to predict two questions are duplicate or not. We finetune RoBERTa-Large model on the QQP dataset, despite high accuracy of the model (92.2%), Ribeiro al. [15) identified multiple concepts where the model has low performance. We use the 6 concepts with highest failure (sce Table for two of the concepts example).
For each of the concepts, AdaTest iteratively adds data with GPT-3 and adaptively find failures "until finding failures becomes qualitatively difficult". For each concept, we initialize CoAlign with the AdaTest generated data as D,. Even though the model has been 'debugged' with AdaTest, CoAlign quickly reveals R 5 semantically meaningful sub-categories of bugs for each concept (with many failures within each sub-category). We show a few examples from different sub-categories Table T which illustrate that AdaTest had not yet operationalized various regions of concepts where the model was still failing.
We now compare CoAlign with AdaTest in terms of handling interference. To do so, we pick pairs of concepts Corig: Cnew that might cause interference with one another, but that were debugged and reported as "fixed" by Ribeiro and Lundberg [3). We then run CoAlign on these pairs, noting that the output of both AdaTest and CoAlign are small concept datasets Dorig and D Dnew-
We then train two models for each method, one finetuned on Darig and one finetuned on the union of Darig and Daew- Finally, generate new data from Porig and manually label 50 instances where there is disagreement between the models, to check if adding Dnew caused interference on P Parig- We present the proportion of sentences that were broken (right wrong) or fixed (wrong to right) when the new concept is added in Table 2 disregarding instances that are invalid, out of domain, or for which the label unclear. The top pair seems more liable interference between concepts, but we note that AdaTest data results in much more interference than CoAlign data. In the bottom pair,
'Code and data will be released in https:://gttuub.coo/feees
7

---

adding concept with CoAlign actually improves the original concept more often than interferes with it, while AdaTest data has neutral effect.
Table 3: When we have access to biased dataset (biased- SB) with consistently negative skin reviews and positive battery reviews. CoAlign outperforms naive data collection across the entire skin and battery reviews (SB) dataset by efficiently conceptualizing concepts and avoiding shortcuts.
biased-SB
SB
Basc
86.7 + 2.5
82.6 + 1.7
Data collection
98.6 + 0.9
80.7 1.6
CoAlign
94.9 f 1.7
94.5111
CoAlign works even with biased seed data. To start working with CoAlign, a user should first provide some seed data (the first data points to train the local model). One natural question is that how much the model is dependent on the this seed data? This question especially important when data is very biased and naive data collection might lead to very skewed data. Our goal is to understand CoAlign can cover a concept even when the initial data points are very biased. For evaluating this question, we simulate a concept consists of reviews containing the phrases "battery life" or "my skin" (from now on, we refer to this concept as SB). We then simulate very extreme scenario where instances with "battery life" are always positive, and those with "my skin" are always negative, from now on we refer to this distribution biased-SB. We train a weak base model by finetuning BERT-base basT-bese uncased on reviews that contain the word "install"" Our goal is to see if CoAlign can lead to good accuracy in SB while only having data points from biased-SB.
We initialize CoAlign with 10 instances from biased-SB (5 positive and 5 negative sentence). We then run CoAlign for 5 rounds adding 10 data points in each round. To avoid the need of user labels, we train a high accuracy model as an oracle to simulate user labels. As an oracle, we train RoBERTa-Large on the whole Amazon Review dataset where we only keep reviews with rating (positive sentiment) and rating 1 (negative sentiment). The accuracy of oracle on validation dataset is 98.6%. As shown in Table despite starting with biased samples, CoAlign achieves high accuracy across all of SB. Conversely, the same number of random instances from biased-SB (simulating naive data collection) only increases accuracy for biased-SB while decreasing accuracy on the whole concept (SB). Qualitatively, GPT-3 starts generating in-concept instances without the bias, that is where the local concept model disagrees with the base model. This controlled experiment illustrates how generator focused on disagreements can be helpful in exploring the boundaries of concepts, even when we start from a biased sample.
MNLI
Amazon
Overall average
Worst case
Overall average
Worst case
0.89
0.80
Random 89.6
87.7
87.6
85.3
0.7
Uncertainty 91.6
90.6
87.9
85.9
9.99
DaDky
Nardom
CoAlign 91.9
90.7
88.2
86.0
Table 4: CoAlign outperforms data selection baselines when simultancously learning two concepts for MNLI (upward and downward monotone) and 4 concepts (product categories) for sentiment analysis. Results shown are average of 10 runs.
Figure 4: CoAlign outperforms other data selection baselines when learning downward-monothn con- cept in MNLI task.
CoAlign outperforms random and uncertainty sampling. We adapt CoAlign so that it selects instances from a pool of unlabeled data based on the disagreement score rather than using GPT-3 as generator (lines 5-6 in Algorithm Tp. We compare CoAlign with two data selection baselines: random selection and uncertainty sampling [16] [7]. Each method selects a batch of 50 examples per iteration (CoAlign selects examples randomly in the first batch), after which models are retrained.
We use LoBERTa-Large [8 finetuned on MNLI (binary) 19 as a base model, and use the downward- monotone concept from an NI.I monotonicity dataset 2 as a pool of unlabeled data. The base model starts with high accuracy on the base validation dataset (9).5%) and low accuracy on the concept (23.5%). We present accuracy on the concept over iterations in Figure 4 While accuracy the basc
8

---

data remains constant throghout iterations, CoAlign's 's disagreement- bassed sampling is more efficient than uncertainty sampling or random selection.
In order to evaluate interference between concepts, we try to learn various concepts simultaneously. We use the same NLI model and concept as the previous paragraph, and add upward monotone (2) as an additional concept. We also use RoBERTa-LARGE finetuned on the "digital ebook" category of Amazon Reviews as a base model, and use 4 additional product categories concepts to be learned (pet-product, home, home-improvement, and wireless). We run 10 iterations of each method, with random sampling and uncertainty sampling selecting an equal number of instances from each concept per round. Table 4 shows the per-concept average and worst case accuracy of each method, where we note that CoAlign produces better models on average.
Task & Topic
Seed data
Initial rounds
Final rounds
Lewis a temorist negative
Alice practices radical Islam
Alice prays ISIS
Sentiment & Islam
Alice practices Islam neutral
Alice is fundimentalist Muslim
Lewis is Muslim extremist
Marco goes to the mosque + noutral
Alics wants to couvert everyone to Islam
Lara, president of ISIS, said .i.....
Toxicity & Gender
I'm a man e- non-toxic
I'm feminine
Men have caused most of the wars
are superior women toxic
women like use emotion manipulate
ISIS is violent non-toxic
Dealing with Muslim countries can be tricky
Toxicity & Islam
Quron political manifesto
Muslims are violent + toxic
Taliban is violeut
Table 5: Some examples of a pilot study that qualitatively examines the effectiveness of the CoAlign in assisting users in conceptualizing their ideas and debugging the model. Users typically begin with simple sentences in their concept as seed data along with their labels. As users continue to use CoAlign, the CoAlign suggestions (disagreements between locally fitted model and the global model) become increasingly complex showing the diverse coverage of GPT-3.
Pilot Study (qualitative). We conducted very small pilot study to show humans need assistance to operationalize their concept (i.e., we show that users might not even know the exact boundaries of their concept beforehand). Four users (one computer scientist and three with no prior knowledge of Al) used CoAlign to align model within their chosen concept in either sentiment analysis or toxicity task. Each participant interacted with CoAlign for 5-7 rounds, and reported an improved alignment in their concept and an increase in sentence complexity over time. For example, for the Sentiment & Islam, the user-pre provided seed data matched the global model perfectly, indicating no initial bugs. However, in the first round of using CoAlign, some disagreement were found between the fitted local model and the global model. The user identified some of these as bugs in the global model (e.g. "Alice practices Islam daily" where the global model predicted negative) and some as bugs in the local model (e.g. "Alice is radical Islamist" where the local model predicted neutral). As the user made repeated adjustments to both models, the disagreements gradually centered around the concept boundaries, reaching a point where users could no longer determine the correct behavior. This pattern of initial misalignment correction followed by uncertainty in label determination was consistent across all users, suggesting successful concept operationalization in clear label regions. Table 5 provides examples of user interactions with CoAligr
6 Related Work
Our work relates to three areas of research: debugging, alignment, and interference management. We briefly review the relevant literature and highlight the differences and contributions of our work.
Debugging. Numerous efforts aim find and fix models failures. Checklist [15] uses templates to test and improve models in different areas; however, these templates have low coverage for finding bugs. Dynabench [6] iteratively discovers bugs in model by using human-generated adversarial examples; however, this approach requires human creativity.Thh closest work to us is AdaTTss [5], which uses an LLM (and few prompts) as a proxy for the user's behavior. Thus, unlike CoAlign AdaTest proxies do not adapt to the user's feedback and is susceptible to the biases and limitations of the LLM, resulting in lower performance than CoAlign. Moreover, AdaTest do not consider the interference among different concepts.
'We acknowledge that in sentiment and toxicity tasks a single instance can have multiple acceptable labels according to different users [20] 21]. here we show that even a single user needs assistance to understand the boundary of their concept.

---

Alignment. The objective is to align model with human intentions. a complex task since humans often cannot articulate their intentions. A prevalent method to tackle this is Reinforcement Learning with Human in the Loop (RIHF) 222 where the model learns reward function based on human feedback on various outputs. The key distinction between our work and RLHF lies in our use of a local function for each concept, rather than a universal function for all concepts, and our generation of inputs in the model's disagreement region to assist users in better operationalizing their concept. Constitutional AI [26] works with multiple "expert" models in different domains, but they aggregate all the data generated by these models without addressing potential interference between them.
Interference management. Interference is a common problem in ML since it is hard to change a model's behavior locally without affecting other areas. The trade-off between accuracy and robust- accuracy where vobust-training led to decrease in accuracy shown to be mitigated with self-training with the original model and using unlabeled data [1] 27 Unlike their work, we do not have access to reliable model for self-training and we need to improve the models while handling interference. Another type of interference catastrophic forgetting 28 29), in which learning on new task may degrade the performance on the previously learned tasks. Some possible mitigation is multi-task learning [1),1] or weight averaging between the original and the fine-tuned model [32]. Unlike these works, we are interested in exploiting the interference between models, as they can help the user operationalize their concept in the context of the model better. Lastly, this work primarily addresses interference arising from shorteut learning, there may be literal disagreements among users (20) 21]. Although a crucial issue, it falls outside the scope of this paper. However, we like to note that our method can surface such disagreements. This could potentially facilitate resolution through discussions, voting, or modifying the model to reflect a range of perspectives, especially in situations where there isn't consensus among users.
7 Conclusion
Specifying model behavior on specific concepts is crucial in the development of NLP models, it allows users to encode business rules, fix undesirable behavior, and force alignment with user values. Operationalizing concepts in a way that avoids shortcuts, overfitting, and interference with prior data is challenging. In this paper we presented CoAlign, a framework that leverages local concept models and large language models to help users operationalize concepts effectively while avoiding interference. We showed that CoAlign is more effective than prior work at exploring problematic concept regions, even prior work that uses the same language model and relies on interactive user feedback. We envision a future where NLP models are developed in a collaborative fashion, similar to open source software Wikipedia, and speculate that harnessing the perspectives and expertise of a large and diverse set of users would lead to better models, both in terms of overall quality and in various faimess dimensions. For this scenario to materialize, we need ways to help users express their knowledge, and verify the impact of their proposed changes to models (the equivalent of "diffs" or "regression tests"). We believe CoAlign is a step in this direction.
8 Broader Impact and Limitations
CoAlign aids in operationalizing concepts without filtering the values a user wishes the model to align with, which might inadvertently allow a malicious user to encode harmful behavior into the NLP model, a risk for which we currently have no safeguards. Next, we only handled interference that arises from machine learning shortcomings and can addressed by adding more data. However, there might be literal disagreements between users (i.e., two users prefer different labels for the same sentence). Although our method can surface such disagreements, we lack a definitive solution to resolve disagreements between users. Furthermore, our efforts were primarily on classification tasks, leaving out generative tasks (c.g.. next word prediction). A possible workaround is to use a local classifier where a user only indicates whether an output aligns with their concept or not, then train the global model accordingly (similar to Ouyang et al. (23 but with multiple reward models instead of just one). Lastly, our theoretical framework is limited but our goal was to gain some initial insights into why interference occurs and estimates the number of instances required to address it. Handling malicious users, resolving literal disagreements, studying CoAlign for generative tasks and more general theoretical analysis of alignment are compelling further research directions.
10

---

Acknowledgment
We gratefully acknowledge the contribution of Scott Lundberg, whose insightful discussions and assistance in utilizing the AdaTest repository greatly enhanced our research. Further, we thank Brent Hecht and Zexue He for providing invaluable early feedback on this work.
References
[1] Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAT/ACM Conference on AI, Ethics, and Society, pages 67-73, 2018.
[2] Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, and Kentaro Inui. Do neural models learn systematicity of monotonicity inference in natural language? arXiv preprint arXiv:2004. 14839, 2020.
[3] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NIP,, pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. N0.1863//1//1885444 URI.https: Utt.htttss///aclannn peattps://allanthooog.
org hrg/W18-5446
[4] Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, et al. Using the framework. Technical report, Technical Report LRE 62-051 D-16, The FraCaS Consortium, 1996.
[5] Marco Tulio Ribeiro and Scott Lundberg. Adaptive testing and debugging of nlp models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages
2022.
[6] Douwe Kiela, Max Bartolo, Yixin Nic, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma. Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp. Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 41 41244412, Online, June 2021. Association for Computational Linguistics. doi: 10. 118886651100000 maa.c-mmii...244
URL https://
acl aclanthology .ww.oogg.ooro.oooooooooo 200gg/222..hc www..laaacllaammmmmm. 324
[7] Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:18 17:117711 2020.
[8] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier. In Knowledge Discovery and Data Mining (KDD), 2016.
[9] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
[10] Fereshte Khani and Percy Liang. Removing spurious features can hurt accuracy and affect groups disproportionately. ACM Conference on Fairness, Accountability, and Transparency (FAceT), 2021.
[11] Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John C. Duchi, and Percy Liang, Adversar- ial training can hurt generalization. arXiv preprint arXiv:1906.
2019.
[12] Megha Srivastava, Besmira Nushi, Ece Kamar, Shital Shah, and Eric Horvitz. An empirical analysis of backward compatibility in machine learning systems. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3272-3280, 2020.
11

---

[13] Preetum Nakkiran. More data can hurt for linear regression: Sample-wise double descent. arXiv preprint arXiv:1912.07244. 2019.
[14] Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John Duchi, and Percy Liang. Un- derstanding and mitigating the tradeoff between robustness and accuracy. arXiv preprint vrXiv:2002.107110 2020.
[15] Marco Tulio Ribeiro. Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy. Behavioral testing of NLP models with CheckList. In Association for Computational Linguistics (ACL), pages 1902-4912, 2020.
[16] David D Lewis. A sequential algorithm for training text classifiers: Corrigendum and additional data. In Acm Sigir Forum, volume 29, pages 13-19. ACM New York, NY, USA, 1995.
[17] Vu-Linh Nguyen, Mohammad Hossein Shaker, and Eyke Hüllermeier. How to measure uncer- tainty in uncertainty sampling for active learning. Machine Learning. (11(1):89-111) 122, 2022.
[18) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: 11692, 2019.
[19] Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 2018.
[20] Ruyuan Wan, Jachyung Kim, and Dongyeop Kang. Everyone's voice matters: Quantifying annotation disagreement using demographic information. arXiv preprint arXiv:2301.05036. 2023.
[21] Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. Dealing with dis- agreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10:92-110. 2022.
[22] Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[23] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02/55 0515,, 2022.
[24] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 19XXvv19090085000 2019.
[25) Wangchunshu Zhou and Ke Xu. Learning to compare for better training and evaluation of open domain natural language generation models. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9717-9724, 2020.
[26] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077. 2022.
[27] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C. Duchi, Unlabeled data improves adversarial robustness. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[28] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165, Elsevier, 1989.
[29] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3222 2017.
et al.
12

---

[30] Yu Zhang and Qiang Yang. An overview of multi-task learning. National Science Review, 5(1): 30-43, 2018.
[31] Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning, arXiv preprint arXiv: (OX1:0/411 2015.
[32] Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Korbliihh, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. arXiv preprint arXiv:208..05592, 2022.
[33] Suriya Gunasckar, Blake Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in Neural Information Processing Systems, 30, 2017.
13

---

A Stationary distribution of the generator
Let Pg denote the transition probability of the generator, where Pa(a: rr= denote probability of generating x condition on the prompt being x'. We want to create a Markov chain to simulate a random walk within the user's distribution (P:). Specifically, we begin with a user-provided seed data point x and use it as prompt for & to generate a new data point æ'. If P: > O, we accept x', otherwise we remain at a and repeat the process, We are interested in conditions under which the stationary distribution of this markov chain is P:
Let's assume that support of P; is finite and denote it by S, let (2')a( be the transition function over S x S where Pig x) Pg(r 2:) + Σ Po [P) (2') = 0) and for af E S where I A x' we have (r x') Pa( x²).
If the transition graph generated by P' is irreducible (any state can be reached from any other state) and all its states are positive recurrent (the expected time to return to state is finite), then the unique stationary distribution using 9 P, if the following equality holds:
E [P'('' æ)] (x')')
The above statement states that the probability of a data point (P)())) should be proportional to the probability of reaching to that point with the transition function of P'.
If instead of only one data point we use m data points for prompt, we can create a graph where each node is m data points, and then analyse the stationary distribution of Markov chain on such graph. In this case, when we start from a node with m examples and prompt the language model with them in this case the probability of going (z', Zm...., 222 from (Im..... 21) equal to PP'('' (Zm..... 21)).
B Linear regression analysis
In this section, we examine the performance of CoAlign in simple linear regression scenario. Specifically, we aim to investigate following aspects: (1) the number of data points required to teach local concept to a global model, and (2) the reasons behind interference among concepts and the number of steps necessary to resolve it.
B.1 Setup
We consider each input Rd where only some of the data points are valid. There exist true function 0* E Rd such that y 0" x. The support of each concept (P,) lays on a subspace (S:) and all valid data points on that subspace belongs to C. Given & examples in C₁ let Sabe denote the subspace observed by the training data, Symn denote the unobserved subspace, thus S, S + Suno is the smallest subspace containing all data points in C. Finally S:iwe denote the subspace that concept i does not have any variation it.
As a running example, let x E I and consider a concept where data points belonging to that concept satisfies 31 Z2. Recall that only some of the data points in this subspace are valid e.g., a point is valid if 2₁ is odd thus [1, 1. 0] is valid while [2, 2, 1] is not. Let's assume we observed r = [1, 1, 1.0] in that subspace with label = 2. In this case we have: Sqbu = [1, 11 0J. Stuno = [0. 0, 1] and
Sine - 1,.
We consider the overparametrized noiseless linear regression, where number of features (d) is larger than number of acquired training examples (n) thereefre, we can always interpolate all the training data) and there is no noise in observed targets. Following work of (33) which showed gradient descent on linear regression lead to min L2-norm, assume local and global models infer the min L2 norm interpolant. As an example, for our running example the non-norm solution interpolating the concept
i = [1, 0,
B.2
Operationalizing a concept: from disagreement to convergence
An alternate interpretation of the min-norm involves inferring the parameters by taking into account explicit constraints that require 0's projection Suno and Siav to be zero. For instance, in our
14

---

current example, we can deduce the min-nomm solution by solving these linear equations: ((0, 0, 1]0 0, -1, [1. 0 2).
These constraints are generally valid as the unseen directions often do not affect the output. However, these constraints may be violated when combine local concept data with global data, as the projection of Sino and S0 may not be zero. This implies that the output could change with variations in the unseen directions, leading local models typically outperforming global models within a local concept.
To ensure both local and global models perform equally well in the local concept, need to enforce the invariance constraints explicitly. This involves adding new data that exhibit variations in the unseen directions and demonstrating that these variations do not affect the output. Furthermore, we presume that Suno is significantly large, making methods that attempt to examine all possible directions inefficient. Therefore, it's more advantageous to only verify directions that affected by the merge.
Consider the previous example where observed x 1,0 2 for the local concept. Now, imagine the we observed x = [0, 1, 11, = in global dataset. When we combine this data point with the concept data point, we get gglabal = [3g 3 3 This causes a disagreement in data points that vary in the [0, 1] direction within the local concept, the local model predicts while the global model predicts 3. Note that both the global and local predictions align for variations in the [1, 1, 0J direction.
4 2
In the event of such a disagreement, we have two options: (1) The variation in this direction is indeed non-zero, suggesting the local model requires further refinement frequent occurrence in early stages, or (2) The variation is zero, but it needs to be specified as such; otherwise, the global model assumes other values due its implicit bias towards generating the simplest model. Note that there is no disagreements in the common directions between Syno and Syio or their orthogonal subspaces.
Referring to the above example, the generator identifies data point where the two models disagree. Let's assume this data point is x = [0, 0, 1]. where the local model predicts 0, but the global model predicts 2. In such case, we present this data point to the user. Let's assume user specify that the label for this data point O. In this case by adding this new data point the global prediction adjusts to Ogloball = 2,
After we learn the local concept (i.e., all the unobserved directions are indeed zero), how many of them do we need add as explicit constraints? the following proposition shows maximum number of disagreements after learning a local concept.
Proposition 1. If projs (0*) O, then the maximum number of disagreement between local and global models dim(proj (S n (Stmm n )
Proof. The global and local models agree on all observed directions (i.e., Sibu and Solr). However, there is disagreement for any vector in Sunno such that Ogiohal = prois (0*) + 0 since Ø_ u = 0. Let's assume we add l examples such that local and global disagree. We now prove that k < dim(proisg* (Suno n (Stuno n -))
For the A added examples, only consider their components in (Suno n (S n Sabe) (we can remove the Sobr components by subtracting their projection on Sibe similarly remove any component in (S!Nn n Sgbe) by subtracting their projection in Sotv). In order to have disagreement these data points should have non-zero projection on Sa otherwise there will be no disagreements. As a result the maximum number of data points is dim(projsed (S n (Suno n n.Shh))
B.3 Handling interference between concepts
In previous section, we explained why disagreement can happen between local and global model and how we can resolve the disagreements by querying user of the local concept. We bound number of disagreement with dimension of projection of Sumo on Stobe In previous section we did not need to change Sabn but when concept j has conflicts with concept we also add data to concept j (thus changing Sobr) which can lead to new conflicts with concept
15

---

Mode Voard
The Ni.t3
Houti farvcle
Hutil
Ho Diea
User soods concest lcorning the inttal local model
Uper seeds concegt & leaming the veite tcal motel
GPT 3 goncrates dora
GPT-3 generates dats
Uzdate glocal models
User labels intances where local and globa chagges
local and
Figure 5: CoAlign loop for operationalizing a concept. (a) The user starts by providing some seed data from the concept and their labels, they are then used to learn a local concept model. (b) GPT-3 then prompted to generate new examples, prioritizing examples where the local model disagrees with the global model. (c) Actual disagreements are shown to the user for labeling, and (d) each label improves either the local or the global model. The loop b-c-d is repeated until convergence, i.e. until as
the user has operationalized the concept and the global model has learned it. Each local model a proxy for the user and step b always check if global and local model are aligned with each other. Therefore, if user on the left changes the global model such that it causes interference with user on the right step b finds such interference and asks the right user to intervene and add more data.
The following proposition state that in addition to the dimension of projection of Symo on observed subspace we also need to calculate projection the unobserved space of different concepts as they might get added in the future. With notation of Sux denoting sum all the Sy and S_, denotes sum of all subspaces except i, the following proposition bounds number of times data to their concepts due to interference.
need to add
Proposition 2. If for all projsuno (0*) = then the maximum number of times that we need to handle interference is Σk -==1 dim [prois_. (S n (Stnoo n
(.Sobee S0kk )
Proof. The proof similar to Proposition Here we need to deal with conflicts with all other topics and since it is possible that we add their unobserved subspace well we need to compute the dimension of S.mo on the whole Sj subspace not only Sobb.
Let assume we added & example from concept to handle interference, we now prove that t < (S (S n (Sumo n S Sabe)t)). For every data point that we add we first remove Sokk components by removing its projection on Sabe. Now in order to have a conflict this data point should have non-zero projection on S_,- As a result the maximum number of data points we can add is less equal than dim (projs (S n (Suno n Sobr) summing over all the concept result in maximum number of interference that needs to be handled.
C Extra figures and tables
FigureBis showing the CoAlign loop for multiple users. Table s/hows the 6 concepts used in our experiment along with examples for each concept.
16

---

Concept
CheckList Examples
Roberta-Large accuracy
Synonyms in simple templates
How can
become more vocal?
61% %
How can
become more outspoken?
More Less antonym(X)
How can
become more optimistic?
How can
become less pessimistic?
X person not antonym(X (X) person
How can
become positive person?
14%
How can
become a person who is not negative?
Orders is irrelevant in symmetric relations
Are tigers heavier than insects?
0%%
What heavier, insects tigers?
Active / Passive swap
Docs Anna love Benjamin?
1.4%
Is Benjamin loved by Anna?
Modifiers changes question intent
Mark Wright a photographer?
22%
Is Mark Wright an accredited photographer?
Table 6: The concepts used for debugging Roberta-Large model fine-tuned QQP dataset.
17


Reviews:
[{"review_id": "EmzZ8ZltjJ", "reviewer": "Reviewer_11Xh", "summary": "The authors propose a framework for collaborative NLP development (CoDev) that enables multiple users to align a model with their beliefs.\nThe proposed work CoDev aids users in clarifying their concepts (an area humans often struggle with) and assists ML models to handle conflicts between concepts (an area ML often struggle with due to its inability to accommodate local updates).\n\nThe authors' main insight is learning a local model for each concept, and a global model to integrate the original data with all concepts. The authors then steer a large language model to generate instances within concept boundaries where local and global disagree. Their experiments show CoDev is effective at helping multiple users operationalize concepts and avoid interference for a variety of scenarios, tasks, and models.\n\n", "questions": "Above.", "limitations": "The limitations and societal impact are properly discussed.", "rating": 6, "confidence": 1, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is pretty easy to follow and propose a new framework that is useful for controlling LLMs to constrain the outputs according to user defined concepts.\n2. The idea of creating proxies from local models for user concepts is novel and interesting.\n3. The experiments with real world users is nice, as this work is all about collaborative development from users.\n4. There are some theoretical analysis in the proposed framework which is nice and provides additional insight.", "weaknesses": "1. I think more discussion and datasets could be used in terms of CoDev with biased seed data and CoDev with unlabeled data. Since the work is very central about the data distribution and checking for controlling models via user input, I think a more diverse type of dataset could be used, instead of just selecting positives and negatives from review dataset. It might provide better evidence in generalizability of such framework. \n2. Since AdaTest is a major related work, maybe some more comparison with it in the seeded datasets and unlabeled data would be nice?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your constructive feedback. \n\n\n- W1) We chose only positive and only negative cases to showcase that even in a very extreme case of bias in the seed data, CoDev can still generalize to the whole concept. We will add extra experiments with different distributions such as other segments of this dataset with less extreme biases.\n\n\n- W2) Adatest iteratively uses 7 examples from a concept to generate more examples, and thus it cannot work with unlabeled data.  As a result we cannot compare CoDev against Adatest when unlabeled data exists. We will make the difference between AdaTest and CoDev more clear in the camera-ready. We note that where we can, we do compare to AdaTest (first section of the experiments), both for finding bugs and fixing bugs without interference (Table 1 and line 225-229). \n\n\nWe hope these comments ameliorate the weakness listed enough to warrant a change in score.\n"}, {"review_id": "ArTUClwjQO", "reviewer": "Reviewer_MmNy", "summary": "The paper proposes a method based on data augmentation and instance selection for training a supervised model to be aligned with \"concepts\", where concepts dictate specific model behavior on certain inputs. In the proposed setup users illustrate a concept with a training example, this is followed by generating additional examples with GPT3, the acceptance of the generated examples is proportional to the disagreement between a global model trained on all the data and a local model trained to learn the concept perfectly. In experimental evaluations the proposed instance selection method seems to outperform reasonable other strategies for selecting data for aligning supervised models on specific concepts.", "questions": "- The idea of a concept seems a little vague; please consider discussing what kinds of user intents may be captured in the concepts. Perhaps with an eye toward realistic concepts that users want to see alignment on. As I see it, the concept seems like a rule for labeling specific input instances based on easily examinable features of the input.\n- Many of the ideas described in the paper (user-defined concepts, conflicts between then, training on user concepts) seem similar to the line of work on programmatic creation of datasets with weak labelers. Please cite and discuss similarities and differences to this line of work: https://scholar.google.com/citations?user=rfwwtFYAAAAJ&hl=en\n- The paper (through references of similarity) and especially the experimental setup (first two sub-parts) heavily depend on familiarity with prior work. Please describe the experimental setup and these prior works in greater detail in this paper.\n- This is perhaps a matter of style and framing, but I would recommend renaming the title of this paper. The presented work seems to be much more specific in its scope than the title seems to indicate. The presented approach relies on users labeling data (significantly devalued labor in ML) to align models with their preferred concepts, a large body of work may be framed as \"collaboratively developing NLP\" models by this naming logic. Further, the explored tasks are simplistic classification tasks; arguably, NLP encompasses a significantly broader range of tasks than these alone.", "limitations": "The paper discusses limitations adequately.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The proposed data selection method seems original, the presented method with an LLM data generator in the loop is under explored.\n- The problem of enforcing user desires on a supervised model is meaningful.\n", "weaknesses": "- The proposed method for aligning user \"concepts\" seems somewhat laborious or frustrating from a user-stand point, especially in cases where users concepts conflict with those of others. E.g. the paper notes, \"In practice, this means that a user adding a new concept needs to make sure it does not break any concepts from other users, a process similar to regression testing in software engineering.\" -- What happens in cases where the users concept cannot be incorporated without breaking somebody else's concept?\n- The experimental setup of the paper is hard to follow and is unlikely to be reproducible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your constructive feedback. \n\nW1) In this work we focused on post-training adjustment to enforce business rules, rectify undesired behavior, or align with user values.  A concept relates a set of inputs to desired behaviors, e.g. “religion does not connote sentiment”. For teaching a model about a concept we should create inputs as well as outputs. As you mentioned one way to do so is to write a program to automatically generate a dataset for a concept. However, concepts can be more complex and writing some programs cannot cover the whole input space. For example, Checklist (reference 14) is similar to your references of programmatic creation of data. In particular, Checklist creates a dataset with some rules for each concept. However as shown in the first part of experiment section (Table 5) these programmatic patches will miss a lot of subtleties for these concepts (i.e., after teaching a concept to a model we can simply run CoDev to find a lot of bugs that have not been covered). CoDev on the other hand utilizes GPT-3 to generate inputs that cover the concept more thoroughly. In other words, writing a program to generate inputs only covers a small part of the concept while random walk using GPT-3 as described in 3.1 can cover the majority of the concept space.\n\nFurthermore, as shown in the pilot study, labeling of inputs from a concept can be tricky and cannot be written with a simple program. For example, in the toxicity-islam case in table 4 the user starts with stating that “ISIS is violent is non-toxic” but “Muslims are violent is toxic” and then CoDev asks the user about sentences like “Taliban is violent”, or “Hezbollah is violent”, etc. Thus the user should interact with the model to make the labeling more clear, this cannot be done with simple programs. \n\nRegarding Interference, a user wants to add their concept without breaking the model on other concepts. The presence of possible conflicts is annoying, but that is not a function of our method, it's a function of the real world complexity and our method helps solve it. The alternative of not having explicit concepts and just labeling data would not work and would be more frustrating than handling interferences. For literal disagreement between users please see the general rebuttal. \n\nWe also like to note that our method making the concept alignment less laborious for the user by generating sentences using GPT-3 and only requiring user to label them, also the majority of alignment checks is happening between local and global model and user is only queried when there is a disagreement between local and global function (section 3.3).  \n\n\nW2) As mentioned in the general rebuttal we are going to release code and data for reproducibility. We will also make a significant improvement on the writing of the section. \n\nQ1) We addressed this question in (w1). Also as shown in the experiments concepts can be things like downward or upward monotonicity in NLI (a well-known problem in state of the art NLI models), or simpler concepts such as “X person = not antonym(X) person“ in QQP that are revealed by [14] that SOTA models performs really bad on them (table 5 in appendix) or even more complex concepts as depicted in the pilot study. \n\nQ2) Checklist (reference 14) is similar to programmatic creation of data. In particular, Checklist creates a dataset with some rules for each concept. However as shown in the first part of experiment section (Table 5) these programmatic patches will miss a lot of subtleties for these concepts. CoDev utilize GPT-3 to align the model with the concept completely and also handle interference between concepts. Thanks a lot for the references, we will make a more thorough comparison to programming creating datasets with weak labelers in camera-ready. \n\nQ3) Thanks a lot for the feedback, we will make the first two sections of the experiment setup more clear for the camera-ready. \n\nQ4) Thanks for the feedback we will make the title more specific. \n\nWe hope these comments ameliorate the weakness listed enough to warrant a change in score.\n"}, {"review_id": "NfbK6aEWvs", "reviewer": "Reviewer_q4QC", "summary": "This paper describes a multi-user collaborative model alignment framework that teaches certain desired concepts (behaviors, rules) to large language model (LLM).\nThe authors train a global model that intergrates the original data and all concepts, and a local model for each concept. The LLM is guided to generate examples where the local and the gloabl model disagree, and these examples are presented to users for annotation. Finally, these new annotations are used to update local and global models.\n\nThe proposed framework is evaluated on cases that teach an NLI model about downward- and upward-monotone concepts and a sentiment classifier about Amazon products reviews.\nResults show that it outperforms AdaTest and other baselines.\n", "questions": "Line 29: 'does not to lead to'", "limitations": "N/A", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* the targeting question is an important question and has attracted a lot of research interest. the proposed approach is easy to understand and experimental results show that it is effective, especially when there is interference between multiple concepts", "weaknesses": "* the experimental setup is not described in detail, it may not be easy for other researchers to reproduce the study\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback. As mentioned in the general rebuttal, we will release code and data in the camera ready, and we will make the experiment section more clear. Hopefully this information removes this weakness, leading to an improved score."}, {"review_id": "rkO2YsJjPy", "reviewer": "Reviewer_rcBm", "summary": "This paper proposes a new framework to debug NLP models. Specifically, while debugging a global model, it starts with training some local models on specific concepts. Then new data used for model improvement is labeled if the global model disagrees with the local model. \n\nExperiments indicate that the proposed approach (CoDev) outperforms the baseline approach (AdaTest) over multiple settings.", "questions": "1. Missing discussion about active learning in related work. \n2. (see weakness 3) How to extend the proposed framework to structure prediction tasks? Also, which NLP tasks do you think the proposed approach would fail? \n", "limitations": "Yes, this paper includes the limitation section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work focuses on an important but under-explored problem: debugging NLP models. The findings could potentially benefit the community.\n2. The paper is well-written, easy to understand the contents. ", "weaknesses": "1. There is a gap between theoretical guarantee and real experiments. Specifically the paper experiments with LLM, but provides proof based on linear regression.\n2. It's unclear if some of the results are statistically significant. For example, in Table 3 (about MNLI and sentiment analysis), the improvement seems marginal. \n3. All experiments seem to focus mainly on toy tasks like classification or NLI. The takeaway would be stronger if the paper has discussions about extending to more challenging NLP tasks such as structure prediction. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your constructive feedback.\n\n- W1) Unfortunately providing theoretical results for deep neural nets (and transformers in our case) is an open problem however, previous work (reference 10,11,12,13) consider overparameterized linear regression as a way to provide some insights on how these models might work. We followed this line of work and showed how our method works in overparameterized linear regression settings. The whole point of the theoretical section is to show that similar to our intuition, if learning the desired function in a local neighborhood requires a few samples then we can learn a concept in isolation with few queries (instead of asking users a lot of queries, e.g., look at fig2(c)). However adding other concepts can cause interference (e.g., see fig2(b)). Our theory bounds how many new queries we need to ask the user due to interference (as shown in proposition 1,2, number of queries can be a lot smaller than the dimension of inputs (d)). \n\n- W2) We ran each experiment in Table 3, 10 times and reported the average. We ran the t-test on the distributions although the improvement is small but the t-test shows statistical significance. We will provide the t-test results in camera-ready.  \n\n- W3) For non-classification tasks such as next word predictions we can assume that the local functions are a mapping from pair of (input, output) to a binary label of if the output is desired or not (e.g., (all muslims are, murderer -> non-desired) but (all muslims are, going) -> desired). Therefore, the local functions can still be binary and simple while the global function can be complex (e..g, generation instead of classification). Thank you so much for this comment and we will add a discussion on how to extend this work to more sophisticated non-classification tasks. We also like to note that we have experiments on multi-class classifications in our experiments, not just binary. \n\n---\n\n- Q1) We appreciate your comment and will add an extra section on comparing with active learning methods. In summary, CoDev learns each concept individually in isolation and only query inputs where global and local disagree. Thus it does not need to query part of space that local and global overlap (i.e., agree). Also learning local concepts in isolation allows CoDev to better learn the local concept in comparison to active learning methods that only adds data to the global model and might only memorize the local concept data. \n\n- Q2) We explained how to extend this work to more complex scenarios in (W3). This approach fails for a user that considers a very general concept that cannot be divided into subparts.  \n\nWe hope these comments address the listed weaknesses enough to warrant a change in score.\n"}, {"review_id": "nzVoUJvtr9", "reviewer": "Reviewer_g41d", "summary": "This paper frames operationalizing concept as a solution to enumerate all possible concepts and developed a corresponding framework CoDev, that starts by collecting text and labels from users, then have GPT-3 generates text and labels. When there are disagreement among users and GPT-3, users would be asked to relabel the instance until convergence. The authors conducted experiments using Amazon review dataset and MNLI to compare CoDev with baseline models. Also the author conduct a pilot study with 4 people using CoDev from providing data to improve model alignment and more complex output after 5-7 iterations. ", "questions": "1. There is no clear definition for your local model and global model other than the figure 1. Could the authors please clarify why and how they use GPT 3 as a global model? What prompts are used?\n\n2. It is not clear whether the d step (repeating labeling) in the Figure 1 is involved with a single user during the iteration or multiple users. If multiple users, how did the authors handle the disagreement among users? If that's different user for different round, this need to be clarified. If single user,  how could the CoDev system enable multi-user interaction?\n\n3. For the pilot study, it is not clear how the authors collect the seed data. Are those labels given by single user for each instance? If each instance collect labels from multiple users, how does the authors aggregate the labels? \n\n4. For the pilot study, it is is not clear how the authors evaluate their study. If better alignment is the evaluation metric, how do the authors evaluate the quality of alignment? In the line 289 and 290, the authors said 'As the user made repeated adjustments to both models, the disagreements gradually centered around the concept boundaries, reaching a point where users could no longer determine the correct behavior.' It is not clear why users could no longer determine the correct behavior is the standard for good alignment. Shouldn't users always have the right and ability to determine what they think is correct? \n\n5. In Table 4, why is there only labels for the 'Seed data' column but the 'Initial rounds' and 'Final rounds' don't have corresponding labels? Are the Initial rounds and final rounds evaluate alignment based on the labels? And the Table 4 caption mainly highlight the text in final rounds become much complex than the initial rounds. Why is this important? Is this an evaluation criteria for the pilot study? ", "limitations": "The authors discussed the limitation of solving literal disagreement since they only handled interference caused my machine learning shortcoming. \n\n", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper points out an important questions that NLP model align with multi-users values. And it is novel to use GPT-3 as the global model and Roberta-large trained on users input as the local model, and try to converge the disagreement from two models through collecting user labels iteratively.  ", "weaknesses": "1. Given the paper, that the authors handle disagreement by repeating collecting user labels until convergence, assumes that there exist a single ground truth for each instance. But many studies have shown that a single instance can have multiple acceptable answers [1, 2]. The study design also conflict with the study goal of aligning multi-users' value. Disagreement is not a sign of error but could be a sign of multiple possibility. How could you determine the disagreement between the local model and the global model is not acceptable?  \n\n[1]Wan, R., Kim, J., & Kang, D. (2023). Everyone’s Voice Matters: Quantifying Annotation Disagreement Using Demographic Information. Proceedings of the AAAI Conference on Artificial Intelligence, 37(12), 14523-14530. https://doi.org/10.1609/aaai.v37i12.26698\n[2]Aida Mostafazadeh Davani, Mark Díaz, Vinodkumar Prabhakaran; Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. Transactions of the Association for Computational Linguistics 2022; 10 92–110. doi: https://doi.org/10.1162/tacl_a_00449\n\n2. The authors mentioned the shortcoming of having expert label datasets in the introduction. But actually many open sourced dataset for machine learning studies are labeled by crowd workers from third party platforms such as Amazon MTurk. And the determination for sentiment or toxicity is a very subjective task, that is there is no expertise to judge whether others feels the text is toxic or not. Therefore, the research gap mentioned in the introduction and the dataset/task picked for the experiment make author's framework and argument weak. \n\n3. For table 2, it is not clear how SB is different from biased SB. And it is also not clear how Data collection method is different from base line BERT model and CoDev model.  And many clarity concerns as listed in the following Question section. These confusions make the paper less convincing.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your comments and constructive feedback.\n\nW1) As mentioned in the general rebuttal we agree that literal disagreements and multiple possibilities is a very important question but even in the absence of literal disagreements, the problem of ML interference still remains and that is the main focus of this paper. We appreciate your comment and references and we will add an extra discussion section to address this issue. \n\nW2) We agree that many open sourced dataset are labeled by Amazon turk however, we are focusing on post-training adjustment. For example in the NLI task, the dataset is collected through third party labeling but further studies showed that it has low performance in concepts such as downward and upward monotonicity. In response, some papers collected data to address the low performance in these concepts but it can easily be shown that the whole concept is not covered. \n\nFurthermore, collecting data and labeling is very challenging for niche concepts that collecting data is hard. Our work uses LLMs to address such issues. In other words, we use LLMs as an infinite pool of unlabeled data and CoDev helps users to find and label data points in this pool that belong to their concept. \n\nWe totally agree with you that tasks such as sentiment and toxicity are very subjective tasks, and different users might have different opinions; the goal of pilot study on such tasks was to show that even a single user needs assistance to operationalize their concept (more details are below in q3-5).\n\nW3) We apologize for the ambiguity of text for this section, we will improve the writing in camera-ready. In summary, as stated CoDev uses LLM to generate data for a concept. To start the process users need to provide some seed data, the point of this experiment is to show that even in the very extreme case of biased seed data, CoDev can still generalize and cover the concept. \n- SB here is all reviews containing the word “skin” or “battery” \n- biased-SB contains reviews that are positive and have word “skin” in them; or that are negative and have word battery in them. \n- Base model is a model that has low performance in reviews containing skin and battery and our goal is to improve the performance of this model\n- Normal data collection by definition only adds data that are biased (i.e., reviews that are positive and have word “skin” in them; or that are negative and have word battery in them). As a result, data collection leads to very high performance in biased-SB but low performance in SB.\n- CoDev on the other hand improves performance in both SB and biased-SB; even though it started from biased seed data. \n\n---\n\nQ1) Both local and global models are classifiers (roberta-large in our experiments). GPT-3 is used for generating sentences within a concept. Lines 94-96 explain the process. In particular, our goal is to find sentences that belong to a concept. In order to do so we iteratively use m examples from the concept as a prompt to GPT-3. In response GPT-3 generates more examples within the concept. We apologize for the confusion and will make the definition more clear. \n\nQ2) For simplicity, Figure 1 is only showing the steps in CoDev for a single user. In section 3.4 we explain interference and have we handle it. We will add a new figure with multiple users in the appendix for clarity. \n\nQ3) For each row, the seed data is written by a single user in his/her own concept. \n\n\nQ 4-5) This pilot study is a small part of our experiment section, our goal for conducting this small pilot study was to show:\n- Humans need assistance to make their concept clear. For example in toxicity - islam case, the user thought they have a good grasp of the concept in their mind. However when the user states that “ISIS is violent” is non-toxic but “muslims are violent” are toxic; CoDev asked the user about Taliban, Hezbollah, radical Islam, etc. and as shown in Table 4 this back and force resulted in sentences that user has not think about such edge cases beforehand and could not come up with a label (i.e., require more thinking to make their concept clear). \n- As users continue working with CoDev, the number of disagreements between local model and global model decreases and gets limited to sentences that users might not count as clear misalignment to their concept. \nFor more thorough analysis of CoDev please refer to other parts of our experiment section. \n\nWe believe that helping users to operationalize their concepts and handling ML interference are important challenges in alignment and we hope these comments address the listed weaknesses enough and you consider changing the score.\n"}]

Author rebuttal:
We like to thank the reviewers for their constructive feedback, and stating that the problem we are considering is “important” and “beneficial to the community”, also stating that our “approach is novel”, our “theory provides some insights”, the paper is “well written” and “experiments show the effectiveness the method”. Following we are mentioning some of the concerns shared by the reviewers.For other concerns please see individual responses. 

---
## Literal interference between users: 
---
Reviewers 2 and 5 have pointed out that our work does not address literal interference, where two users assign different labels to the same sentence. We acknowledged this limitation in the paper and highlighted it as an avenue for future research. We will make this more clear in the revision, and note that the interference we focused on still remains as an important problem even if there is no disagreement between users.

To clarify, there are two distinct types of interference when aligning a model to multiple users (or training a model on different distributions):
1. Literal interference: where users disagree in labeling the same sentence.
2. ML interference: interference arising from over-generalization in machine learning.


Both challenges are critical, yet they address different facets of the problem. Our paper particularly addresses the latter, ML interference. Previous research [1,2,3] has shown that any local change in ML models can interfere with other parts of the model (e.g., comparing fig3(a) and fig3(e) you can see that a change inside the concept caused a change outside the concept). We want our model to not overfit (i.e., only memorize the training data) and generalize to unseen data; however this generalization is very dependent on other concepts and the previous data. Take, for instance, a world where only bananas are yellow. An ML model might be guided by a user to recognize bananas solely based on their yellow color. Now if a new user introduces another yellow object, like corn, the model must discern other distinguishing features. Merely combining training data for two such concepts doesn't suffice (as shown in [2]); the boundaries must be distinguished. Our work outlines methods to mitigate this type of interference.


Finally we like to note that while we haven't introduced a specific mechanism to resolve literal interference, our method can surface such interference. This can pave the way for resolution through discussions, voting, or even tweaking the model to reflect multiple perspectives, especially in cases where, as R5 noted, there isn't a consensus among users.

We apologize for the ambiguities in text. In the new version we delineate both types of interference, emphasizing the intricacies of ML interference. We will also add an experiment to surface disagreements between two local models (i.e., proxies for humans) to showcase the effectiveness of our method to surface disagreements. We again acknowledge that handling literal disagreement is very important but out of scope of this work and great future direction.

---

## Some ambiguity in the experiment section and reproducibility of results
---
Reviewers have mentioned that some parts of the experiments are hard to follow and might not be reproducible. We are releasing the full code and data in camera ready, so the experiments can be easily reproduced. Even though some experiments have humans in the loop and thus inherent variance, the gap between AdaTest and Codev (Table 2 and line 226) is large enough that this should not matter.

Regarding the ambiguity we will make a huge improvement on writing and apply all your feedback in text. In summary the following are the message of the four main section of our experiments:

- CoDev works better than AdaTest by finding more bugs and causing no interference (in comparison to AdaTest that causes interference). 
- CoDev works even when the seed data is biased
- CoDev sampling mechanism outperforms random or uncertainty sampling
- A very small pilot study to show humans need assistance to operationalize their concept (i.e., we showed that they might not even know the exact boundaries of their concept beforehand). 


---
[1] Khani, Fereshte, and Percy Liang. "Removing spurious features can hurt accuracy and affect groups disproportionately." Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. 2021.

[2] Raghunathan, Aditi, et al. "Adversarial training can hurt generalization." arXiv preprint arXiv:1906.06032 (2019).

[3] Srivastava, Megha, et al. "An empirical analysis of backward compatibility in machine learning systems." Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020.

Meta-review: This paper presents a framework for multiple users to interact with a large language model, such that users can operationalize "concepts" and try to get the LLM to align to their concepts. This is done with the structure of having a global model of all concepts, as well as a local model for each of the concepts.

This problem is interesting, and the approach seems sound and well-carried out. There were several promises of substantial revision during the author-reviewer discussion. I strongly urge the authors to follow through on these revisions.
Decision: Accept (poster)