Title: Collaborative Alignment of NLP Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: alignment

collaborative alignment

debugging

nlp

interference

multi-user interaction
Abstract: Despite substantial advancements, Natural Language Processing (NLP) models often require post-training adjustments to enforce business rules, rectify undesired behavior, and align with user values.  
These adjustments involve operationalizing "concepts"—dictating desired model responses to certain inputs. 
However, it's difficult for a single entity to enumerate and define all possible concepts, indicating a need for a multi-user, collaborative model alignment framework. 
Moreover, the exhaustive delineation of a concept is challenging, and an improper approach can create shortcuts or interfere with original data or other concepts.

To address these challenges, we introduce CoAlign, a framework that enables multi-user interaction with the model, thereby mitigating individual limitations. 
CoAlign aids users in operationalizing their concepts using Large Language Models, and relying on the principle that NLP models exhibit simpler behaviors in local regions. 
Our main insight is learning a \emph{local} model for each concept, and a \emph{global} model to integrate the original data with all concepts.
We then steer a large language model to generate instances within concept boundaries where local and global disagree.
Our experiments show CoAlign is effective at helping multiple users operationalize concepts and avoid interference for a variety of scenarios, tasks, and models.

Collaborative Alignment of NLP models

Abstract

Despite substantial advancements, Natural Language Processing (NLP) models often require post-training adjustments to enforce business rules, rectify undesired behavior, and align with user values. These adjustments involve operationalizing "concepts"-dictating desired model responses to certain inputs. However, it's difficult for a single entity to enumerate and define all possible concepts, indicating a need for a multi-user, collaborative model alignment framework. Moreover, the exhaustive delineation of a concept is challenging, and an improper approach can create shortcuts or interfere with original data or other concepts.

To address these challenges, we introduce CoAlign, a framework that enables multiuser interaction with the model, thereby mitigating individual limitations. CoAlign aids users in operationalizing their concepts using Large Language Models, and relying on the principle that NLP models exhibit simpler behaviors in local regions. Our main insight is learning a local model for each concept, and a global model to integrate the original data with all concepts. We then steer a large language model to generate instances within concept boundaries where local and global disagree. Our experiments show CoAlign is effective at helping multiple users operationalize concepts and avoid interference for a variety of scenarios, tasks, and models.

1 Introduction

NLP models have showcased remarkable capabilities, yet they are not exempt from imperfections. Unacceptable values embedded in their training data, persistent errors, or violations of business rules highlight the need to teach certain concepts to these models. A concept relates a set of inputs to desired behaviors, e.g. in sentiment analysis, a concept may dictate that 'religion does not connote sentiment' (e.g., 'I'm Muslim' is Neutral). Similarly, in natural language inference (NLI), the broader concept of 'downward monotonicity' specifies entailment relations when certain parts of expressions are made more specific (e.g., 'All cats like tuna' entails 'All small cats like tuna').

The standard way of teaching concepts to models is adding new training data that exemplifies the concept, e.g. adding neutral sentences containing religious words [1], or adding entailment pairs that illustrate downward monotonicity [2]. However, it is hard to ensure that the data provided does not lead to shortcuts , i.e. spurious correlations or heuristics that allow models to make predictions without capturing the true underlying concept, such as 'all sentences with religious terms are neutral', or 'going from general to specific leads to entailment'. Further, the model may overfit - fail to generalize from the instances provided to the actual concept, e.g. only recognizing pairs in the form ('all X...', 'all ADJECTIVE X...'), and not pairs like ('all animals...', 'all cats...'). Finally, both shortcuts and overfitting can lead to interference with the original data or other concepts, e.g. leading to failures on 'I love Islam' or pairs like ('Some cats like tuna', 'Some small cats like tuna'). In sum, operationalizing concepts is challenging, since users typically cannot think of all concept boundaries and interactions in advance.

a

Global

Local

1prayin mosque  Neutral

I'm Muslim

Neutral

b

The protagonist was Musllim

Neutral

d

User labels

e

Positive

Neutral

instanceswhere

Training the initial

1 fast in Ramadan

Neutral

Neutral

disagree

localandglobal

Update local and

local model

1 hate Muslims

Negative

Negative

global models

Userseedsconcept

GPT-3generates data and finds disagreements

RepeatUntil Convergence

One possible solution is to ask domain experts to create data that covers the concept as comprehensively and precisely as possible, e.g. the GLUE diagnostics dataset [3] or the FraCaS test suite [4]. However, these datasets are often expensive to create, small (and thus not suitable for training), and not exhaustive, as even experts still fail to think about all aspects and nuances of a concept [5]. Another solution is to have users provide data incrementally while they receive feedback from the model, e.g. adversarial training [6] or adaptive testing [5]. These do not require users to think about everything in advance, and can expose and correct model weaknesses. However, adversarial training does not include the notion of concepts explicitly, and adaptive testing does not address the interaction between different concepts or between a concept and the original data. As a result, users may not explore concept boundaries efficiently, and thus do not know when they have sufficiently covered a concept or whether they have introduced interference with other concepts.

In this paper, we introduce Collaborative Alignment of NLP Models (CoAlign). CoAlign leverages the collective knowledge of multiple users to cover many concepts instead of relying on a single user. CoAlign allows users to collaborate with AI systems and other users in specifying concepts and teaching them to models. Our key insight is making concepts explicit by learning a local model for each concept, in addition to a global model that integrates the original data and all additional concepts. When operationalizing a new concept, we rely on a large language model (LLM) to generate new instances where the local and global model disagree, and ask users to label such examples (Figure ?? ). Intuitively, these examples are either cases where the local model is not yet fully specified, or where the global model still makes errors on the concept (due to overfitting or shortcut reliance). As users label these examples, both models are updated until convergence, i.e. until the concept has been learned in a way that does not conflict with prior data or prior concepts. In a sense, each local model is ever-improving cheap expert in its respective concept. The speed of prediction of local models and diversity of examples generated by the LLM enable users to explore the boundaries between concepts and existing data, an exploration that could be challenging for users to do without aid.

Our experimental results demonstrate the effectiveness of CoAlign in operationalizing concepts and handling interference. We first show CoAlign outperforms AdaTest [5], a SOTA tool for debugging NLP models that also uses GPT-3 by revealing more bugs and fixing them without interference. We then demonstrate that CoAlign operatinalize concepts even when user starts with biased data, outperforming a model that relies solely on data collection. We compare the data selection mechanism of CoAlign to random selection and uncertainty sampling by running simplified version of CoAlign where instead of using GPT-3 we iteratively select examples from an unlabeled pool. We show CoAlign outperforms both baselines in teaching an NLI model about downward- and upwardmonotone concepts [3], as well as teaching a sentiment analysis model about Amazon products reviews. Finally, in a pilot study, we demonstrated that CoAlign helped users clarify their concepts.

2 Setup

Let x be a text string, and y a categorical label, e.g. sentiment (positive, negative, neutral). We assume there is a true function f ( x ) = y that we aim to approximate with a model ˆ f ( x ) . Assume

we have access to a 'base' dataset D 0 = { ( x 1 , y 1 ) , . . . , ( x n , y n ) } , e.g. of movie reviews, from base distribution P 0 . We refer to the model trained on D 0 as the base model ˆ f 0 .

A concept C i is associated with a distribution P i over the input space, e.g. a concept might place probability mass exclusively on sentences containing religious words. We say x ∈ C i if P i ( x ) > 0 . Since it's hard for users to be exhaustive, we do not assume users can generate from P i , but that they can label any x with f ( x ) , and as to whether it is in C i or not. We further assume users can provide a very small number samples in the support of P i .

̸

Without loss of generality, we assume that we have k users developing a model collaboratively, each with their own concept. Our goal is to train a model ˆ f that does well on both the base distribution and all concepts, i.e., minimizing 1 k +1 ∑ k i =0 E x ∼ P i [ ˆ f ( x ) = f ( x )] .

3 Collaborative Alignment of NLP models

In this section we describe how a single user operationalizes their concept by producing a dataset D i in the context of an existing global model ˆ f trained on the base dataset D 0 and previous concepts D 1: i -1 . If ˆ f 0 is already 'aligned' within the concept, i.e. ˆ f 0 ( x ) = f ( x ) for all x ∈ C i , we would be done and there would be no need for D i . Thus, what we really want is for D i to specify the boundary around 'failures', cases not currently handled by ˆ f 0 . We abstract away the choice model and learning procedure, assuming that the model can learn the concept given D i (in practice, we use neural networks that are expressive enough).

3.1 Sampling from the concept

Since by assumption we cannot sample from the concept distribution P i , it is a challenge to find regions of P i where ˆ f 0 fails. To address this, we use GPT-3 [7] as a generator G to simulate a random walk within the concept. To do so, we construct a prompt with m in-concept examples, and use this prompt as input to G to generate more samples. Then, we ask the user in the loop to accept or reject each generated sample x ′ ( x ′ is accepted if x ′ ∈ C i ), and also to label the accepted x ′ with f ( x ′ ) . The value of m controls the tradeoff between precision and recall, with high m generating more in-concept examples and low m exploring the space more broadly.

Under some conditions it can be shown that G simulates a Markov chain with stationary distribution P i (Appendix A), but the weaker condition of connectivity suffices for finding the concept failures, i.e. there must be a path between any x ′ , x ′′ ∈ C i with nonzero transition probabilities according to G and the prompt. That is, if the concept is connected, with enough time we should be able to find regions of P i that are not already learned by ˆ f 0 .

While sampling from G eventually leads to the yet-unlearned concept regions, it is an inefficient use of human effort, as it does not use the user labels to guide generation (i.e. the user has to label many examples that the current model already handles correctly). A better approach would be ask the user to label in a way that maximizes the expected information gain for the concept, to which we now turn.

3.2 Local Concept Models

Complex functions can be approximated by simpler functions in a local neighborhood, as evidenced by theoretical results (e.g. Taylor expansion) and empirical applications [8, 9]. Since a concept is a natural local neighborhood, we use this insight and learn a local model ˆ f i to approximate f ( x ) in C i .

We present a toy example for intuition in Figure 2, where we show toy distributions P 0 (Figure 2a) and P 0 with an additional concept P 1 (Figure 2b). In 2b, ˆ f 0 learned on samples from P 0 (dashed line) predicts most of P 1 correctly, except for a small region in the bottom left quadrant. However, we would need many random samples from P 1 in order to learn that region, and reach the best model ˆ f (solid line in Figure 2b). In contrast, we can learn a good local model ˆ f 1 for P 1 with a trivial number of samples (Figure 2c). This local model can be used to produce a disagreement region between ˆ f 1 and ˆ f 0 (Figure 2d), and sampling from that region would lead to failure discovery much faster.

(a) The initial data ( P 0 ) is elliptical Gaussian

(c) local function for the user data

̸

More generally, we define a score function as the disagreement between the local and global function. This score function is used to steer generation such as to maximize the score of generated samples x ′ , by adding instances to the prompt for G with probability proportional to their score (similar to Ribeiro and Lundberg [5], who use a different score function). We note that models may present false agreement on some samples, i.e. ˆ f i ( x ′ ) = ˆ f 0 ( x ′ ) = f ( x ′ ) . To account for this, we also sample from the agreement region sometimes, with a probability that decays over time as we gain more confidence in the local model.

3.3 Operationalizing a concept: from disagreement to convergence

The local and global models disagree on regions where the concept has not yet been learned (global is wrong) or the local model does not fit the user's concept correctly (local is wrong). Thus, every label from the disagreement region results in an improvement in whichever model is incorrect. As labeling progresses, we update both the concept model ˆ f i and the global model ˆ f until we reach convergence (i.e. ˆ f has learned the concept C i ). Note that while ˆ f i is trained on data from P i , ˆ f is trained to do well on the base distribution P 0 , as well as on all concepts P 1: k .

(a) seed

(b) local fit

(c) generation

(d) selection

(e) update

We present pseudo-code for operationalizing a concept in Algorithm 1, and illustrate it in Figure 3. The local and global models are initialized in lines 1-2 (Figure 3a and b). In line 5 (Figure 3c), G is prompted such as to generate in-concept samples that are likely to be in the disagreement region. Then, in line 6 (Figure 3d), we select a batch of b examples for labeling according to the disagreement score of the generated instances. This generation-labeling loop is repeated for L iterations, after which both the local and global models get updated in lines 7-8 (Figure 3e). For the sake of interactivity, in practice we do not train from scratch when updating the models, and instead finetune them from a previous checkpoint. After a few iterations, ˆ f and ˆ f i converge (i.e. the user has operationalized the concept and the global model has learned it). If the number of generated samples b ∗ L is large

enough, we can assume that no disagreement on generated sample between ˆ f and ˆ f i means they have converged, and thus we stop and output the generated data D i (line 9).

3.4 Handling interference between concepts

When aligning a model to multiple users (or training a model on different distributions), two types of conflicts can happen: (1) Literal disagreements: where two users disagree in labeling the same input. (2) Interference: where over-generalization in one concept interfere with another concept (e.g., if a user adds a lot of neutral examples for the 'religion does not connote sentiment' concept, then training the model on these new examples can cause the model to predict neutral for sentences like "I love Islam" which interfere with a concept that says 'love X' is positive). As stated in Section 2, we assume there is a single true function f (thus no literal disagreement) and we only focus on handling interference. 2 Handling interference is crucial since any local change in ML models can interfere with other parts of the model [10-12].

Having local functions (cheap experts) enable us to check interference efficiently. Every time that a user operationalizes a concept according to Algorithm 1, we check the resulting global model ˆ f against the local models for all previous concepts. To do so, we re-run Algorithm 1, only asking the user for labels if there is a newfound disagreement region. In practice, this means that a user adding a new concept needs to make sure it does not break any concepts from other users, a process similar to regression testing in software engineering.

̸

While we want to handle interference, having multiplicity of concepts can be beneficial in refining a concept. In particular, we want the global model to not overfit to a concept (i.e., only memorize the training data) and generalize well; however generalizing well is dependent on other concepts and the previous data. Take, for instance, a world where only bananas are yellow. A user might guide an ML model to recognize bananas solely based on their yellow color. Now if a new user introduces another yellow object, like corn, the model must discern other distinguishing features. Merely combining training data for two such concepts doesn't suffice (as shown in [11]); the boundaries must be distinguished. Furthermore, interference can be beneficial by exposing false agreement regions between the global model and any individual concept model. In other words, while both ˆ f and ˆ f i may be relying on the same shortcut to solve some regions of P i , it is unlikely that this shortcut does not cause interference between ˆ f and all local models ˆ f j for j = i . In this case, the interference caused by adding C j is actually beneficial, as it further refines concept C i .

In practice the original dataset ( D 0 ) is often very large and we cannot fine-tune the model on D 0: k . Instead of choosing the whole D 0 , every time we sample data points with highest disagreement between ˆ f 0 and ˆ f from D 0 . In other word, we treat ˆ f 0 as a user with concept distribution P 0 , this enable us to deal with interference with original model as well (as an example in Figure 2 we might only choose the data points from elliptical Gaussian in disagreement region).

4 Linear regression analysis

We now study CoAlign in a noiseless overparametrized linear regression scenario. This particular setup is used in recent literature to gain some insights into the behaviors of deep networks [10, 11, 13, 14]. For more details and proofs see Appendix B.

Setup. Let x ∈ R d where only a subset the data points are valid. We assume θ ⋆ ∈ R d such that y = θ ⋆ ⊤ x . Let S i denote the smallest subspace containing all valid data points of C i . Given k examples in C i , let S obv i denote the subspace observed by the training data, S uno i denote the unobserved subspace (thus S i = S obv i + S uno i ), and S inv i is the subspace that concept i does not have any variation in it. We consider overparametrized noiseless linear regression, where the number of features ( d ) exceeds the number of training data, enabling us to always interpolate all training data. We assume local and global models infer the min L 2 norm interpolant.

As a running example, let x ∈ R 3 and consider a concept where data points belonging to that concept satisfies x 1 = x 2 . Let's assume we observed x = [1 , 1 , 0] with label y = 2 . In this case we have: S obv 1 = { [1 , 1 , 0] } , S uno 1 = { [0 , 0 , 1] } and S inv 1 = { [1 , -1 , 0] } . The min-norm solution interpolating the concept is ˆ θ i = [1 , 1 , 0] .

Operationalizing a concept. The min-norm solution can also be constructed by introducing constraints that projection of ˆ θ i on S uno i and S inv i has to be zero. In the above example, the minnorm solution is the unique answer of the following linear equations ( [0 , 0 , 1] θ = 0 , [1 , -1 , 0] θ = 0 , [1 , 1 , 0] θ = 2 ). When we teach the global model about a concept, the naive combination of data points could violate these constraints, leading to disagreements between local and global models. For our running example, suppose the original data point is x = [0 , 1 , 1] , y = 2 . Combining the concept data point with the original data point for inferring global model results in ˆ θ global = [ 2 3 , 4 3 , 2 3 ] , which leads to a disagreement between local and global model for data points varying in the [0 , 0 , 1] direction.

In case of disagreement, two scenarios could arise: (1) the direction is non-zero and the local model needs more specification (happens a lot at the beginning), or (2) the direction is zero, but an explicit constraint is needed to prevent the global model from assuming other values. In our running example, the generator can help us to find a data point that the two model disagree 3 . Let's assume the data point is x = [0 , 0 , 1] where local model predicts 0 but global model predicts 2 3 . We show this data point to the user and either the user confirms [0 , 0 , 1] as non-zero or makes the zero explicit, resulting in a global prediction of ˆ θ global = [0 , 2 , 0] .

Once we learn the local concept (i.e., all unobserved directions are indeed zero), how many of S uno i directions need to be added as explicit constraints? Intuitively, we do not need to query user for any direction in S uno i that has already been observed in S obv 0 or are orthogonal S obv 0 since there is no interference. The following proposition indicates the maximum number of disagreements for teaching global model about the local concept.

Proposition 1. If proj S uno i ( θ ⋆ ) = 0 , then the maximum number of disagreement between local and global models is dim(proj S obv 0 ( S uno i ∩ ( S uno i ∩ S obv 0 ) ⊥ )) .

Handling interference between concepts. During the operationalization of concept i , we maintain S obv 0 unchanged. However, in handling interference (Section 3.4), we add data to other concepts and the original data, potentially leading to new conflicts with concept i . Therefore, in addition to considering the projection of S uno i on observed subspaces we need to consider the unobserved subspaces as well. Similar to above, if an unobserved direction in one concept has been observed in another concept we do not need to query user. With notation of S obv 0: k denoting sum of all the S obv i , and S -i denotes sum of all subspaces except i , the following proposition bounds number of times users need to add data to their concepts due to interference.

Proposition 2. If for all i , proj S uno i ( θ ⋆ ) = 0 then the maximum number of times that we need to handle interference is ∑ k i =1 dim ( proj S -i ( S uno i ∩ ( S uno i ∩ S obv 0: k ) ⊥ ) ) .

Concept

Examples

Examples of bugs found by CoAlign

X person = not antonym (X) person

How can I become a positive person?

How can I become a person who is not negative?

predicts duplicate

shortcut bugs

{

How can I become a mysterious person?

How can I become someone with no mystery?

predicts non-duplicate

overfit bugs

{

How can I become a blind person?

How can I become someone who has lost his (physical) vision?

Modifiers changes question intent

Is Mark Wright an accredited photographer?

predicts not-duplicate

shortcut bugs

{

Is he an artist?

Is he an artist among other people?

predicts duplicate

overfit bugs

{

Is Joe Bennett a famous American court case?

Table 2: Comparison of CoAlign and AdaTest in terms of handling interference. For both methods, we labeled 50 sentences in the disagreement region of a model that only learned the original concept and a model that learned both original and the new concepts. CoAlign outperforms AdaTest when the new concept conflicts (top) or is similar (bottom) to the original concept.

5 Experiments

We first show CoAlign outperforms AdaTest [5], a leading NLP debugging tool that also uses GPT3, by revealing more bugs and resolving them without interference. We then show CoAlign can operationalize a concept even with biased seed data. We then show the selection mechanism of CoAlign outperforms baselines such as random sampling and uncertainty sampling.We conclude with a small pilot study of using CoAlign. 4

CoAlign outperforms AdaTest[5]. We consider a scenario where a user finds multiple bugs and wants to fix them. Following Ribeiro and Lundberg [5], we use Quora Question Pairs (QQP) dataset where the goal is to predict if two questions are duplicate or not. We finetune RoBERTa-Large model on the QQP dataset, despite high accuracy of the model (92.2%), Ribeiro et al. [15] identified multiple concepts where the model has low performance. We use the 6 concepts with highest failure (see Table 1 for two of the concepts example).

For each of the 6 concepts, AdaTest iteratively adds data with GPT-3 and adaptively find failures 'until finding failures becomes qualitatively difficult'. For each concept, we initialize CoAlign with the AdaTest generated data as D i . Even though the model has been 'debugged' with AdaTest, CoAlign quickly reveals ≈ 5 semantically meaningful sub-categories of bugs for each concept (with many failures within each sub-category). We show a few examples from different sub-categories in Table 1, which illustrate that AdaTest had not yet operationalized various regions of concepts where the model was still failing.

We now compare CoAlign with AdaTest in terms of handling interference. To do so, we pick pairs of concepts C orig , C new that might cause interference with one another, but that were debugged and reported as 'fixed' by Ribeiro and Lundberg [5]. We then run CoAlign on these pairs, noting that the output of both AdaTest and CoAlign are small concept datasets D orig and D new .

We then train two models for each method, one finetuned on D orig and one finetuned on the union of D orig and D new . Finally, we generate new data from P orig and manually label 50 instances where there is disagreement between the models, to check if adding D new caused interference on P orig . We present the proportion of sentences that were broken (right to wrong) or fixed (wrong to right) when the new concept is added in Table 2, disregarding instances that are invalid, out of domain, or for which the label is unclear. The top pair seems more liable to interference between concepts, but we note that AdaTest data results in much more interference than CoAlign data. In the bottom pair,

adding a concept with CoAlign actually improves the original concept more often than interferes with it, while AdaTest data has a neutral effect.

Table 3: When we have access to a biased dataset (biasedSB) with consistently negative skin reviews and positive battery reviews. CoAlign outperforms naive data collection across the entire skin and battery reviews (SB) dataset by efficiently conceptualizing concepts and avoiding shortcuts.

CoAlign works even with biased seed data. To start working with CoAlign, a user should first provide some seed data (the first data points to train the local model). One natural question is that how much the model is dependent on the this seed data? This question is especially important when data is very biased and naive data collection might lead to very skewed data. Our goal is to understand if CoAlign can cover a concept even when the initial data points are very biased. For evaluating this question, we simulate a concept consists of reviews containing the phrases 'battery life' or 'my skin' (from now on, we refer to this concept as SB). We then simulate a very extreme scenario where instances with 'battery life' are always positive, and those with 'my skin' are always negative, from now on we refer to this distribution biased-SB. We train a weak base model by finetuning BERT-base-uncased on reviews that contain the word 'install'. Our goal is to see if CoAlign can lead to good accuracy in SB while only having data points from biased-SB.

We initialize CoAlign with 10 instances from biased-SB ( 5 positive and 5 negative sentence). We then run CoAlign for 5 rounds adding 10 data points in each round. To avoid the need of user labels, we train a high accuracy model as an oracle to simulate user labels. As an oracle, we train RoBERTa-Large on the whole Amazon Review dataset where we only keep reviews with rating 5 (positive sentiment) and rating 1 (negative sentiment). The accuracy of oracle on validation dataset is 98.6%. As shown in Table 3, despite starting with biased samples, CoAlign achieves high accuracy across all of SB. Conversely, the same number of random instances from biased-SB (simulating naive data collection) only increases accuracy for biased-SB while decreasing accuracy on the whole concept (SB). Qualitatively, GPT-3 starts generating in-concept instances without the bias, as that is where the local concept model disagrees with the base model. This controlled experiment illustrates how a generator focused on disagreements can be helpful in exploring the boundaries of concepts, even when we start from a biased sample.

CoAlign outperforms random and uncertainty sampling. We adapt CoAlign so that it selects instances from a pool of unlabeled data based on the disagreement score rather than using GPT-3 as a generator (lines 5-6 in Algorithm 1). We compare CoAlign with two data selection baselines: random selection and uncertainty sampling [16, 17]. Each method selects a batch of 50 examples per iteration (CoAlign selects examples randomly in the first batch), after which models are retrained.

Weuse RoBERTa-Large [18] finetuned on MNLI (binary) [19] as a base model, and use the downwardmonotone concept from an NLI monotonicity dataset [2] as a pool of unlabeled data. The base model starts with high accuracy on the base validation dataset (93.5%), and low accuracy on the concept (23.5%). We present accuracy on the concept over iterations in Figure 4. While accuracy on the base

data remains constant throghout iterations, CoAlign's disagreement-based sampling is more efficient than uncertainty sampling or random selection.

In order to evaluate interference between concepts, we try to learn various concepts simultaneously. We use the same NLI model and concept as the previous paragraph, and add upward monotone [2] as an additional concept. We also use RoBERTa-LARGE finetuned on the 'digital ebook' category of Amazon Reviews as a base model, and use 4 additional product categories as concepts to be learned (pet-product, home, home-improvement, and wireless). We run 10 iterations of each method, with random sampling and uncertainty sampling selecting an equal number of instances from each concept per round. Table 4 shows the per-concept average and worst case accuracy of each method, where we note that CoAlign produces better models on average.

Table 5: Some examples of a pilot study that qualitatively examines the effectiveness of the CoAlign in assisting users in conceptualizing their ideas and debugging the model. Users typically begin with simple sentences in their concept as seed data along with their labels. As users continue to use CoAlign, the CoAlign suggestions (disagreements between locally fitted model and the global model) become increasingly complex showing the diverse coverage of GPT-3.

Pilot Study (qualitative). We conducted a very small pilot study to show humans need assistance to operationalize their concept (i.e., we show that users might not even know the exact boundaries of their concept beforehand). Four users (one computer scientist and three with no prior knowledge of AI) used CoAlign to align a model within their chosen concept in either sentiment analysis or toxicity task. Each participant interacted with CoAlign for 5-7 rounds, and reported an improved alignment in their concept and an increase in sentence complexity over time. For example, for the Sentiment & Islam, the user-provided seed data matched the global model perfectly, indicating no initial bugs. However, in the first round of using CoAlign, some disagreement were found between the fitted local model and the global model. The user identified some of these as bugs in the global model (e.g. 'Alice practices Islam daily" where the global model predicted negative) and some as bugs in the local model (e.g. 'Alice is a radical Islamist" where the local model predicted neutral). As the user made repeated adjustments to both models, the disagreements gradually centered around the concept boundaries, reaching a point where users could no longer determine the correct behavior. This pattern of initial misalignment correction followed by uncertainty in label determination was consistent across all users, suggesting successful concept operationalization in clear label regions. Table 5 provides examples of user interactions with CoAlign 5 .

6 Related Work

Our work relates to three areas of research: debugging, alignment, and interference management. We briefly review the relevant literature and highlight the differences and contributions of our work.

Debugging. Numerous efforts aim to find and fix models failures. Checklist [15] uses templates to test and improve models in different areas; however, these templates have low coverage for finding bugs. Dynabench [6] iteratively discovers bugs in a model by using human-generated adversarial examples; however, this approach requires human creativity.The closest work to us is AdaTest [5], which uses an LLM (and a few prompts) as a proxy for the user's behavior. Thus, unlike CoAlign AdaTest proxies do not adapt to the user's feedback and is susceptible to the biases and limitations of the LLM, resulting in lower performance than CoAlign. Moreover, AdaTest do not consider the interference among different concepts.

Alignment. The objective is to align a model with human intentions, a complex task since humans often cannot articulate their intentions. A prevalent method to tackle this is Reinforcement Learning with Human in the Loop (RLHF) [22-25], where the model learns a reward function based on human feedback on various outputs. The key distinction between our work and RLHF lies in our use of a local function for each concept, rather than a universal function for all concepts, and our generation of inputs in the model's disagreement region to assist users in better operationalizing their concept. Constitutional AI [26] works with multiple "expert" models in different domains, but they aggregate all the data generated by these models without addressing potential interference between them.

Interference management. Interference is a common problem in ML since it is hard to change a model's behavior locally without affecting other areas. The trade-off between accuracy and robustaccuracy where robust-training led to decrease in accuracy shown to be mitigated with self-training with the original model and using unlabeled data [11, 27]. Unlike their work, we do not have access to a reliable model for self-training and we need to improve the models while handling interference. Another type of interference is catastrophic forgetting [28, 29], in which learning on a new task may degrade the performance on the previously learned tasks. Some possible mitigation is multi-task learning [30, 31], or weight averaging between the original and the fine-tuned model [32]. Unlike these works, we are interested in exploiting the interference between models, as they can help the user operationalize their concept in the context of the model better. Lastly, this work primarily addresses interference arising from shortcut learning, there may be literal disagreements among users[20, 21]. Although a crucial issue, it falls outside the scope of this paper. However, we like to note that our method can surface such disagreements. This could potentially facilitate resolution through discussions, voting, or modifying the model to reflect a range of perspectives, especially in situations where there isn't a consensus among users.

7 Conclusion

Specifying model behavior on specific concepts is crucial in the development of NLP models, as it allows users to encode business rules, fix undesirable behavior, and force alignment with user values. Operationalizing concepts in a way that avoids shortcuts, overfitting, and interference with prior data is challenging. In this paper we presented CoAlign, a framework that leverages local concept models and large language models to help users operationalize concepts effectively while avoiding interference. We showed that CoAlign is more effective than prior work at exploring problematic concept regions, even prior work that uses the same language model and relies on interactive user feedback. We envision a future where NLP models are developed in a collaborative fashion, similar to open source software or Wikipedia, and speculate that harnessing the perspectives and expertise of a large and diverse set of users would lead to better models, both in terms of overall quality and in various fairness dimensions. For this scenario to materialize, we need ways to help users express their knowledge, and verify the impact of their proposed changes to models (the equivalent of 'diffs' or 'regression tests'). We believe CoAlign is a step in this direction.

8 Broader Impact and Limitations

CoAlign aids in operationalizing concepts without filtering the values a user wishes the model to align with, which might inadvertently allow a malicious user to encode harmful behavior into the NLP model, a risk for which we currently have no safeguards. Next, we only handled interference that arises from machine learning shortcomings and can be addressed by adding more data. However, there might be literal disagreements between users (i.e., two users prefer different labels for the same sentence). Although our method can surface such disagreements, we lack a definitive solution to resolve disagreements between users.

Furthermore, our efforts were primarily on classification tasks, leaving out generative tasks (e.g., next word prediction). A possible workaround is to use a local classifier where a user only indicates whether an output aligns with their concept or not, then train the global model accordingly (similar to Ouyang et al. [23] but with multiple reward models instead of just one). Lastly, our theoretical framework is limited but our goal was to gain some initial insights into why interference occurs and estimates the number of instances required to address it. Handling malicious users, resolving literal disagreements, studying CoAlign for generative tasks and more general theoretical analysis of alignment are compelling further research directions.

9 Acknowledgment

We gratefully acknowledge the contribution of Scott Lundberg, whose insightful discussions and assistance in utilizing the AdaTest repository greatly enhanced our research. Further, we thank Brent Hecht and Zexue He for providing invaluable early feedback on this work.

References

Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society , pages 67-73, 2018.

Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, and Kentaro Inui. Do neural models learn systematicity of monotonicity inference in natural language? arXiv preprint arXiv:2004.14839 , 2020.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-5446. URL https://aclanthology. org/W18-5446 .

Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, et al. Using the framework. Technical report, Technical Report LRE 62-051 D-16, The FraCaS Consortium, 1996.

Marco Tulio Ribeiro and Scott Lundberg. Adaptive testing and debugging of nlp models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3253-3267, 2022.

Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 4110-4124, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.324. URL https:// aclanthology.org/2021.naacl-main.324 .

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier. In Knowledge Discovery and Data Mining (KDD) , 2016.

Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems , 30, 2017.

Fereshte Khani and Percy Liang. Removing spurious features can hurt accuracy and affect groups disproportionately. In ACM Conference on Fairness, Accountability, and Transparency (FAccT) , 2021.

Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John C. Duchi, and Percy Liang. Adversarial training can hurt generalization. arXiv preprint arXiv:1906.06032 , 2019.

Megha Srivastava, Besmira Nushi, Ece Kamar, Shital Shah, and Eric Horvitz. An empirical analysis of backward compatibility in machine learning systems. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 3272-3280, 2020.

Preetum Nakkiran. More data can hurt for linear regression: Sample-wise double descent. arXiv preprint arXiv:1912.07242 , 2019.

Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John Duchi, and Percy Liang. Understanding and mitigating the tradeoff between robustness and accuracy. arXiv preprint arXiv:2002.10716 , 2020.

Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Association for Computational Linguistics (ACL) , pages 4902-4912, 2020.

David D Lewis. A sequential algorithm for training text classifiers: Corrigendum and additional data. In Acm Sigir Forum , volume 29, pages 13-19. ACM New York, NY, USA, 1995.

Vu-Linh Nguyen, Mohammad Hossein Shaker, and Eyke Hüllermeier. How to measure uncertainty in uncertainty sampling for active learning. Machine Learning , 111(1):89-122, 2022.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 , 2018.

Ruyuan Wan, Jaehyung Kim, and Dongyeop Kang. Everyone's voice matters: Quantifying annotation disagreement using demographic information. arXiv preprint arXiv:2301.05036 , 2023.

Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics , 10:92-110, 2022.

Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS) , 2017.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022.

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019.

Wangchunshu Zhou and Ke Xu. Learning to compare for better training and evaluation of open domain natural language generation models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 9717-9724, 2020.

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 , 2022.

Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C. Duchi. Unlabeled data improves adversarial robustness. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation , volume 24, pages 109-165. Elsevier, 1989.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

Yu Zhang and Qiang Yang. An overview of multi-task learning. National Science Review , 5(1): 30-43, 2018.

Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv:1511.06342 , 2015.

Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. arXiv preprint arXiv:2208.05592 , 2022.

Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in Neural Information Processing Systems , 30, 2017.

A Stationary distribution of the generator

Let P G denote the transition probability of the generator, where P G ( x | r = x ′ ) denote probability of generating x condition on the prompt being x ′ . We want to create a Markov chain to simulate a random walk within the user's distribution ( P i ). Specifically, we begin with a user-provided seed data point x and use it as a prompt for G to generate a new data point x ′ . If P i ( x ′ ) > 0 , we accept x ′ , otherwise we remain at x and repeat the process. We are interested in conditions under which the stationary distribution of this markov chain is P i

̸

Let's assume that support of P i is finite and denote it by S , let P ′ G ( x ′ | x ) be the transition function over S × S where P ′ G ( x | x ) = P G ( x | x ) + ∑ P G ( x ′ | x ) I [ P i ( x ′ ) = 0] and for x, x ′ ∈ S where x = x ′ we have P ′ G ( x | x ′ ) = P G ( x | x ′ ) .

If the transition graph generated by P ′ is irreducible (any state can be reached from any other state) and all its states are positive recurrent (the expected time to return to a state is finite), then the unique stationary distribution using G is P i if the following equality holds:

The above statement states that the probability of a data point ( P i ( x ) ) should be proportional to the probability of reaching to that point with the transition function of P ′ .

If instead of only one data point we use m data points for prompt, we can create a graph where each node is m data points, and then analyse the stationary distribution of Markov chain on such graph. In this case, when we start from a node with m examples and prompt the language model with them in this case the probability of going to ( x ′ , x m , . . . , x 2 ) from ( x m , . . . , x 1 ) is equal to P ′ G ( x ′ | r = ( x m , . . . , x 1 )) .

B Linear regression analysis

In this section, we examine the performance of CoAlign in a simple linear regression scenario. Specifically, we aim to investigate following aspects: (1) the number of data points required to teach a local concept to a global model, and (2) the reasons behind interference among concepts and the number of steps necessary to resolve it.

B.1 Setup

We consider each input x ∈ R d where only some of the data points are valid. There exist a true function θ ⋆ ∈ R d such that y = θ ⋆ ⊤ x . The support of each concept ( P i ) lays on a subspace ( S i ) and all valid data points on that subspace belongs to C i . Given k examples in C i , let S obv i denote the subspace observed by the training data, S uno i denote the unobserved subspace, thus S i = S obv i + S uno i is the smallest subspace containing all data points in C i . Finally S inv i denote the subspace that concept i does not have any variation in it.

As a running example, let x ∈ R 3 and consider a concept where data points belonging to that concept satisfies x 1 = x 2 . Recall that only some of the data points in this subspace are valid e.g., a point is valid if x 1 is odd thus [1 , 1 , 0] is valid while [2 , 2 , 1] is not. Let's assume we observed x = [1 , 1 , 0] in that subspace with label y = 2 . In this case we have: S obv 1 = [1 , 1 , 0] , S uno 1 = [0 , 0 , 1] and S inv 1 = [1 , -1 , 0] .

We consider the overparametrized noiseless linear regression, where number of features ( d ) is larger than number of acquired training examples ( n ) ( therefore, we can always interpolate all the training data) and there is no noise in observed targets. Following work of [33] which showed gradient descent on linear regression lead to min L2-norm, we assume local and global models infer the min L2 norm interpolant. As an example, for our running example the min-norm solution interpolating the concept is ˆ θ = [1 , 1 , 0] .

B.2 Operationalizing a concept: from disagreement to convergence

An alternate interpretation of the min-norm involves inferring the parameters by taking into account explicit constraints that require ˆ θ 's projection on S uno i and S inv i to be zero. For instance, in our

current example, we can deduce the min-norm solution by solving these linear equations: ( [0 , 0 , 1] θ = 0 , [1 , -1 , 0] θ = 0 , [1 , 1 , 0] θ = 2 ).

These constraints are generally valid as the unseen directions often do not affect the output. However, these constraints may be violated when we combine local concept data with global data, as the projection of S uno i and S obv 0 may not be zero. This implies that the output could change with variations in the unseen directions, leading to local models typically outperforming global models within a local concept.

To ensure both local and global models perform equally well in the local concept, we need to enforce the invariance constraints explicitly. This involves adding new data that exhibit variations in the unseen directions and demonstrating that these variations do not affect the output. Furthermore, we presume that S uno i is significantly large, making methods that attempt to examine all possible directions inefficient. Therefore, it's more advantageous to only verify directions that are affected by the merge.

Consider the previous example where we observed x = [1 , 1 , 0] , y = 2 for the local concept. Now, imagine the we observed x = [0 , 1 , 1] , y = 2 in global dataset. When we combine this data point with the concept data point, we get ˆ θ global = [ 2 3 , 4 3 , 2 3 ] . This causes a disagreement in data points that vary in the [0 , 0 , 1] direction within the local concept, the local model predicts 0 while the global model predicts 2 3 . Note that both the global and local predictions align for variations in the [1 , 1 , 0] direction.

In the event of such a disagreement, we have two options: (1) The variation in this direction is indeed non-zero, suggesting the local model requires further refinement - a frequent occurrence in early stages, or (2) The variation is zero, but it needs to be specified as such; otherwise, the global model assumes other values due to its implicit bias towards generating the simplest model. Note that there is no disagreements in the common directions between S uno 0 and S uno i or their orthogonal subspaces.

Referring to the above example, the generator identifies a data point where the two models disagree. Let's assume this data point is x = [0 , 0 , 1] , where the local model predicts 0 , but the global model predicts 2 3 . In such a case, we present this data point to the user. Let's assume user specify that the label for this data point is 0 . In this case by adding this new data point the global prediction adjusts to ˆ θ global = [0 , 2 , 0] .

After we learn the local concept (i.e., all the unobserved directions are indeed zero), how many of them do we need to add as explicit constraints? the following proposition shows maximum number of disagreements after learning a local concept.

Proposition 1. If proj S uno i ( θ ⋆ ) = 0 , then the maximum number of disagreement between local and global models is dim(proj S obv 0 ( S uno i ∩ ( S uno i ∩ S obv 0 ) ⊥ )) .

̸

Proof. The global and local models agree on all observed directions (i.e., S obv i and S obv 0 ). However, there is a disagreement for any vector u in S uno i such that ˆ θ global = proj S obv 0 ( θ ⋆ ) ⊤ u = 0 since ˆ θ ⊤ i u = 0 . Let's assume we add k examples such that local and global disagree. We now prove that k ≤ dim(proj S obv 0 ( S uno i ∩ ( S uno i ∩ S obv 0 ) ⊥ )) .

For the k added examples, only consider their components in ( S uno i ∩ ( S uno i ∩ S obv 0 ) ⊥ ) (we can remove the S obv i components by subtracting their projection on S obv i similarly remove any component in ( S uno i ∩ S obv 0 ) by subtracting their projection in S obv 0 ). In order to have a disagreement these data points should have non-zero projection on S obv i otherwise there will be no disagreements. As a result the maximum number of data points is dim(proj S obv 0 ( S uno i ∩ ( S uno i ∩ S obv 0 ) ⊥ )) .

B.3 Handling interference between concepts

In previous section, we explained why disagreement can happen between local and global model and how we can resolve the disagreements by querying user of the local concept. We bound number of disagreement with dimension of projection of S uno i on S obv 0 . In previous section we did not need to change S obv 0 but when concept j has conflicts with concept i we also add data to concept j (thus changing S obv j ) which can lead to new conflicts with concept i .

b

b

a

Global

Local

Global

Local

a

I'm Muslim

Neutral

I'm Iranian  Neutral

I pray in mosque  Neutral

He was Muslim

Negative

Neutral

I'm not American

Negative

Neutral

1 love USA  Positive

1love Muslims

Positive

Neutral

1 love Canadian

Positive

Positive

1 fast in Ramadan

Neutral

Neutral

He lives in China

Neutral

Neutral

1 hate Muslims

Negative

Negative

1 hate Mexicans

Negative

Negative

User seeds concept &learning

User seeds concept &learning

the initial local model

the initial local model

GPT-3generates data

GPT-3generates data

Update local and

global models

User labels instances where

User labelsinstances where

localandglobaldisagree

localandglobaldisagree

The following proposition state that in addition to the dimension of projection of S uno i on observed subspace we also need to calculate projection on the unobserved space of different concepts as they might get added in the future. With notation of S obv 0: k denoting sum of all the S obv i , and S -i denotes sum of all subspaces except i , the following proposition bounds number of times users need to add data to their concepts due to interference.

Proposition 2. If for all i , proj S uno i ( θ ⋆ ) = 0 then the maximum number of times that we need to handle interference is ∑ k i =1 dim ( proj S -i ( S uno i ∩ ( S uno i ∩ S obv 0: k ) ⊥ ) ) .

Proof. The proof is similar to Proposition 1. Here we need to deal with conflicts with all other topics and since it is possible that we add their unobserved subspace as well we need to compute the dimension of S uno i on the whole S j subspace not only S obv j .

Let assume we added t example from concept i to handle interference, we now prove that t ≤ dim(proj S -i ( S uno i ∩ ( S uno i ∩ S obv 0: k ) ⊥ )) . For every data point that we add we first remove S obv 0: k components by removing its projection on S obv 0: k . Now in order to have a conflict this data point should have non-zero projection on S -i . As a result the maximum number of data points we can add is less or equal than dim(proj S -i ( S uno i ∩ ( S uno i ∩ S obv 0: k ) ⊥ )) , summing over all the concept result in maximum number of interference that needs to be handled.

C Extra figures and tables

Figure 5 is showing the CoAlign loop for multiple users. Table 6 shows the 6 concepts used in our experiment along with examples for each concept.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper frames operationalizing concept as a solution to enumerate all possible concepts and developed a corresponding framework CoDev, that starts by collecting text and labels from users, then have GPT-3 generates text and labels. When there are disagreement among users and GPT-3, users would be asked to relabel the instance until convergence. The authors conducted experiments using Amazon review dataset and MNLI to compare CoDev with baseline models. Also the author conduct a pilot study with 4 people using CoDev from providing data to improve model alignment and more complex output after 5-7 iterations.\n\nSTRENGTHS:\nThe paper points out an important questions that NLP model align with multi-users values. And it is novel to use GPT-3 as the global model and Roberta-large trained on users input as the local model, and try to converge the disagreement from two models through collecting user labels iteratively.\n\nWEAKNESSES:\n1. Given the paper, that the authors handle disagreement by repeating collecting user labels until convergence, assumes that there exist a single ground truth for each instance. But many studies have shown that a single instance can have multiple acceptable answers [1, 2]. The study design also conflict with the study goal of aligning multi-users' value. Disagreement is not a sign of error but could be a sign of multiple possibility. How could you determine the disagreement between the local model and the global model is not acceptable?  \n\n[1]Wan, R., Kim, J., & Kang, D. (2023). Everyone’s Voice Matters: Quantifying Annotation Disagreement Using Demographic Information. Proceedings of the AAAI Conference on Artificial Intelligence, 37(12), 14523-14530. https://doi.org/10.1609/aaai.v37i12.26698\n[2]Aida Mostafazadeh Davani, Mark Díaz, Vinodkumar Prabhakaran; Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. Transactions of the Association for Computational Linguistics 2022; 10 92–110. doi: https://doi.org/10.1162/tacl_a_00449\n\n2. The authors mentioned the shortcoming of having expert label datasets in the introduction. But actually many open sourced dataset for machine learning studies are labeled by crowd workers from third party platforms such as Amazon MTurk. And the determination for sentiment or toxicity is a very subjective task, that is there is no expertise to judge whether others feels the text is toxic or not. Therefore, the research gap mentioned in the introduction and the dataset/task picked for the experiment make author's framework and argument weak. \n\n3. For table 2, it is not clear how SB is different from biased SB. And it is also not clear how Data collection method is different from base line BERT model and CoDev model.  And many clarity concerns as listed in the following Question section. These confusions make the paper less convincing.\n\nQUESTIONS:\n1. There is no clear definition for your local model and global model other than the figure 1. Could the authors please clarify why and how they use GPT 3 as a global model? What prompts are used?\n\n2. It is not clear whether the d step (repeating labeling) in the Figure 1 is involved with a single user during the iteration or multiple users. If multiple users, how did the authors handle the disagreement among users? If that's different user for different round, this need to be clarified. If single user,  how could the CoDev system enable multi-user interaction?\n\n3. For the pilot study, it is not clear how the authors collect the seed data. Are those labels given by single user for each instance? If each instance collect labels from multiple users, how does the authors aggregate the labels? \n\n4. For the pilot study, it is is not clear how the authors evaluate their study. If better alignment is the evaluation metric, how do the authors evaluate the quality of alignment? In the line 289 and 290, the authors said 'As the user made repeated adjustments to both models, the disagreements gradually centered around the concept boundaries, reaching a point where users could no longer determine the correct behavior.' It is not clear why users could no longer determine the correct behavior is the standard for good alignment. Shouldn't users always have the right and ability to determine what they think is correct? \n\n5. In Table 4, why is there only labels for the 'Seed data' column but the 'Initial rounds' and 'Final rounds' don't have corresponding labels? Are the Initial rounds and final rounds evaluate alignment based on the labels? And the Table 4 caption mainly highlight the text in final rounds become much complex than the initial rounds. Why is this important? Is this an evaluation criteria for the pilot study?\n\nLIMITATIONS:\nThe authors discussed the limitation of solving literal disagreement since they only handled interference caused my machine learning shortcoming.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper proposes a new framework to debug NLP models. Specifically, while debugging a global model, it starts with training some local models on specific concepts. Then new data used for model improvement is labeled if the global model disagrees with the local model. 

Experiments indicate that the proposed approach (CoDev) outperforms the baseline approach (AdaTest) over multiple settings.

STRENGTHS:
1. This work focuses on an important but under-explored problem: debugging NLP models. The findings could potentially benefit the community.
2. The paper is well-written, easy to understand the contents.

WEAKNESSES:
1. There is a gap between theoretical guarantee and real experiments. Specifically the paper experiments with LLM, but provides proof based on linear regression.
2. It's unclear if some of the results are statistically significant. For example, in Table 3 (about MNLI and sentiment analysis), the improvement seems marginal. 
3. All experiments seem to focus mainly on toy tasks like classification or NLI. The takeaway would be stronger if the paper has discussions about extending to more challenging NLP tasks such as structure prediction.

QUESTIONS:
1. Missing discussion about active learning in related work. 
2. (see weakness 3) How to extend the proposed framework to structure prediction tasks? Also, which NLP tasks do you think the proposed approach would fail?

LIMITATIONS:
Yes, this paper includes the limitation section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper describes a multi-user collaborative model alignment framework that teaches certain desired concepts (behaviors, rules) to large language model (LLM).
The authors train a global model that intergrates the original data and all concepts, and a local model for each concept. The LLM is guided to generate examples where the local and the gloabl model disagree, and these examples are presented to users for annotation. Finally, these new annotations are used to update local and global models.

The proposed framework is evaluated on cases that teach an NLI model about downward- and upward-monotone concepts and a sentiment classifier about Amazon products reviews.
Results show that it outperforms AdaTest and other baselines.

STRENGTHS:
* the targeting question is an important question and has attracted a lot of research interest. the proposed approach is easy to understand and experimental results show that it is effective, especially when there is interference between multiple concepts

WEAKNESSES:
* the experimental setup is not described in detail, it may not be easy for other researchers to reproduce the study

QUESTIONS:
Line 29: 'does not to lead to'

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method based on data augmentation and instance selection for training a supervised model to be aligned with \"concepts\", where concepts dictate specific model behavior on certain inputs. In the proposed setup users illustrate a concept with a training example, this is followed by generating additional examples with GPT3, the acceptance of the generated examples is proportional to the disagreement between a global model trained on all the data and a local model trained to learn the concept perfectly. In experimental evaluations the proposed instance selection method seems to outperform reasonable other strategies for selecting data for aligning supervised models on specific concepts.\n\nSTRENGTHS:\n- The proposed data selection method seems original, the presented method with an LLM data generator in the loop is under explored.\n- The problem of enforcing user desires on a supervised model is meaningful.\n\nWEAKNESSES:\n- The proposed method for aligning user \"concepts\" seems somewhat laborious or frustrating from a user-stand point, especially in cases where users concepts conflict with those of others. E.g. the paper notes, \"In practice, this means that a user adding a new concept needs to make sure it does not break any concepts from other users, a process similar to regression testing in software engineering.\" -- What happens in cases where the users concept cannot be incorporated without breaking somebody else's concept?\n- The experimental setup of the paper is hard to follow and is unlikely to be reproducible.\n\nQUESTIONS:\n- The idea of a concept seems a little vague; please consider discussing what kinds of user intents may be captured in the concepts. Perhaps with an eye toward realistic concepts that users want to see alignment on. As I see it, the concept seems like a rule for labeling specific input instances based on easily examinable features of the input.\n- Many of the ideas described in the paper (user-defined concepts, conflicts between then, training on user concepts) seem similar to the line of work on programmatic creation of datasets with weak labelers. Please cite and discuss similarities and differences to this line of work: https://scholar.google.com/citations?user=rfwwtFYAAAAJ&hl=en\n- The paper (through references of similarity) and especially the experimental setup (first two sub-parts) heavily depend on familiarity with prior work. Please describe the experimental setup and these prior works in greater detail in this paper.\n- This is perhaps a matter of style and framing, but I would recommend renaming the title of this paper. The presented work seems to be much more specific in its scope than the title seems to indicate. The presented approach relies on users labeling data (significantly devalued labor in ML) to align models with their preferred concepts, a large body of work may be framed as \"collaboratively developing NLP\" models by this naming logic. Further, the explored tasks are simplistic classification tasks; arguably, NLP encompasses a significantly broader range of tasks than these alone.\n\nLIMITATIONS:\nThe paper discusses limitations adequately.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a framework for collaborative NLP development (CoDev) that enables multiple users to align a model with their beliefs.\nThe proposed work CoDev aids users in clarifying their concepts (an area humans often struggle with) and assists ML models to handle conflicts between concepts (an area ML often struggle with due to its inability to accommodate local updates).\n\nThe authors' main insight is learning a local model for each concept, and a global model to integrate the original data with all concepts. The authors then steer a large language model to generate instances within concept boundaries where local and global disagree. Their experiments show CoDev is effective at helping multiple users operationalize concepts and avoid interference for a variety of scenarios, tasks, and models.\n\nSTRENGTHS:\n1. The paper is pretty easy to follow and propose a new framework that is useful for controlling LLMs to constrain the outputs according to user defined concepts.\n2. The idea of creating proxies from local models for user concepts is novel and interesting.\n3. The experiments with real world users is nice, as this work is all about collaborative development from users.\n4. There are some theoretical analysis in the proposed framework which is nice and provides additional insight.\n\nWEAKNESSES:\n1. I think more discussion and datasets could be used in terms of CoDev with biased seed data and CoDev with unlabeled data. Since the work is very central about the data distribution and checking for controlling models via user input, I think a more diverse type of dataset could be used, instead of just selecting positives and negatives from review dataset. It might provide better evidence in generalizability of such framework. \n2. Since AdaTest is a major related work, maybe some more comparison with it in the seeded datasets and unlabeled data would be nice?\n\nQUESTIONS:\nAbove.\n\nLIMITATIONS:\nThe limitations and societal impact are properly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-12"}