A State-Vector Framework for Dataset Effects
Authors: Esmat Sahak
Zining Zhu
Frank Rudzicz
Venue: EMNLP 2023 Main (2023)
Keywords: data influence
probing
fine-tuning
multi-task learning
datasets
TL;DR: We propose a state-vector framework to analyze the effects of datasets on deep learning language models using probing tasks.

Abstract: The impressive success of recent deep neural network (DNN)-based systems is significantly influenced by the high-quality datasets used in training. However, the effects of the datasets, especially how they interact with each other, remain underexplored. We propose a state-vector framework to enable rigorous studies in this direction. This framework uses idealized probing test results as the bases of a vector space. This framework allows us to quantify the effects of both standalone and interacting datasets. We show that the significant effects of some commonly-used language understanding datasets are characteristic and are concentrated on a few linguistic dimensions. Additionally, we observe some ``spill-over'' effects: the datasets could impact the models along dimensions that may seem unrelated to the intended tasks. Our state-vector framework paves the way for a systematic understanding of the dataset effects, a crucial component in responsible and robust model development.

A State- Vector Framework for Dataset Effects
Esmat Sahak Zining Zhu Frank Rudzicz4.2.
University of Toronto, Vector Institute for Artificial Intelligence, Dalhousie University esmat. Schak@mail utoronto. ca. ziningêcs toronto... edu, frank@dal... ca
Abstract
Dataset effect of ss12 conditioned on ['mrpe' 'rte|]
The impressive success of recent deep neural network (DNN)-based systems significantly influenced by the high-quality datasets used in training. However, the effects of the datasets, especially how they interact with each other, remain underexplored. We propose state- vector framework to enable rigorous studies in this direction. This framework uses ideal- ized probing test results as the bases of a vec- tor space. This framework allows us to quan- tify the effects of both standalone and inter- acting datasets. We show that the significant effects some commonly-used language un- derstanding datasets characteristic and concentrated on a few linguistic dimensions. Additionally, we observe some "spill-over" ef- fects: the datasets could impact the models along dimensions that may seem unrelated to the intended tasks. Our state-vector framework paves the way for systematic understanding the dataset effects, crucial component in responsible and robust model development.
86
end
04
82
80-
78
Deptis
Figure 1: An example of the individual dataset ef- feet. This figure shows how two linguistic abilities of RoBERTa, two probing tasks: syntactic tree depth and tense, are changed by fine-tuni fine-tuning the SST2 dataset.
characterized by the accuracies of
opers apply different datasets. Model developers use large corpora that may include multiple existing datasets (Sun et al., 2021; Wei et al., 2021; Brown al., 2020). How these data relate the progress in linguistic ability is not systematically studied yet. Each dataset has desired effects, but does it have some under-specified effects? When using multiple datasets together, do they have undesired, interaction effects? These questions become more contingent as the engineering of datasets becomes more essential, yet existing framework allows convenient quantification of these dataset effects. Probing provides convenient frameworks study the linguistic abilities of DNN systems from multiple perspectives. Probing analyses show how the components of DNNs demonstrate linguistic abilities (Tenney et al., 2019; Rogers et 2020; Belinkov, 2021; Niu et al., 2022). The prob- ing results relevant to the model's ability the extent that the probing accuracies can predict the model's downstream performance (Zhu et al., 2022a). These findings make probing classification promising candidate for setting up a framework to describe the effects of datasets.
1 Introduction
recent years, data-driven systems have shown im- pressive performance on a wide variety of tasks and massive, high-quality data is a crucial component for their success (Brown et al., 2020; Hoffmann al., 2022). Currently, the availability of language data grows much more slowly than the computa- tion power (approximately Moore's law), raising the concern "data exhaustion" in the near future (Villalobos et al., 2022). This impending limitation calls for more attention to study the quality and the effects of data.
The data-driven systems gain linguistic abilities multiple levels ranging from syntax, semantics, and even some discourse-related abilities during the training procedures (Liu et al., 2021). The training procedures almost always include multiple datasets usually there is a "pre-training phase and "fine-tuning" phase, where the model devel-

---

In this paper, we set up a state-vector framework for describing the dataset effects. We formalize ide- alized probes, which give the true linguistic ability of the DNN model in a state of training. Then, we derive two terms, individual and interaction effects, that describe the dataset effects along multiple di- mensions of linguistic abilities. A benefit of our state-vector framework is that it allows a conve- nient setup of statistical tests, supporting rigorous interpretation of how the datasets affect the models. The state framework allows us to frame tran- sitions and set up experiments efficiently. Many frequently-used datasets have "spill-over" effects besides the purposes they are originally collected to achieve. Additionally, the interaction effects are concentrated and characteristic. Our framework provides systematic approach to studying these effects of the datasets, shedding light on an aspect of model developments that deserves more atten- tions. All scripts, data, and analyses are available at our GitHub repository.
used downstream performance to evaluate the mul- tidimensional effects. We operate from a dataset perspective and use the probing performance to evaluate the multidimensional effects. Weller et al. (2022) compared two multitask fine-tuning settings, sequential and joint training. The former can reach higher transfer performance when the target dataset is larger in size. We use the same dataset size on all tasks, either multitask setting is acceptable.
3 A Vector framework
This section describes the procedure for formulat- ing state-vector framework for analyzing dataset effects in multitask fine-tuning.
An abstract view of probes There have been many papers on the theme of probing deep neural networks. The term "probing" contains multiple senses. A narrow sense refers specifically to ap- plying post-hoc predictions to the intermediate rep- resentations of DNNs. A broader sense refers to examination methods that aim at understanding the intrinsics of the DNNs. We consider an abstract view, treating probing test map from multi- dimensional representations result describing a designated aspect of the status of the DNN, Ideally, the test result faithfully and reliably reflects the designated aspect. We refer to the probes as idealized probes henceforth.
2 Related Works
Understanding the datasets Recent work has investigated various properties of datasets. Swayamdipta et al. (2020) used the signals of the training dynamics to map individual data samples onto "easy to learn", "hard to learn", and "ambiguous" regions. Ethayarajh et al. (2022) used an aggregate score, predictive V-information (Xu et al., 2020), to describe the difficulty of datasets. Some datasets can train models with complex decision boundaries. From this perspective, the complexity can be described by the extent to which the data samples are clustered by the labels, which can be quantified by the Calinski-Habase index. Recently, Jeon et al. (2022) generalized this score to multiple datasets. We also consider datasets in aggregate manner but with unique perspective. With the help of probing analyses, are able to evaluate the effects of the datasets along multiple dimensions, rather than as a single difficulty score.
scalar-valued test
Idealized probes vectorize model states Train- ing DNN models is complex process consisting of sequence of states. In each state S, we can apply battery of K idealized probes to the DNN and obtain a collection of results describing the lin- guistic capabilities of the model's 'ss state at timestep T = DNN model during multitask training can be de- scribed by state vector S C R
In this way, the state of a
Without loss of generality, we define the range of each probing result in IR. Empirically, many scores are used as probing results, including correlation scores (Gupta et al., 2015), usable V-information (Pimentel and Cotterell, 2021), minimum descrip- tion length (Voita and Titov, 2020), and combina- tions thereof (Hewitt and Liang, 2019). Currently, the most popular probing results are written in ac- curacy values, ranging from 0 to 1. We do not require the actual choice of probing metric as long as all probing tasks adopt the same metric.
Multitask fine-tuning Mosbach et al. (2020) studied how fine-tuning affects the linguistic knowl- edge encoded in the representations. Our results echo their finding that fine-tuning can either en- hance or remove some knowledge, as measured by probing accuracies. Our proposed dataset effect framework formalizes the study in this direction. Aroca-Oucllees and Rudziez (2020) studied the effects of different losses on the DNN models and
From model state to dataset state Now that we have the vectorized readout values of the states,

---

E(X)
S(X.
dataset leading to the reference state Sy. In edge case, the dataset X is a subset of the dataset I, so S(x,]] = S₁. This can be attributed to the fact that our definition S is the global optimum of a model fine-tuned on / among all possible training proce- dures, including re-sampling. Hence, the dataset X should have no effect on the model, which can be verified by (
S(I)
S(XX. Y. ID
EIX, YD
Int(X, Y)
E(XHE(())
E(Y)
S(D + E(X) + E(Y)
I)Y.)
Figure 2: Schematic illustration of the dataset effects. E-)) are the individual effects, Int(X, Y) the interac- tion effect.
4[) S(I) O.
Remark 2: In another scenario, X consists of plus only one other data point 2. Then E(X) degenerates to the effect of the data point 2.
how is the state S determined?? In general, there three factors: the previous state, the dataset, and the training procedure applied to the model since the previous state. We can factor out the effects of the previous state and the training procedure.
Remark 3: We assume X does not overlap with in the rest of this paper. This assumption stands without loss of generality since
always define X the non-overlapping data subset.
Remark The dataset effects form an Abelian group - Appendixx $A.3 contains the details.
To factor out the effect of the previous state, we introduce the concept of an "initial state". The initial state is specified by the structures and the parameters of the DNNs. If we study the effects of data in multitask fine-tuning BERT, then the initial state is the BERT model before any fine-tuning Let us write the initial state as Sp. Based on this initial state, the dataset X specifies a training task that leads the model from the initial state So to the current state Sx.
4.2 Interaction effect
Motivating example Let us first consider an ex- ample of detecting the sentiment polarity. Suppose three abilities can contribute to addressing the sen- timent polarity task:
A1: Parsing the structure of the review.
A2: Recognizing the tense of the review.
A3: Detecting some affective keywords such as "good".
To factor out the effect of the training procedure, assume the dataset X, together with the training objective, defines unique global optimum. Ad- ditionally, we consider the training procedure can eventually reach this optimum, which we write as the dataset state, Sx- Practically, various factors in the training procedures and the randomness in the sampled data may lead the models towards some lo- cal optima, but the framework considers the global optimum when specifying the dataset state.
Consider two sentiment polarity datasets, X and Y. In X, all positive reviews follow unique syntax structure where all negative reviews do not. In Y. all positive reviews written in the present tense where all negative reviews in the past tense. The problem specified by dataset X can be solved by relying on both A and A3, and the problem speci- fied by dataset Y can be solved by relying on both A2 and A3. Imagine a scenario where after train- ing on both X and Y, model relies solely on A3 to predict the sentiment polarity. This behavior is caused by the interaction between X and Y. Using the terms of our state-vector framework, there is an interaction effect between X and Y with a positive value along the dimension of A3, and a negative value along the dimensions of Al and A2.
4 Dataset effects
4.1 Individual dataset effect
We define the effect of a dataset x as:
E(X)= = S(X, 7]) R
(1)
Here [X, [] denotes combining datasets X and I, and s([x, I]) and S(I) are the state vectors of the states S|x./] and S, respectively. E(X) describes how does the linguistic ability of the model shift along the K dimensions that probe. In other words, E(X) describes the multi-dimensional fine- tuning effect of the dataset X.
Definition the interaction effect Let us define the interaction effect between two datasets, X and Y as:
-)X,))))B((() (E(X) + E(Y)) (2)
Remark 1: The effect of X depends on a "ref- erence state" S, where is used to describe the
This is the difference between the dataset effect
of [X, Y] (X and Y combined), and the sum of

---

the effects of X and Y (as if X and Y have no interactions all).
2019) and RoBERTa-base (Liu al., 2019). Doing this allows us to compare results and determine whether dataset effects hold despite model choice.
Remark 1: An equivalent formulation is:
Int(X Y) -((XX, S((x,]) -[) + S(1)
5.2 Fine-tuning
(3)
The GLUE benchmark (Wang al., 2018) consists of 3 types of natural language understanding tasks: single-sentence, similarity and paraphrase, and in- ference tasks. Two tasks from each category were selected to fine-tune models.
What X and share some com- mon items? This would introduce an additional effect when merging X and Y. An extreme exam- ple is when X and Y are the same datasets, where -
Remark 2:
Y] collapses to X. Then Int(X, where it should be O. The reason for this counter- intuitive observation is that the "collapse" step itself constitutes a significant interaction. Our "datasets do not overlap" assumption avoids this problem. This assumption goes without loss of generality because we can always redefine X and Y to contain distinct data points.
Single-sentence tasks COLA (Warstadt et al., 2018) labels whether sentence is grammatical English sentence not. SST2 (Socher et al., 2013) labels whether sentence has positive or negative sentiment.
Similarity and paraphrase tasks MRPC (Dolan and Brockett, 2005) labels whether sentences a pair are semantically equivalent or not. STSB (Cer et al., 2017) accomplishes the same objective except it provides similarity score between 1-5.
A linear regression view The interaction effect defined in Eq. 2, equals the Ba coefficient com- puted by regressing for the state vector along each of the K dimensions:
Inference tasks QNLI (Rajpurkar et al., 2016) takes a sentence-question pair as input and labels whether the sentence contains the answer the question. RTE (Dagan et al., 2005; Haim et al., 2006; Giampiccolo et al., 2007: Bentivogli et al., 2009) labels whether given conclusion is implied from some given text.
(k) i by + (4) ((k) = 000 2(10) iz B, i +05
where i and by indicator variables, and is the residual. If dataset X appears, d 1, and iz otherwise. The same applies to ig. (k)) is the 4h dimension in the state vector. The correspondence between the indicator variables and S as listed in Table 1:
The multitask model resembles that of Radford et al. (2018). It consists of a shared encoder with separate classification layer per task Figure 4 shows an illustration. This was made possible by leveraging HuggingFace's Transformers library (Wolf et al., 2020). Model hyperparameters are the same as those listed in Table of Mosbach et al. (2020), except epochs were used for all experiments. This is valid, their experiments incorporated both BERT-base-cased and RoBERTa- base models. Model checkpoints were saved every 6 optimization steps, with the final model being the one with the lowest training loss.'
iz
by
0
0
S())
0
S((X,I)
1
S(Y, I])
1
S([,,Y,1)
Table 1: Correspondence between the indicator vari- ables and the regression targets.
Please refer to Appendix A.4 for a derivation of the equivalence. The Eq. 4 formulation allows us to apply an ANOVA, which allows us to decide the interaction effect is significant (i.e., the p-value of 03 is smaller than the significance threshold).
To mitigate the effect train set size per task, train datasets were reduced to 2,490 examples per task, which corresponds to the size of the smallest train dataset (COLA). GLUE benchmarks such as QQP, MNLI, and WNLI (Levesque et al., 2012) were excluded because their train sets were too large (more than 300K examples) or too small (fewer than 1K examples).
5 Experiments
5.1 Models
Experiments are conducted using two popular lan- guage models: BERT-base-cart cased (Devlin et al.,
'Donnstrcam task performance can he accessed in our repository.

---

5.3 Probing
Interaction of cols and rtc, conditioned on
We use the SentEval suite (Conneau and Kiela, 2018) to build proxies for the idealized probes that vectorize the model states. SentEval contains the following tasks:
5(4]
Sly]
S(xyy
cn
Length: given a sentence, predict what range its length falls within (0: 5-8, 1: 9-12, 2: 13-
16, 3: 17-20, 4: 21-25, 5: 26-28).
76
WC: given sentence, predict which word it contains from target set of 1,000 words.
SubjNumbbrr
Depth: given sentence, predict the maxi- mum depth of its syntactic tree.
Figure 3: An example of interaction effect between x=COLA and y=RTE, RoBERTa. The purple arrow visualizes the interaction effect. This figure plots two linguistic ability dimensions syntax tree depth and subject number.
TopConst: given sentence, predict its con- stituent sequence (e.g. NP_VP_.: noun phrase followed by verb phrase).
BigramShift: given sentence, predict whether any two consecutive tokens of the original sentence have been inverted.
the dataset states in realistic time, we group the experiments by the format of the tasks (e.g., single- sentence, similarity, inference) and impose selec- tion conditions, reducing the total number of fine- tuning experiments from around 10k to 300 per model. Note that 300 fine-tuning experiments per model are still nontrivial, but they can be completed within two months. Section A.2 in Appendix con- tains the details.
Tense: given sentence, predict whether its main verb is in the past or present tense.
SubjNumber: given sentence, predict whether the subject of its main clause is sin- gular or plural.
ObjNumber: given a sentence, predict whether the direct object of its main clause is singular or plural.
OddManOut: given sentence, predict whether any verb noun of the original sen- tence was replaced with another form with the same part of speech.
Results
6.1 Individual dataset effect
CoordInv: given a sentence, predict whether two coordinated casual conjoints the origi- nal sentence have been inverted.
Some observations regarding individual dataset fects are noted below. Tables 2 and summarize average individual GLUE dataset effects for BERT and RoBERTa models, respectively. Tables 4 and 9 break down the average individual effects of COLA and SST2, respectively.
For each probing task, we downsample the datasets to 10% of their original sizes or 12K sam- ples per task (10K train, IK validation, 1K test). This is valid, datasets of similar sizes usually have sufficient statistical power (Card et al., 2020; Zhu et al., 2022b). WC was removed from con- sideration, as its performance would have been significantly compromised given that it possesses 1000 ground truths. We built our training pipeline based on the SentEval library and used the default config. The default architecture is single classifi- cation layer on top of the frozen fine-tuned encoder.
Model choice Per Tables 2 and 3, there is no con- sistent agreement between significant dimensions (i.e., dimensions marked as * or ***). In fact, of the combined 33 significant individual effects observed in Tables and 3, only 13 (+e 40%) be confirmed by both BERT and RoBERTa models. This demonstrates that datasets can have different effects depending on model architecture.
Dataset composition MRPC and STSB accom- plish very similar tasks, but impact different di- mensions. Matching significant dimensions for MRPC and STSB amount to 2 of 7 for BERT (see Table 2) and I of 5 for RoBERTa (see Table 3).
5.4 Multitask settings
As the number of tasks increases, the number of combinations increases exponentially. To compute
ESentEyal code and defaultt config are accessible here.
"Tables for other datascts can be accessed in our repository.

---

Dataset
Model
Length
Depth
TopConst
BigramShin
Tense
SubjNumber
ObiNummeer
OddManOOt
Coondinv
COLA
BERT
0.36
0.47
6.35
0.23
0.38
0.69"
1.93
1.09
SST2
BERT
-4.08**
3.92
-1.99
0.24
-1.75*
1.73
0.79"
0.06
MRPC
BERT
0.65
0.75
2.34
0.55
0.45
0.78
1.07"
1.13
1.34
STSB
BERT
0.53
-1666
0.24
0.3"
0.2
0.36
0..9
QNLI
BERT
0.05
0.33
0.85
0.49
0.16
0.81
0.16
0.34
1.33
BERT
0.83
0.34
0.24
0.21
0.15
0.65"
0.16
0.64
Table Individual effects of GLUE datasets, on BERT. * and *** stand for p < 0.05, p < 0.01, and p 0.001.
respectively, using two-sample t-test with dof 8.
Datasct
Model
Leagth
Depth
TopConst
BigramShift
Tense
SubjNumber
ObjNumber
OddManOut
CoordInv
COLA
RoBERTa
2.75
0.28
2.17'
9.17
1.21
0.26
0.59
3.66
2.0
SST2
RoBERTa
7.122*
-1.86*
-5.88*
-4.43"
1.57
3.92
-3.96
-1.85*
-2.8
MRPC
RoBERTa
-0.06
0.25
-0.87
-1.68
-1.01*
-0.61
-0.45
0.07
1.0
STSB
RoBERTa
0.03
0.14
-3.17
-3.0"
-1,84
-3,11
-1.65*
-0.43
0.16
QNLI
RoBERTa
0.67
0.04
-0.13
0.57
-0.56
0.33
-1.47"
0.5
0.13
RTE
RoBERTa
0.78
0.84
-0.58
-0.32
-0.81
0.85
-2.04
-0.25
0.0
Table 3: Individual effects of GLUE datascts, on RoBERTa. "
and .*. stand for < 0.05, p < 0.01, and
p 0.001, respectively, using two-sample l-tests with dof 8.
Although they both are paraphrasing tasks, their samples are extracted from different sources and have different ground truths (i.e., MRPC is a binary task, STSB is an ordinal task). Hence, the compo- sition of datasets can affect what individual effects they will contribute.
ments, besides BigramShift (for both models) and OddManOut (positive effect for RoBERTa only). The same conclusion is valid for datasets other than COLA. This implies that there are inherent interaction effects between datasets that also influ- results. Note that if we add dataset a large number of varying reference states and observe that there are persistent, significant dimensions across these experiments, then this is a strong empirical indication of a dataset's effect on this set of dimen- sions (e.g.. see COLA's effect on BigramShift in Table 4 and SST2's effect on Length in Table 9). In the case of our experiments, for argument's 's sake, impose that dataset effect must appear in at least 70% reference states. This lower bound can be adjusted to 60% or 80%, but wouldn't result in any major adjustments as dataset effects tend to ap- pear in fewer than 50% of our experiments. Table summarizes such instances and supports previous statements made regarding model choice (some ef- fects are only observed on RoBERTa) and dataset type (both datasets are single-sentence tasks). The low number of table entries further justifies that there other confounding variables, including but not limited to model choice, hyperparameter selection, and dataset interactions.
Dataset type The inference datasets (QNLI, RTE) do not have much significant impact on the set of probing dimensions. In all cases, both datasets have no more than two significant dimen- sions (see Tables 2 and 3). This is far fewer than single-sentence tasks (COLA, SST2), whose ef- feets span many more dimensions (see Tables and 3). We hypothesize that this can be attributed to the fact that SentEval probing tasks assess lin- guistic information captured on the sentence level. By fine-tuning on single- single-sentence datasets, mod- els are more likely to learn these relevant sentence properties and incorporate them in their embed- dings. Inference datasets are more complex and require models to learn linguistic properties be- yond the sentence scope, such as semantic rela- tionships between sentences. Similarity and para- phrase datasets fall in between single-sentence and inference datascts with respect to complexity and linguistic scope, which explains why MPRC and STSB impact more dimensions than QNLI and RTE but fewer dimensions than COLA and SST2.
Spill-over We observed that some syntactic datasets have effects along the semantic dimensions and vice-versa. This is odd, as learning sentiment shouldn't be correlated to e.g., losing the ability to identify the syntactic constituents or swapped word order. We refer to such effects as "spill-over" ef-
Reference state
In most cases, significant dataset dimensions varied with different reference states. From Table 4, is clear that significant individual effects of COLA are inconsistent between experi-

---

Dataset
Reference
Model
Length
Depth
TopConst
BigramShift
Tense
SubjNumber
ObjNumber
OddManOut
CoardIay
COLA
BERT
1.32
0.8
-2.4
0.32
-1.14
-0.02
-0.34
3,56
0.3
SST2
1.16
7,4
COLA
MRPC
BERT
-0.6
0.78
-000
5.6
0.44
0.14
0.18
0.56
0.92
COLA
STSB
BERT
-284
-1.0
-0.8
6.48
0.94
0.08
-1.94
3.34
3.54
1.24
-1.26
-1.26
7,2
-0.96
COLA
RTE
BERT
1.5
- -1,82*
-3.3
5,94
-026
2.02
-0.88
2.16
0.96
COLA
MRPC QNLI
BERT
0.62
0.46
-2.66
5.22
-1.08
-0.02
0.466
1.12
0.28
-0.64
5.52
1.16
COLA
STSB QNLI
BERT
0032
-0.2
1.12
6.56
0.24
- 1.26
0.44
1,38
0.38
COLA
STSB RTB
BERT
0.18
0.26
0.92
7.24
0.2
-0.7
1.22
1.08
1.08
COLA
RoBERT&
2.18
1:21
8 82
5.78"'
0:28
0.44
-1.78'
1.61"
2.7"
COLA
SST2
RoBERTs
-0.66
2.18
6.18
13.08
2.44
3.28
2.42
4.4
4.52"
COLA
MRFC
RoBERIa
-3.18"
-0.34
1.58
7.9
0.14
0.16
0.78
2.62"
1.6
COLA
SISB
Ra
-3.84
11.9
3.46"
-D.1
4.6*
2.2
COLA
QNLI
RoBERTa
-3.68
-0.5
0.58
5,80
-0.1
1.88
0.96
3.54
3.02
COLA
RTE
RoBERTa
-1.82
1.04
6.04
7,78**
0.36
-0.3
0.28
4.18
1.28*
COLA
QNLI
Ro
-5.58"
8,8
0.78
1.0
2,18"
COLA
MRPC RTE
RoBERTa
0.56
1.34
3.1
8.64
3.52"
1.54
2.54
3,5"
0.62
COLA
STSB QNLI
RoBERTa
-1.04
0.38
0.58
11.02
1.22
0.28
1.28
3.4
0.1
Ro
-5.12
10.96"
0.22
Table 4: Individual effects of COLA dataset with different reference states. * and stand for p 0.05. < 0.01, and < 0.001, respectively, computed using two-sample t-test with dof = 8.
Dataset
Dimension
Effect
Model(s)
out of 30) scenarios listed in Tables 6 and Table 7, the significant interactions concentrate on no more than three dimensions. The remaining two scenarios are both SST2 and RTE (on RoBERTa and BERT, respectively).
COLA
BigramShift
Both
COLA
OddManOut
RoBERTa Both
SST2
Length
SST2
TopConst
RoBERTa
SST2
BigramShift
Both
Interaction effects occur with insignificant individual effects Many interaction effects observed along the linguistic dimensions where neither of the participating datasets has a signif- icant individual effect. For example, all signifi- cant interactions along the "Depth" dimension for BERT have this characteristic. Apparently, even if dataset does not have significant individual effect along dimension, it could still interact with the other dataset along this dimension.
Table 5: Individual effects that are observed in at least 70% of reference states.
fects. For instance, Tables and 3 suggest that fine- tuning on COLA (a syntactic dataset) has positive effect on OddManOut and CoordInv (semantic di- mensions). This is unexpected, given OddManOut and CoordInv probing datasets consist of gram- matically acceptable sentences, they only violate semantic rules. Conversely, fine-tuning on SST2 (a semantic dataset) hurts TopConst and BigramShift (syntactic or surface-for forracce-ormm dimensions). We hypoth- esize that the spill-over individual effects are likely due to the aforementioned confounding variables inducing false correlations. More rigorous analy- sis is required to identify these variables and their effects.
Interaction effects are characteristic The sig- nificant dimensions differ across the datasets. Even the datasets with similar purposes do not demon- strate identical interaction effects with the same "third-party" dataset. For example, both MRPC and STSB target detecting paraphrasing. When in- teracting with COLA, STSB has an insignificant effect on the TopConst dimension, while MRPC has significant negative effect along the same mension. Can we predict the dimensions of the significant effects, just by analyzing the datasets? Rigorous, systematic studies in the future are nec- essary to give deterministic answers.
6.2 Interaction effect
The average interaction effects of datasets on RoBERTa and BERT are listed in Table and Table 7, respectively.
Interaction effects are concentrated The inter- action effects are not always significant along the probing dimensions. More specifically, in most (28
Similar datasets interact less The interactions of similar datasets appear to interact less signifi- cantly than those "less similar" datasets. Among

---

Dataset effect dimensions
X
Model
Length
Depth
TopConst
BigrumShift
Tense
SubjNumber
ObjNumber
OddManOut
Coordlav
COLA
SST2
RoBERTa
-2.8
0.94
-244
7.30"
2.16
2.84
4.20'
-0.24
1.82
COLA
MRPC
RoBERTa
-7.00
-158
-10.20
2,16"
0.14
-0.28
2.56
-2.02
-1.10
COLA
STSB
RoBERTa
6602"
-221"
-4.8
6.12
3.18
0.54
-0.50
COLA
QNLI
RoBERTa
-5.86"
-1.74
-8.24
0.08
0.38
-2.22
0.82
-0.00
0.32
COLA
RTE
RoBERTa
-1.00
-0.20
-2.88
2,00"
0.08
-..4
2.06
00.66
1.58
SST2
MRFC
RoBERTa
-3.74"
0.96
-2.72
5.61"
1,98
1-40"
3.74'
0.88
-0.50
SST2
STSB
RoBERTa
-2.80
0.06
0.36
6.6(**
4.40"
3.00"
SST2
QNLI
RoBERTa
0.18
1.02
0.18
2.12
2.64
3.08
3.84
0.60
2.82"
SST2
RTE
RoBERTa
-0.20
1.48
8.14"
5.78
4.68
5.56
5.48
0.50
5.96
MRPC
STSB
RoBERTa
-3.34
-3.11"
-1008
3.81"
3.58
2 2.96
0.50
-1.92
-0.88
MRPC
QNLI
RoBERTa
-2.01
-1.11
-4.4
0.64
0.12
0.32
1.44
0.90
1.06
MRPC
RTE
RoBERTa
-6.58"
-1.88*
-3.10
1.88
-1.48
0.78
0.04
-0.10
1.72
STSB
QNLI
RoBERTa
0.12
-242
-3.10
1.18
2.78
0.51
0,78
0.11
1.16
STSB
RTE
RoBERTa
6.28*
3.28"
3.12
1.84
0.26
2.00
QNLI
RTE
ROBERTH
-0.0
-0.74
-0.68
0.96
0.06
0.76
1.30
0.40
1.96
Table 6: Interaction effects between GLUE datasets, on RoBERTa. *, ** andd stand for < 0.05, p 0.01, and p < 0.001, respectively, from the ANOVA test of the interaction effect.
the 30 scenarios in Tables 6 and 7, only two sce- narios show significant interactions along any dimensions: (MRPC, QNLI) and (QNLI, RTE), both on RoBERTa, and both involve strong simi- larities between the datasets: QNLI and RTE test the same downstream task (infer the textual en- tailment), and MRPC and QNLI both involve an intricate understanding of the semantics of the text,
the AI community.
From data difficulty to "dataset effects" While the difficulty of the dataset is a uni-dimensional score, the effect of datasets can be multi- dimensional. Improving the difficulty of datasets (c.g., by identifying adversarial examples and chal- lenging datasets) has been shown to improve perfor- mance (Ribeiro et al., 2020; Gardner et al., 2020). The considcration muuti-ddimessooall dataset fects can potentially introduce similar benefits.
7 Discussion
Checking the effects before using datasets Con- sidering that the datasets show spill-over effeets when used independently and interaction effects when used jointly, we call for more careful scrutiny of datasets. While the model developers already have a busy working pipeline, we call for model developers to at least be aware of spill-over effects of the datasets. Considering the possibly negative effects to the models' linguistic abilities, adding datasets to the model's training might not always be beneficial.
Conclusion
We propose state-vector framework to study dataset effects. The framework uses probing clas- sifiers to describe the effects of datasets on the resulting models along multiple linguistic ability dimensions. This framework allows us identify the individual effects and the interaction effects of a number of datasets. With extensive experiments, we find that the dataset effects are concentrated and characteristic. Additionally, we discuss how the state-vector framework to study dataset effects can improve the dataset curation practice and responsi- ble model development workflow.
Documentation for datasets The transparency of model development pipelines can be improved, crucial
and better documentation of the data improvement area (Gebru et al., 2021; Paullada et al., 2021). Recently, Pushkarna et al. (2022) de- scribed some principles for unified documentation of datasets: flexible, modular, extensible, accessi- ble, and content-agnostic. The dataset effect can be module in the dataset documentation. In addition documenting the basic properties of the datasets, it would be great to also note how the dataset has potential "spill-over" effects and "interaction ef- fects". This is better done via joint effort from
Limitations
Probing tests may not be idealized When for- mulating the framework, consider idealized probes 100% valid and reliable. In reality, prob- ing tests are unfortunately not ideal yet. We follow the common practice of setting up the probing clas- sifiers to allow fair comparison with their literature.

---

Dataset effect dimensions
X
Model
Length
Depth
TopConst
BigramShin
Tense
SubjNumher
ObjNumber
OddMaamQut
Courdhny
COLA
SST2
BERT
0.16
-2.00"
2.94
1.08*
0.44
0.24
-0.36
-1.34
1.32
COLA
MRPC
BERT
0.72
-0.02
2,88
-0.72
1.58"
0.16
0.16
-3.00
0.62
COLA
STSB
BERT
-1.52
-1.80
2.14
0.16
2.08
0.10
-1.60
-0.22
3.24
COLA
QNLI
BERT
2.56
-2.06*
1.68
0.88
0,90
-0.94
-0.32
-1.10
0.28
COLA
RTE
BERT
2.82
2.62
0.36
0.38
0.88
2.00
0.34
-1.40
0.66
SST2
MRFC
BERT
0.66
-0.32
2,74
-0.56
1.08
0.60
-1.62
-2.10"
0.16
SST2
STSB
BERT
00.70
-0.90
1.50
-.82
1.12
-1.56
-3.06
-0.52
2,80'
SST2
QNLI
BERT
0.36
-0.58
0.06
0.46
0.68
-2.00
-2.00"
-3.04
-1.28
SST2
RIE
BERT
1.82
-2.26"
-2.02
-062
0.64
-2.96"
-2.00"
-2.00"
-1.30
MRPC
STSB
BERT
2.00
0.48
5.48"
0.94
2.22"
2.12"
2.36
0.12
3.42"
MRPC
QNLI
BERT
-0.28
-1.10
4.42"
0.50
1.76"
1.04
-0.94
-1.50
-0.36
MRPC
RTE
BERT
2.38
-0.48
0.34
-0.41
1.42"
22.88"
-2.31"
22.00
-0.32
STSB
QNLI
BERT
-0.48
-0.58
1.28
0.46
1.44"
0.14
-3.18*
0.96
1.70"
STSB
RTE
BERT
1.06
-1.88
0.30
-0.82
1.70"
-1.00
-3.08
2:20
QNLI
RTE
BERT
3.46
1:32
2.12
0.12
0.36
3.38
2.76
0.70
1.20
Table 7: Interaction effects between GLUE datasets, on BERT. ", and *** stand for p < 0.05, < 0.01, and p 0.001, respectively, for the ANOVA test of the interaction effect.
We run the probing experiments on multiple ran- dom seeds to reduce the impacts of randomness.
work naturally generalizes to other tasks, including cloze and next-sentence prediction, the empirical observations are left to future works as well. We consider the fine-tuning setting in the experiments. Another setting, language model pre-training, also involves classification-type tasks and usually has larger sets of labels. Our theoretical framework generalizes the pre-training setting as well.
Model training may not be optimal Empiri- cally, the datasets included in our analyses are lim- ited to the fine-tuning stage. Previous work found distinct "stages" during the training of DNNs where the DNNs respond to the data samples differently. For example, Shwartz-Ziv and Tishby (2017) re- ferred to the stages as "drift phase" and the "diffu- sion phase". The means of the gradients are dras- tically different between the two stages. Tänzer et al. (2022) identified a "second stage" where the models do not overfit to noisy data labels. In the framework of this paper, we consider the ideal model training, where our states are defined as the global optimum where the model arrives.
References
Stéphane Coecl-Ouellette and Frank Rudzicz. 2020. On Losses for Modern Language Models. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4970 4981, Online. Paline.Assee Association for Computational Linguistics.
Interaction effects of more than two tasks The interaction effect is defined between two tasks. We believe this framework can generalize more than two tasks, but the empirical verification is left to future work.
Yonatan Belinkov, 2021. Probing Classifiers: Promises, Shortcomings, and Alternatives. arXiv:2102.12452.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009, The fifth pascal recognizing textual entailment challenge. TAC. 7:8.
Coverage of experiments As the number of datasets we consider increases, the number ex- periments in total grows exponentially. It is unre- alistic to go through the set of all combinations in our experiments, so we picked some experiments and organized them given the categories of the de- sired effects (instead of the observed effects) of the datasets. Additional experiments that test the exact interaction effects are left to future works. Also, we only considered classification-type tasks in the experiments. While this seate-vector frame-
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Ad- vances Neural Information Processing Systems. volume 33, pages 1877-1901.

---

Dallas Card, Peter Henderson, Urvashi Khandelwal. Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020. With Little Power Comes Great Responsibility. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9263-9274 Online. Association for Computa- tional Linguisties.
textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1-9.
Abhijeet Gupta, Gemma Boleda, Marco Baroni, and Sebastian Padó. 2015. Distributional vectors encode referential attributes. In Proceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing, pages 12-21, Lishon, Portugal. Association for Computational Linguistics.
Daniel Cer, Mona Diab, Eneko Agire,, lñigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-2011 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 1 11th International Workshop on Semantic Evaluation (SemEval-20177, pages 1-14, Vancouver, Canada, Association for Computational Linguisties.
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Damilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising textual entail- ment challenge. In Proceedings of the Second PAS- CAL Challenges Workshop on Recognising Textual Entailment, volume pages 785-794.
Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representa- tions. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA))
John Hewitt and Percy Liang. 2019. Designing and in- terpreting prohes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLA) pages 2733-2743, Hong Kong, China. Association for Computational Linguistics.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Machine learning challenges workshop. pages 177-190. Springer.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch. Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Train- ing compute- optimal large language models. arXiv preprint arXiv:2203. 15556.
Jacob Devlin, Ming-Wi Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-41888 Minneapolis, Minnesota. Association for Computational Linguistics.
Hyeon Jeon, Michael Aupetit, DongHwa Shin, Aeri Cho, Scokhyeon Park, and Jinwook Sco. 2022. San- ity check for external clustering validation bench- marks using internal validation measures. arXiv preprint arXiv:2209. 10042.
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing corpus of sentential paraphrases. Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2022. Understanding dataset difficulty with V-usable information. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine PMLR.
Leo Z. Liu, Yizhong Wang. Jungo Kasai, Hannaneh Ha- jishirzi, and Noah A. Smith. 2021. Probing Across Time: What Does RoBERTa Know and When? arXiv:2104.0758 [cs].
Learning Research, pages
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretrain- ing Approach. arXiv: 1907.1 1692 [es].
Mat Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. 2020. Evaluating models' local decision boundaries via contrast sets. arXiv preprint arXiv:2004.02709.
Marius Mosbach, Anna Khokhlova, Michael A. Hed- derich, and Dietrich Klakow. 2020. On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformers. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 2502-25166 Online. Association for Computational Linguisties.
Timnit Gebru, Jamie Morgenstern, Briana Vec- chione, Jennifer Wortman Vaughan. Hanna Wallach, Hal Daumé lii, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM, 64(12):86 92.
Jingcheng Niu, Wenjie Lu, and Gerald Penn. 2022. Does BERT rediscover the classical NLP pipeline? In Proceedings of the 29th International Conference
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. 2007. The third pascal recognizing

---

on Computational Linguisties. International Commit- tee on Computational Linguistics.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannanch Hajishirzi. Noah A. Smith, and Yejin Choi. 2020. Dataset carlography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275-9293, Online. Association for Computa- tional Linguistics.
Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. 2021. Data and its (dis)contents: A survey of dataset development and use in machine learning research. Patterns, 2(1 10:3033..
Tiago Pimentel and Ryan Cotterell. 2021. A Bayesian framework for information- thecoretc probing. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 2869- 2887. Online and Punta Cana. Dominican Republic. Association for Computational Linguistics.
Michael Tänzer, Sebastian Ruder, and Marek Rei. 2022. Memorisation versus generalisation in pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Pupers), pages 7564-7578, Dublin, Ireland. Association for Computational Lin- guistics.
Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjar- tansson. 2022. Data Cards: Purposeful and Trans- parent Dataset Documentation for Responsible AL. arXiv preprint urXiv:2204.01075.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT Rediscovers the Classical NLPP Pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4593- 4601, Florence, Italy. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, and others. 2018. Improving language understanding by generative pre-training. Publisher: OpenAT.
Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. 2022. Will we run out of data? An analysis of the limits of scaling datasets in Machine Learning.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Elena Voita and Ivan Titov. 2020. coormatiinnthhorreee probing with minimum description length. In Pro- ceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 183-196, Online. Association for Computa- tional Linguisties.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond Accuracy: Be- havioral Testing of NLP Models with CheckList. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 4902- 4912, Online. Association for Computational Lin- guistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, 2018. GLUB: A multi-tar task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Con- putational Linguistics.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A Primer in BERTology: What We Know About How BERT Works. Transactions of the Asso- ciation for Computational Linguistics, 8:842-866.
Alex Warstadt, Amanpreet Singh, and Samuel R Bow- man. 2018. Neural network acceptability judgments. arXiv preprint 1rXiv:1805.122 12471.
Ravid Shwartz-Ziv and Naftali Tishby, 2017. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv: 1703.00810.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.01652.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013, Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Orion Weller, Kevin Seppi, and Matt Gardner. 2022. When to use multi-task learning vs intermediate fine- tuning for pre-trained encoder transfer learning. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 272 282, Dublin, Ireland. As- sociation for Computational Linguistics.
Yu Sun. Shuohuan Wang, Shikun Feng. Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yan- hin Zhao, Yuxiang Lu, and others. 2021. ERNIE 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiy preprint arX
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen,

---

Clara Ma. Yacine Jernite. Julien Plu, Canwen Xu. Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stew- art, and Stefano Ermon. 2020. A theory of usable information under computational constraints. ICLR.
Zining Zhu, Soroosh Shahtalebi, and Frank Rudzicz. 2022a. Predicting fine-tuning performance with proh- ing. In EMNLP.
Zining Zhu, Jixuan Wang, Bai Li, and Frank Rudzicz. 2022b. the data requirements probing. Find- ings of the Association of Computational Linguistics. Association for Computational Linguistics.

---

A Appendix
Marker
N. Groups
N. Tasks
Experiments
A.1 Figure illustrating the experimental setup
A
B
3
C
2
12
Multi-task Model
D
3
8
E
2
2
3
Task Model
Task Model
Task Model
F
3
2
Table 8: Multitask states with counts of groups, tasks per group, and experiments per encoder.
Task Head
Task Head 2
Task Head
illustrate potential individual and interaction effects clearly.
Encoder (shared)
Encoder (shared)
Encoder (shared)
To address these issues, we impose the follow- ing condition: experiments must constitute of equal task counts per task group OR all tasks must belong to the same task group. Recall the task groups from Section 5.2 to be single-sentence, similarity and paraphrase, and inference. This en- ables us to organize the experiments by marking the states as follows:
map from task
Figure 4: Multitask model architecture for fine-tuning experiments."
A.2 Additional details on multitask settings
1: The initial state.
A: The model is trained on one dataset.
Consider the number of different multitask fine-
B: The model is trained on two datasets from
tuning settings possible assuming constant ran- dom seed. If we fine-tune encoders task-by-task (e.g., first COLA, then MRPC) such that order mat- ters, then this problem is equivalent the number of ordered subsets that can be formed from the set = {COLA, SST2, MRPC, STSB, QNLI, RTE). This computes to En nJ 1957 models per encoder, or 1957 2 = 3914 4 total models. Note the lower bound of the summation is 0, as the empty set corresponds to the baseline model (i.e., no fine- tuning).
the same group.
C: The model is trained on two datasets from different groups
D: The model is trained on three datasets from different groups.
E: The model is trained on four tasks from two groups (two per group).
F: The model is trained on six tasks from three groups (two per group)
As demonstrated in Table 8. the total number of models we need to train reduced 34-2-5 340. Designing experiments this way allows framing the dataset effects.
If we discount ordering effects, then the prob- lem reduces the number of subsets T contains, This evaluates to 26 models per encoder or 64-2 = 128 total models, which results in far fewer experiments. Ordering effects can be discounted by aggregating training samples of each task, then ran- domly sampling from this combined dataset when training. Each sample will have tag indicating which task it pertains to, it be redirected to the correct classification head during training.
The individual effects can be framed as transi- tions between marked states (i.e. adding some task to one state yields another state). For example, - A can reflect the individual effect of a dataset, conditioned on the ino-fine-tuning initial state" J.
A B denotes the individual effect of a dataset X, conditioned on the initial state that contains a dataset (Y), where in the same group as X.
Note that we initially assume a constant random seed. Later, we expanded to five random seeds (42, 1, 1234, 123, 10) to allow statistical significance testing. This increases total models to 128-5 = 641. which requires excessive compute resources. There is also the need to organize experiments better to
The interaction effects can be framed as com- binations between multiple states. For example, two states labeled A (with dataset X and Y, spectively) and state labeled B (with datasets [X, YJ) can jointly define the interaction between the datasets and Y. This can be written as B A A.
'Experimental setup hased on this guide

---

This labeling mechanism of the states can sup- port the following effects:
A.4
On the equivalence between formulations of interaction effects
Plugging in the indicator variables into Eq. 4 yields the following equation (writing in matrix form):
Individual effects: A, B, -C
C++ D
Interaction effects: A+ A,C A+A,
8(1)
1
0
0
30
D A C. E = B B, E C+C.
S((X,
1
0
0
31
F =
S((,1)) S((X)),
1
1
0
32
+
1
1
Note that although it is possible to compute in- teraction effects of more than two datasets, chose not to focus these cases as it adds extra layer of complexity. Hence, we only con- sider the following interaction effects: B A+ A,
(5)
The standard variable elimination operations give us expression for the expression for the interac- tion effect parameter Bz:
= A + A. This means we don't need to train models for states and F, reducing total experi- ments to 30 2 300.
B3 = S([X,Y,]]) - -((X S(Y, +
(6)
which exactly recovers the definition for Int( aut(X,) Y) using the equivalent formulation (Eq. 3).
A.3 Additional math motivation
A.5 Additional experiment results
Tables 9 present some additional experiment results.
Here we provide some additional mathematical mo- tivations for the proposed state-vector framework: dataset effects form Abelian group.
Given a reference state Sr, the collection of all possible dataset effects (XX) E forms an addi- tive Abelian group. Here we show that & satisfies the requirements.
Existence of zero. We already know that the identity element is E K Intuitively, the identity element corresponds to "no effect" for this dataset.
Existence and closure of addition. The addition operation refers to the vector addition. Since E is defined under K it is closed under addition. Note that due to the interaction effect, addition does not refer to applying two datasets together to the "bucket" of data for multitask training.
Existence and closure of negation. The negation operation refers to flipping the direction of vec- tor in E. Empirically, negating counterfactual query of the effect of a dataset: if X were not applied, what would have been the effect the state of the model?
E(X) involves
Associativity and commutativity. Vector addition satisfies associativity: (E(X) ) E(Y)) E E(Z) X(X) + (E(Y)) + E())) and commutativity:
E E(X) + E(Y) = EYY + E(X).

---

Length
Depth
TopConst
RigramShift
SubjNumber
ObjNumher
SST2
I
BERT
4.58
0.3
4.94
-188
-0.88
00.6"
0.0
1.22
0.0
SST2
COLA
BERT
-142"
-1.7
-20
-0.78"
-0.11
-0.62
-0.3
-0.12
1.52
S512
МRРС
BERT
-3.2"
-0.02
-22
-2.12"
0.2
-0.6
-1.62
-.88
SST2
STSB
BERT
-5.29
-0.6
-3.44
-
0.24
2..2"
-3.66
0.7
2.8'
SS12
QNLI
BERT
-1.22"
-0.28
-188"
-1.1
-0.2
-3.16"
-28"
-182
-1.28
S512
BERT
-2.66"
-1.9
-6.96
-2.88
-024
-3.82"
-20"
1.39
-1.3
SST2
MRPC QNLI
BERT
-2.2
0.34
-1.2"
2.08"
-0.66
1.9
-146
1.5
-0.58
S812
MRPC KIB
BERT
-1.22"
-0.84
-2.
-2.88"
-0.58
0.5
-1.88
-1.35*
-036
QNLI
SST2
STSB RTE
BERT
5.48"
1.04
-.66"
2.24
-0.22
2.56'
2.48"
2.36
-1.12
SST2
RoBERTa
55.44"
-2.84
5.7
-8.78
-100"*
-7 -7.28
88.0
-232
33.6
1.9*
-8.14"
-1.84
3.80
1.94
SST2
MRPC
RoBERTa
-9.18
1.88
8.42
3.12
2202
2,58
4,32 "
1.44
4,26
SST2
STSB
RoBERTa
-
-2.78"
5.31"
-2.12
0.4
3308
-,46
11.88
-2.22"
QNLJ
-6.44
-0.4
SST2
RTE
RoBERTa
-5.64
-136
2.44
-
0.68
1.72
-258
1.82
2.2"
SST2
MRPC QNLI
RoBERTa
-7.1"
-2.26"
-7 -7991
-1.6"
-2.12"
-1.8
-2.3
-3.2"
-2.58
-0.58
6.96'
SST2
STSB QNLI
RoBERTa
-7.81"
-0.86
-35
-3.22
-1.7
-236
-212
-196
-1.91
SS12
STSB RTE
RoBERia
-9.76
-2314
-9722"
-114"
-2.18
-55
-288
-118
-1.61
Table Individual effects of SST2 dataset with different reference states. ", and *** stand for p < 0.05, p < 0.01, and p < 0.001, respectively, for two-sample f-test with dof 8.


Reviews:
[{"review_id": "dTgxrkld99", "reviewer": "Reviewer_PRSZ", "summary": "", "questions": "Line 96-97: What does dataset with complex decision boundaries mean? Ideally a model has decision boundaries.\n\nTable 2,3,4: Are the numbers shown here difference in accuracy levels for each of the probes across 3 epochs?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the reviews! To the points mentioned in weaknesses and questions:\n\n- Decision boundary: it should be \"the optimal models trained on this dataset\". We will make the usage of terminology more precise.\n- Tables 2-4 show individual effects, which are the differences in the mean accuracy levels across random seeds. Since there is some randomness in the probing procedure, we take the average of the accuracy to approximate the accuracy of the global optimum.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Well written paper with simple mathematically grounded formulation.\n    \n2. New Framework for analysing effects of datasets. Such a probing state-vector framework is helpful in understanding the effects of a dataset along multiple dimensions (probes) rather than just having a single “qualitative” score.\n    \n3. Comprehensive evaluation on 300 fine-tuning experiments per model.\n    \n4. Provides guidelines as to how the introduction of new dataset into the task might affect the performance of the models on different tasks.", "reasons_to_reject": "1. It would help to show the working of how individual effects were calculated in the tables 2,3,4.", "questions_to_authors": "Line 96-97: What does dataset with complex decision boundaries mean? Ideally a model has decision boundaries.\n\nTable 2,3,4: Are the numbers shown here difference in accuracy levels for each of the probes across 3 epochs?", "paper_topic_and_main_contributions": "This paper studies the effect of using multiple datasets on models using a state-vector based probing framework. Such a framework helps in quantifying not only the effect of single or standalone dataset but also interacting datasets. The paper addresses important questions about dataset effects as well as provides how the framework can be useful in curating datasets for responsible model development. Additionally the paper discusses spill-over effects that might creep in via the introduction of the new dataset for a task.\n\nThe paper provides comprehensive framework as well as analysis on standard classification tasks using the aforementioned framework.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 585: ther -> their"}, "extra_text": {}}, {"review_id": "1T2hZefI7d", "reviewer": "Reviewer_9zjj", "summary": "", "questions": "- Figure 3: What does the \"<\" shaped line represent?\n- plenty of related works for Table 2 and 3?\n- L481: \"aforementioned confounding variables\" to which variables do you refer to? Would you please elaborate?\n- L42: \"Model developers use large corpora that may include multiple existing datasets (Sun et al., 2021; Wei et al., 2021; Brown et al., 2020)\" I believe that the authors are discussing the pretraining phase, in which we mix the corpus from different domains (e.g., Wikipedia, News, and Webtext). Is this correct? If so, this sentence seems to be misleading, because this paper is mostly about finetuning, instead of pretraining.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed reviews! To the points mentioned in the review:\n\n- You are right in pointing out that downstream performance is not included. The downstream performance itself is not our focus of this paper. In most fine-tuning settings, we combine multiple datasets, so comparing the fine-tuning results to Zhu et al (2022a) would be apple-to-orange.\n- Variance of the results: there are a lot of them; we will include the raw results in the open-sourced github repo. While they appear large on the figures, these variances are considered when conducting the statistical tests (t-tests for individual effects, and ANOVA for interaction effects). When the statistical tests give p<0.05 (those results marked with stars in the tables — there are a lot of them), we consider the variances to be not very large.\n- Figure 3: the “<” shaped segments contain two parts: E(X) and E(Y). The intersection point is S(I).\n- L481: the confounding variables refer to those mentioned in L462. The model choice, hyperparameter selection, etc., can contribute to the confounding.\n- L42: You are right in pointing out that fine-tuning and pre-training are distinct phases. In either phase, people use the combination of multiple datasets. This paper’s experiments only cover the effects in the fine-tuning phase, but the theoretical framework can generalize to the pre-training phase (the difference is that LM pretraining datasets are usually larger, and have more labels than the fine-tuning datasets). We will add this point to the Limitations section.\n- Grammar styles, references to GLUE etc. and presentation points: Thanks. We’ll fix them in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- I agree that investigating the dataset effect is an important direction. Given that the recent LLMs have billions of parameters, we need a method for investigating the effectiveness of finetuning in advance for computational efficiency.\n- The authors have conducted thorough experiments on two language models (BERT and RoBERTa) with some heuristics to reduce the search space.", "reasons_to_reject": "### Downstream performance is completely ignored\n\nThis paper almost completely ignores the downstream performance (e.g., accuracy and F1 score) of finetuning tasks through the entire experiments.\nI acknowledge that linguistic abilities are important.\nHowever, I wish to see the discussion on downstream performance and how it relates to the findings in this paper.\nFor example, does a negative influence on linguistic abilities mean that model performs poorly in the target task?\nIs it consistent to the findings of the previous studies such as (Zhu et al, 2022a)?\nThis aspect is important because the authors regard model developers as one of their targets.\nI believe that it is likely that such developers are interested in downstream performance in addition to linguistic abilities.\nPlease note that I am not asking for the extra experiments; the performance can be reported because the authors have already finetuned the models.\n\n### Variance of Individual Effects and Interactions?\n\nIt seems that the authors have conducted the experiments using multiple random seeds. Is this true?\nDoes it mean that the authors report the average score on Table 6?\nIf so, I would like to see the variance value in the table.\nThis is because Figure 1 and 3 seems to present a high variance value; I am highly concerned if the findings in this paper can be reproduced using the other random seeds.", "questions_to_authors": "- Figure 3: What does the \"<\" shaped line represent?\n- plenty of related works for Table 2 and 3?\n- L481: \"aforementioned confounding variables\" to which variables do you refer to? Would you please elaborate?\n- L42: \"Model developers use large corpora that may include multiple existing datasets (Sun et al., 2021; Wei et al., 2021; Brown et al., 2020)\" I believe that the authors are discussing the pretraining phase, in which we mix the corpus from different domains (e.g., Wikipedia, News, and Webtext). Is this correct? If so, this sentence seems to be misleading, because this paper is mostly about finetuning, instead of pretraining.", "paper_topic_and_main_contributions": "This paper proposes a framework for analyzing the dataset effects, which enables us to investigate if training models on certain dataset(s) influences their linguistic abilities, e.g., sentence length prediction, syntactic tree depth prediction, etc.\n\nThe contribution of this paper  is as follows:\n\n- The authors propose a novel framework called State-Vector Framework. The framework allows us to investigate if certain datasets have under-specified (or undesirable) effects on the model's linguistic abilities\n- Given the proposed framework, the authors present some interesting findings, e.g., some datasets have negative influence to the abilities that seems unrelated to the task nature.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- If I understand the paper correctly, the act values in Tables 2,3,4,6  are not important. What is important here is the statistical significance. If this is the case, the presentation could be much improved if the authors use heatmap-like representations, in which each cell represents if statistical significance exists or not.\n- Please include Table 9 in the main text, instead of burying it into the Appendix. I believe this is necessary to ensure that the paper is self-contained.\n- L740: ai --> AI\n- Throughout the table: roberta should be RoBERTa and bert should be BERT for consistency."}}, {"review_id": "OZmk810L3O", "reviewer": "Reviewer_6EYi", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your reviews! The transfer learning setting is indeed a very interesting scenario. This scenario will be considered in future works. We will fix the typo in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors introduced a novel vector-based framework aimed at assessing the effects of datasets and their interactions on each other.\nThe notable advantage of this framework lies in its high level of generality, making it capable of potentially assessing the impacts of a vast array of datasets on different probing dimensions.\n\nThe experiments are quite extensive spanning over 2 models, 3 types of datasets, containing 2 datasets per type, and 9 probing tasks.\nThe authors systematically discuss their findings. Some of them are intriguing, such as the \"spill-over\" effect that indicates that certain syntactic datasets exhibit impacts along the semantic dimensions and vice-versa. They also discovered that Interaction effects are concentrated in a small number of dimensions and that similar datasets interact less with each other.\n\nOverall, this is a well-constructed paper that has the potential to significantly influence future research in measuring dataset transferability and the impact of a set of datasets on a new collection.", "reasons_to_reject": "I find the general framework of vector-space to be quite redundant with respect to the conducted experiments. Ultimately, the authors showcase two categories of tables: the first measures the difference in the accuracy of each probing task before and after fine-tuning a model on a specific dataset, while the second gauges the difference between the accuracy of each probing task after jointly fine-tuning a model on two datasets and the sum of accuracies obtained by fine-tuning the model on each dataset separately. The equations that correspond to these two metrics apply basic linear algebra (\\probability theory).\n\nSecondly, although the authors used a well-known public probing dataset (SentEval), I found the probing tasks to be rather uninteresting, in the sense that it is quite hard to yield highly impactful findings from them. This is attributed to the limited correlation between the various tasks. As a result, the papers' findings have a broad scope but do not unveil any captivating discoveries within the group of probing tasks.\n\nWithin the provided framework, it would have been intriguing to include at least one result on transfer learning, specifically, examining the effect of first fine-tuning a model on one dataset and then fine-tuning it again on a second dataset. I am of the opinion that the paper could gain valuable insights from such a discussion.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a framework for quantifying the effects of single datasets as well as the interaction effects between sets of datasets for a given model. The framework is formulated as a vector-space, where the i'th dimension of a dataset corresponds to its (general) score on a given probing task.\n\nThe strength of this framework lies in its generality, as it does not condense into a singular real-value score for each dataset. Although, I might say that the usage of vector-space is quite redundant in this case as all effects were introduced and computed by very basic linear algebra/probability operations.\n\nThe authors conducted experiments on 2 models (bert and roberta), 3 types of datasets, containing 2 datasets per type on 9 probing tasks.\nThe experiments measured the effect of each dataset as well as the interactions between each pair of datasets on the 9 probing tasks.\nThe authors included an extensive discussion of the results, highlighting some interesting findings, such as what they refer to as the \"spill-over\" effect. This phenomenon indicates that certain syntactic datasets exhibit impacts along the semantic dimensions and vice-versa.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Lines 389-392: \"Tables 2 and 3 (in Appendix)... Tables 4 and 8 break down ...\" -> only Table 8 appears in the Appendix."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a framework for analyzing the effect of a dataset on a model's linguistic abilities, as quantified by probing task performance. The framework allows us to investigate if certain datasets have undesirable effects on the model's linguistic abilities. An interesting finding from the work is that some datasets negatively influence abilities that seem unrelated to the task nature.

Overall, the reviewers agree that the soundness of the proposed work is sufficient to support its main arguments, though several reviewers point out additional points that should be addressed in a revision. Specifically, a common note among reviewers is to clarify the results in Tables 2-4 relating to individual effects and display a measure of the variance in a more prominent way. Another limitation that is brought up is that downstream task performance is not reported. Even though downstream task performance is not the focus of the work, it's important to report as the degree of encoded linguistic abilities may relate to the degree of fine-tuning success. We recommend that these points are addressed in a revision. On the positive side, reviewers praised the clarity of the writing and thought the research question to be of significance to the EMNLP audience.
Decision: Accept-Main