SpanDrop: Simple and Effective Counterfactual Learning for Long Sequences
Authors: Peng Qi
Guangtao Wang
Jing Huang
Venue: ICLR 2022 Submitted (2022)
Keywords: sequential data
sample efficiency
data augmentation

Abstract: Distilling supervision signal from a long sequence to make predictions is a challenging task in machine learning, especially when not all elements in the input sequence contribute equally to the desired output. In this paper, we propose SpanDrop, a simple and effective data augmentation technique that helps models identify the true supervision signal in a long sequence with very few examples. By directly manipulating the input sequence, SpanDrop randomly ablates parts of the sequence at a time and ask the model to perform the same task to emulate counterfactual learning and achieve input attribution. Based on theoretical analysis of its properties, we also propose a variant of SpanDrop based on the beta-Bernoulli distribution, which yields diverse augmented sequences while providing a learning objective that is more consistent with the original dataset. We demonstrate the effectiveness of SpanDrop on a set of carefully designed toy tasks, as well as various natural language processing tasks that require reasoning over long sequences to arrive at the correct answer, and show that it helps models improve performance both when data is scarce and abundant.

Under review as a conference paper at ICLR 2022
SPANDROP: SIMPLE AND EFFECTIVE COUNTERFAC- TUAL LEARNING FOR LONG SEQUENCES
Anonymous authors Paper under double-blind review
ABSTRACT
Distilling supervision signal from long sequence make predictions is a chal- lenging task in machine learning, especially when not all elements in the input sequence contribute equally the desired output. In this paper, we propose SPAN- DROP, a simple and effective data augmentation technique that helps models iden- tify the true supervision signal in a long sequence with very few examples. By di- rectly manipulating the input sequence, SPANDROP randomly ablates parts of the sequence at a time and ask the model to perform the same task to emulate counter- factual learning and achieve input attribution. Based on theoretical analysis of its properties, we also propose variant of SPANDROP based on the beta-Bernoulli distribution, which yields diverse augmented sequences while providing learn- ing objective that is more consistent with the original dataset. We demonstrate the effectiveness of SPANDROP on set of carefully designed toy tasks, as well as various natural language processing tasks that require reasoning over long se- quenees to arrive at the correct answer, and show that it helps models improve performance both when data is scarce and abundant.
INTRODUCTION
Building effective machine learning systems for long sequences is challenging and important task, which helps us better understand underlying patterns in naturally occurring sequential data like long texts (Radford et al. 2019). protein sequences (Jumper et al. 202I). financial time series (Bao et al. 2017D, etc. Recently, there is growing interest in studying neural network models that can capture long-range correlations in sequential data with high computational, memory, and statistical efficiency, especially widely adopted Transformer models (Vaswani et al. 2017p.
Previous work approach long-sequence learning in Transformers largely by introducing computa- tional approaches to replace the attention mechanism with more efficient counterparts. These ap-
proaches include limiting the input range over which the attention mechanism is applied (Kitaev)
et al.
2019) to limiting scquence-leve attention to only handful of positions (Beltagy al. [020
Zaheer et al. 2020). Other researchers make use of techniques akin to the kernel trick to eliminate
the need to compute instantiate the costly attention matrix (Peng al., 2020 Katharopoulos al.)
2020 Choromanski et al. 2020). Essentially, these approaches aim to approximate the original pair-
wise interaction with lower cost, and are often interested in still capturing the interactions between every pair of input elements (e.g., the long sequence benchmark proposed by Tay et al.) 20200)
In this paper, instead investigate learning problems for long sequences where not all input ele- ments contribute equally to the desired output. Natural examples that take this form include senti- ment classification for long customer review documents (where a few salient sentiment words con- tribute the most), question answering from a large document (where each question typically requires small number of supporting sentences to answer), key phrase detection in audio processing (where small number of recorded frames actually determine the prediction), as well as detecting a specific object from a complex scene (where, similarly, a small amount of pixels determine the outcome), to name few. In these problems, it is usually counterproductive to try and make direct use of the entire input if the contributing portion is small or sparse, which results a problem of underspec- ification (i.e., the data does not sufficiently define the goal for statistical models). One approach to address this problem is annotating the segments or neighborhoods that directly contribute to the outcome in the entire input. This could take the form of a subset of sentences that answer a question

---

Under review as a conference paper at ICLR 2022
or describe the relation between entities in a paragraph (Ynn al. 2018 Yao al. 2019 which function as explainable evidence that supplements the answer. When such annotation is not feasible, researchers and practitioners often need to resort either collecting more onput-output pairs or de- signing problem-specific data augmentation techniques to make up for the data gap. For real-valued data, this often translates to random transformations (e.g., shifting or flipping an image); for sym- bolic data like natural language, techniques like masking or substitution are more commonly used (e.g. randomly swapping words with a special mask token or other words). While these approaches have proven effective in some tasks, each has limitations that prevents it from being well-suited for the underspecification scenario. For instance, while global feature transformations enhance group- invariance in learned representations, they do not directly help with better locating the underlying true stimulus. On the other hand, while replacement techniques like masking and substitution help ablate parts of the input, they are susceptible to the position bias of where the true stimulus might oc- cur in the input. Furthermore, while substitution techniques can help create challenging contrastive examples, it is significantly more difficult to design them for complex symbolic sequences (e.g., replacing a phrase naturally in a sentence).
To address these challenges, we propose SPANDROP, simple and effective technique that helps models distill sparse supervision signal from long sequences when the problem is underspecified. Similar to replacement-basee
techniques such masking and substitution, SPANDROP directly ab- lates parts of the input at random to construct counterfactual examples that preserve the original supervision signal with high probability. Instead of preserving the original sequence positions, how- ever, SPANDROP directly removes ablated elements from the input to mitigate any bias that is related to the absolute positions of elements (rather than the relative positions between them) in the input. Upon closer examination of its theoretical and empirical properties, we further propose more effec- tive variant of SPANDROP based on the Beta- Bernoulli distribution that enhances the consistency of the augmented objective function with the original one. We demonstrate via carefully designed toy experiments that SPANDROP not only helps models achieve up to 20 sample-effiiiency in low-data settings, but also further reduces overfitting even when training data is abundant. We find that it is very effective at mitigating position bias compared to replacement- based counterfactual approaches, and enhances -iitr-i-ittiittionn generalization effectively. We further experiments four natural language processing tasks that require models to answer question or extract entity relations from long texts, and demonstrate that SPANDROP can improve the performance of already competitive neural models without any change in model architecture.
2
METHOD
In this section, we first formulate the problem of sequence inference, where the model takes sequen- tial data as input to make predictions. Then, we introduce SPANDROP, simple and effective data augmentation technique for long sequence inference, and analyze its theoretical properties.
2.1 PROBLEM DEFINITION
Sequence Inference. We consider task where model takes sequence S as input and predicts the output y. We assume that consists of disjoint but contiguous spans, each representing a part of the sequence in order 5 = (SI...., sn). One example of sequence inference is sentiment classification from a paragraph of text, where S is the paragraph and y the desired sentiment label. Spans could be words, phrases, sentences, or a mixture of these in the paragraph. Another example is time series prediction, where is historical data, y is the value at the next time step.
assume that y is Supporting facts. Given input-output pair (S, y) for sequence prediction,
truly determined by only a subset of spans in S. More formally, we assume that there is a subset of spans Ssup {s1, $2,.... s, such that y is independent of s;, if s; Ssup- In sentiment classification, Ssup could consist of important sentiment words or conjunctions (like "good", "bad", "but"); in time series prediction, it could reflect the most recent time steps as well as those a few cycles away if the series is periodic. For simplicity, we will denote the size of this set m = [Ssupl- and restrict our attention to tasks where & n, such those described in the previous section.
2.2 SPANDROP
In a long sequence inference task with sparse support facts (m < n), most of the spans in the input sequence will not contribute the prediction of y. but they will introduce spurious correlation in low-data scenario. SPANDROP generates new data instances ($,y) y) by ablating these spans at
2

---

Under review as a conference paper at ICLR 2022
0.6
10
Log noise-frete free prob.
SPANDROP Rete-SrAN
2
10
Avg. nats/span
Proportion
0.1
0.4
0.2
0
0
50
100
10
20
10
102
10*
Sequence length (n')
supporting facts (rn)
Sequence length (n)
(a) Length Š = 100, p = 0.2) (b) supporting fact noise (p = 0.2)
(c) Typical set size (p 0.1)
Figure I: Theoretical comparison between SPANDROP and Deta-SPANDROP.
random, while preserving the supporting facts with high probability that the model is still trained to make the correct prediction y. This is akin to counterfactually determining whether each span truly determines the outcome y by asking what the prediction would have been without it.
Definition 1 (SPANDROP). Formally, given a sequence S that consists of spans (81, 82. s,), s.,))
SPANDROP generates new sequence S as follows:
6 f.dd ~ Bernoulli - pp)
$ =
(1)
where is the hyperparameter that determines the probability to drop a span.
Note that SPANDROP does not require introducing substitute spans artificial symbols when ab- lating spans from the input sequence. It makes the most of the natural sequence as it occurs in the original training data, and preserves the relative order between spans that often helpful in understanding sequential data (c.g., time series or text). It is also not difficult to es- tablish that the resulting sequence S can preserve all of the m supporting facts with high probability regardless of how large n is.
dropped, which is
Remark 1. The new sequence length n' = 1 and the number of preserved supporting facts m' |$ ( Soupl follow binomial distributions with parameters (n, p) and (m, p), respectively:
P(r'(n.p) = 71')
11
(1 (1-p)"
P(m' (m, =
m (1 p) pm m
m
(2)
Therefore, the proportion of sequences where all supporting facts are retained (i.e., m' = m) is
(1 -pm which is independent of n. This means that long the total number of supporting facts in the sequence bounded, then regardless of the sequence length, we can always choose p carefully such that we end up with many valid new examples with bounded noise introduced to supporting facts. Note that our analysis so far relies only on the assumption that m is known or can be estimated, and thus it can be applied tasks where the precise set of supporting facts Ssup is unknown. More formally, the amount of new examples can be characterized by the size of the typical set of 5, t.e set of sequences that the randomly ablated sequence will fall into with high probability. The size of the typical set for SPANDROP is approximately (p), where H (p) is the binary entropy of Bernoulli random variable with probability p. Intuitively, these results indicate that the amount of total counterfactual examples generated by SPANDROP scales exponentially in n, but the level of supporting fact noise can be bounded as long as m is small.
However, this formulation of SPANDROP does have notable drawback that could potentially hinder its efficacy. Because the new sequence length n' follows binomial distribution, its mean is (2(1p) and its variance is npp11 - p). For sufficiently large n, most of the resulting Š will have lengths that concentrate around the mean with width of O(vR). which creates artificial and permanent distribution drift from the original length (see Figure ((a) Furthermore, if we know the identity of Soup and keep these spans during training, this length reduction will bias the training set towards easier examples to locate spans in Saup- In the next subsection, we will introduce a variant of SPANDROP based the beta- Bernoll distribution that alleviates this issue.
3

---

Under review as a conference paper at ICLR 2022
2.3 BETA-SPAND PPTAAS
To address the problem of distribution drift with SPANDROP, we introduce a variant that is based on the beta-Bemoulli distribution. The main idea is that instead of dropping each span in a sequence independently with a fixed probability p. we first sample a sequence-level probability и at which spans are dropped from Beta distribution, then use this probability to perform SPANDROP.
Definition (Beta-SPANDRR
Let 7. 7 1-2, where ? is a scaling hyperparameter. Beta- Reta-SPANDROP generates S over S aas
π ,),
6, iid & Bernoulli(1 - m),
$ -8:6))]=1,66-
where B(a, B) is the beta-distribution with parameters a and
can be easily demonstrated that in PR.RRS the probability that each span is dropped is still controlled by p. same as in SPANDROP: [6 p] [[][|6] E[l x = a+ p. fact, we can show that as y + oo, Peta-SPANDROP degenerates into SPANDROP since the deta-distribution would assign all probability mass on implementation, Beta-SPANDROP is significantly less likely to introduce unwanted data distribution drift, while is capable of generating diverse counterfactual examples to regularize the training of sequence inference models. This due to the following properties:
Despite the simplicity in its
Remark 2. The new sequence length n' = S and the number of preserved supporting facts m' = |$ Suupl follow binomial distributions with parameters (n, B, a) and (m, B,a), respectively:
T(n + 1)
T(n' ((n'+,8)) +, n' + a) I(a + 3)
P(n'In,,, T(ne + In n' +1)
(4)
I(n+ a + (3) [(a)T(3))
T(m +1)
T(m' + [m m' +a) [(a
Pim'mm,,,11
T(m' + 1)m T(mm m' +1)
T(m+
r(a)r(8)
(5)
where [(z) = S0 se-Le-"dar is the gamma function.
As a result, we can show that the probability that Deta-SPANDROn preserves the entire original sequence with the following probability
I(n + ()I
P(n' n/n, a, B)
(6)
[(n + a + +)T(()))
When 7 L, this expression simply reduces to n+ when 7 * 1, this quantity tends O(n as n grows sufficiently large. Comparing this to the O(11 pp)") rate from SPANDROP, we can that when is large, Beta-SPANDROP recovers more of the original distribution represented by (S y) compared SPANDROP. In fact, as evidenced by Figure ((a) the counterfactual sequences generated by SPAND-S are also more spread-out in their length distribution besides covering the original length n with significantly higher probability. A similar analysis can be performed by substituting and nı' with m and m', where we can conclude that as m grows,
much better at generating counterfactual sequences that preserve the entire supporting fact set Saup- This is shown in Figure [(b) where the proportion of "noise-free" examples (i.e., m = m) decays exponentially with SPANDROP (7 oc) while remaining much higher when 7 is sufficiently small. For instance, when p 0.1, SPANDROP just 34.9%, while that for Seta-SPANDROP is 47.4%.
1 and 10, the proportion of noise-free examples for
As we have seen, Peta-SPAADDOO is significantly better than its Bernoulli counterpart at assigning probability mass to the original data as well as generated sequences that contain the entire set of sup- porting facts. A natural question is, does this come at the cost of diverse counterfactual examples? To answer this question and normalize it by n to study the size of typical set of this distribution. As can be seen in Figure I(c) long r is large enough, the average entropy per span H degrades very little from the theo- retical maximum, which is H (p), attained when 7 = co. Therefore, to balance between introducing noise in the supporting facts and generating diverse examples, we y = 1 in our experiments.
study the entropy of the distribution that Š follows by varying 7 and n,
Using the beta-Bernoulli distribution in dropout. The beta-Bernould distribution has been studied in prior work in seeking replacements for the (Bernoulii) dropout mechanism (Srivastava et al.
4

---

Under review as a conference paper at ICLR 2022
2014 Liu et al. (2019)) set |3 for the beta distribution in their formulation, which limits the dropout rate to always be 0.5. Lee et al. (2018) fix 3 = and vary a to control the sparsity of the result of dropout, which is similar to Beta-SPANDRoO when y 1. However, we note that these approaches (as with dropout) are focused more on adding noise to internal representations of neural networks to introduce regularization, while SPANDROP operates directly the input to ablate different components therein, and thus orthogonal (and potentially complementary) to these approaches. Further, SPANDROP has the benefit of not having to make any assumptions about the model or any changes to it during training, which makes it much more widely applicable.
FINDCATS: DISTILLING SUPERVISION FROM LONG- SENG-SE
In this section, we design a synthetic task of finding the animal name "cat" in character sequence to a) demonstrate the effectiveness of SPANDROP and Beta-SPANDROP in promoting the performance over a series of problems with different settings, b) analyze the various factors that may affect the efficacy of these approaches, and c) compare it to other counterfactual augmentation techniques like masking on mitigating position bias.
3.1 EXPERIMENTAL SETUP
To understand the effectiveness of SPANDROP and Beta-SPANDROP in experimental setting, we designed a synthetic task called FINDCATS where the model trained to discern that given an animal name "cat", whether character string contains it as a subsequence (i.e., contains characters in "cat" in order, for instance, "abedafgbijktma") or not (e.g., "abedefhtijllmn) This allows us easily control the total sequence length n, the supporting facts size m. as well easily estimate the supporting fact noise that each SPANDROP variant might introduce. To generate the synthetic training data of FINDCATS, we first generate sequence consisting of lowercase letters (a to z) that does not contain "cat" as subsequence. For half these sequences, we label the tuple (cat, 5) with negative class indicate that S does not contain "cat" as subsequence; for the other half, we choose arbitrary (but not necessarily contiguous) positions in S to replace the letters with letters in "cat" from left to right to generate positive examples.
FINDCATS.
In all of our experiments, we evaluate model performance on a held-out set of 10,000 examples to observe classification error. We sequence length and chose positions for the letters in the animal name "cat" uniformly at random in the sequence unless otherwise mentioned.
300 where each letter is a separate span,
Model. We employ three-layer Transformer model Vaswaai et al. 2017) with position embeddings (Devlin et al., 2019) the sequence encoder, which implemented with HuggingFace Transform- ers (Woff et | 2019). For each example ("cat", SS, y), we feed "[CLS] cat [SEP] S [SEP]" to the sequence encoder and then construct binary classifier over the output representation of "[CLS]" to predict To investigate the effectiveness of SPANDROP, we simply apply SPANDROP to S first before feeding the resulting sequence into the Transformer classifier.
3.2 RESULTS AND ANALYSIS
In each experiment, we compare SPANDROP and Beta-SPANDROP at the same drop ratio p. And further use rejection sampling remove examples that do not preserve the desired supporting facts to understand the effect of supporting fact noise.
Data efficiency. We begin by analyzing the contribution of SPANDROP and Beta-S SPANDROP to improving the sample efficiency of the baseline model. To achieve this goal, we vary the size of the training set from to 50,000 and observe the prediction error on the held-out set. We observe from the results in Figure
a a) that: 1) Both SPANDROP and Beta-SPANDRRP significantly improve data efficiency in low-data settings. For instance, when trained on only 200 training examples, SPANDROP variants can achieve the generalization performance of the baseline model trained on 5x to even 20x data. 2) Removing supporting fact noise typically improves data efficiency further by about 2x. This indicates it is helpful not drop spans in Soup during training when possible, that the model is always trained with true counterfactual examples rather than sometimes noisy ones. 3) Reta-SPANDReN consistently improves upon the baseline model even when data is abundant. This
5

---

Under review as a conference paper at ICLR 2022
Baseline
SPANDROF- e- SPANDROP (miis--rree)
Beta-SrANDROP
E- Beta SPANDROP (noise-frec)
(a) Data efficiency
(b) Noise in supporting facts
(c) Varying sequence length
Error/Noisc (%)
30
Noise from SPANDROP
Noise from SPANDROK
Noise from Bera-SPANDROP
Noise from Beta- SPAN
10
30
10
3
10
1
3
477
0.1
10
10
10
0
0.1 0.2 0.3 0.4 0.5
10
30
100
300
# Training examples
Drop rate
Train/test sequence length
(d) OOD length generalization
(e) SPANDROP vs SPANMASK (f) Position & zero-shot generalization
40
60
SPANMASK
SPANMASK
30
Emron (6)
30
40
20
20
20
10
10
0
200 225 250 275 300 325
Bernoulli Beta-Boc Bernoulli
Fixed
First 100
Test sequence length
Drop/mask distribution
Setting
Figure 2: Experimental results of SPANDROP variants and SPANMASK on the FINDCATS synthetic tasks.
is likely due to the difficulty of the task when n 300 and m = 3. Similar to many real-world tasks, the task remains underspecified even when the generalization error is already very low thanks the large amount of training data available. 4) SPANDROP introduces inconsistent training objective with the original training set, which leads to performance deterioration when there is sufficient training data, which is consistent with our theoretical observation.
Effect of supporting fact noise and sequence length. Since SPANDROP introduces noise in the supporting facts (albeit with low probability), it is natural to ask if such noise is neg- atively correlated with model performance. We study this by varying the drop ratio p from (0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5) on fixed training sets of size 1,000, and observe the resulting model performance and supporting fact error. As can be seen in Figure 2 b). supporting fact noise increases rapidly p grows However, note that although the performance of SPANDROP dete- riorates as p increases, that of reta-SPANDROP stays relatively stable. Inspecting these results more closely, we find that even the performance of the noise-free variants follow similar trend, which should not be affected by supporting fact noise.
Recalling the observations from data efficiency experiments, we turn the hypothesis that this discrepancy is mainly caused by the inconsistent length distribution SPANDROP introduces. To test this hypothesis, we conduct two separate sets of experiments: 1) training and testing the model on varying sequence lengths {10, 20, 30, 50, 100, 200, 300, 500]. where longer sequences suffer more from the discrepancy between SPANDROP-resulted sequence lengths and the original sequence length; and 2) testing the model trained on = 300 on test sets of different lengths, and our hypothesis about distribution drift were correct, we should see SPANDROP models' perfor- mance peaking around n'' = n(1-p), while the performance of Beta-SPANDROP is less affected by sequence length. As can seen from Figures 2 () and d), our experimental results seem to well supporting this hypothesis. Specifically, in Figure _ c). while the performance of both SPANDROP variants deteriorates as n grows and the task becomes more challenging and underspecified, SPAN- DROP deteriorates at faster speed even when we remove the effect of supporting fact noise. On the other hand, we can clearly see in Figure d that SPANDROP performance peak around sequences of length 270 (= n21 p) 300 (1 0.1)) before rapidly deteriorating, while Beta-SPANDROP is unaffected until test sequence length exceeds that of all examples seen during training.
Note that the noise in our experiments are lower than what would be predicted by theory, because in practice the initial sequence S might already contain parts of "cat" before it is inserted. This creates redundant sets of supporting facts for this task and reduces supporting fact noise especially when n large.

---

Under review as a conference paper at ICLR 2022
Mitigating position bias. Besides SPANDROP, replacement- based techniques like masking can also be applied to introduce counterfactual examples into sequence model training, where elements in the sequence are replaced by special symbol that is not used at test time. We implement SPANMASK in the same way as SPANDROP except spans are replaced rather than removed when the sampled "drop mask" 6; is O. We first inspect whether SPANMAS
benefits from the same beta- neta-Bern distribution we use in SPANDROP. As can be seen in Figure ee), the gain from switching to beta- Bernoulli distribution provides negligible benefit to SPANMASK, which does not alter the sequence length of the imput to begin with. We also see that SPANMASK results in significantly higher error than both SPANDROP and Seta-SPANDROP in this setting. We further experiment with introducing position bias into the training data (but not the test data) to test whether these method help the model generalize to an unseen setting. Specifically, instead of selecting the position for the characters "cat" uniformly at random, we train the model with "fixed position" dataset where they always occur at indices (10, 110, 210), and a "first 100" dataset where they are uniformly distributed among the first 100 letters. As can be seen in Figure f). both the baseline and SPANMASK models overfit to the position bias in the "fixed" setting, while SPANDROP techniques significantly reduce zero- shot generalization error. In the "first 100" setting, Beta-SPANDROI consistently outperforms its Bernoulli counterpart and SPANMASK at improving the performance of the baseline model as well, indicating that SPANDROP variants are effective reducing the position bias of the model.
4 EXPERIMENTS ON NATURAL LANGUAGE DATA
To examine the efficacy of the proposed SPANDROP techniques on realistic data, we conduct exper- iments on four natural language processing datasets that represent the tasks of single- and multi-hop extractive question answering, multiple-choice question answering, and relation extraction. We fo- cus on showing the effect of SPANDROP instead of pursuing the state of the art in these experiments.
Datasets. We use four natural language processing datasets: SQuAD 1.1 (Raiuuraar et al. 2016)) where models answer questions on a paragraph of text from Wikipedia; MultiRC (Khashabi al. 2018), which is multi-choice reading comprehension task in which questions can only be answered by taking into account information from multiple sentences; HotpotQA (Yang et al., 2018). which requires models to perform multi-hop reasoning over multiple Wikipedia pages to answer questions; and DocRED YYao et al. 2019), which a document-leved data set for relation extraction.
For the SQuAD dataset, we define spans collections of one or more consecutive tokens to show that SPANDROP can be applied to different granularities. For the rest three datasets, we define spans to be sentences since supporting facts are provided at sentence level. For all of these tasks, we report standard exact match (EM) and F, metrics where applicable, for which higher scores are better. We refer the reader to the appendix for details about the statistics and metrics of these datasets.
Model. We build our models for these tasks using ELECTRA (Clark et al., 2019), since it is shown to perform well across range of NLP tasks recently. We introduce randomly initialized task- specific parameters designed for each task following prior work on each dataset, and finetune these models on each dataset to report results. We refer the reader to the appendix for training details and hyperparameter settings.
Main results. We first present the performance of our implemented models and their combination with SPANDROP variants on the four natural language processing tasks. We also include results from representative prior work on each dataset for reference (detailed the appendix), and summarize the results in Table | We observe that: 1) our implemented models achieve competitive and sometimes significantly better performance (in the cases of HotpotQA, SQuAD, and DocRED) compared to published results, especially considering that we do not tailor our models to each task too much; 2) SPANDROP improves the performance over these models even when the training set large and that the model is already performing well; 3) Models trained with Beta-SPANDRe perform better or equally well with their SPANDROP counterparts across all datasets, demonstrating that our observations on the synthetic datasets generalize well to real-world ones. We note that the performance gains on real-world data is less significant, which likely results from the fact spans in the synthetic task are independent from each other, which is not the case in natural language data.
consistently
We further evaluate the performance of our trained models on the MultiRC testing data, and obtain results of EM/F1: 41.1/79.8, 39.9/78.5 and 1/78. for models with Beta- Beta-SPANDROP, SPANDROP,
7

---

real-world data is Under review as a conference paper at ICLR 2022
(a) HotpolQA dev
(b) MultiRC dev
Model
Ans Fi
Sup Fi
Joint Fi
Model
FM
Fi
RoBERTa-baa
73.5
83.4
63.5
BERT-base
26.6
74.2
Longformer-bar
74.3
84.4
64.4
RoBERTa-basc
38.7
79.2
SAE BERT-base
73.6
84.6
65.0
REPT RoBERTa-base
40.4
80.0
Our implementation
Our implementation
ELECTRA-basc
74.2
86.3
66.2
ELECTRA-bass
40.1
80.4
SPANDROP
74.7
86.7
66.8
SPANDROP
42.3
81.7
PRO
74.7
86.9
67.1
Beeta-SPANDROP
44.8
81.6
(c) DocRED dev
(d) SQuAD dev
Model
Ign Fi
RE Fi
Evi Fi
Model
EM
Fi
E2GRE BERT-basc
55.2
58.7
47.1
RoBERTa-base ELECTRA-basc
90.6
ATLOP BERT-base
59.2
61.1
84.5
90.8
SSAN BERT-base
57.0
59.2
XLNet-large
89.7
95.1
Our implementation
Our implementation
ELECTRA base
59.6
61.6
50.8
ELECTRA A-base
86.6
92.4
+ SPAN
59.9
61.9
+ SPANDROP
86.8
92.6
+ Betaa-SPANDRO
60.1
62.1
51.2
+ Beta-SPANDROO
86.9
92.7
Table 1: Main results on four natural language processing datasets.
and without SPANDROP, respectively. This indicates that both Beta- SPaa-SPANDROP and SPANDROP improve the model generalization ability, and Reta-SPANDRRe better than SPANDROP, improving EM/F1 with 2.0/1.6 absolute over the baseline.
Next, to better understand whether the properties of SPANDROP and Beta-SPANDROP we observe on the synthetic data generalize to real-world data, we further perform of analysis experiments on SQuAD. Specifically, we are interested in studying the effect the amount of training data, the span drop ratio p. and the choice of span size performance.
Baseline
SPANDRoP
Resa-SeNNDRoP
93
Dev Fi (%)
80
92.5
92.5
92
92
60
91.5
91.5
40
91
91
0.1
10
100
0.2
0.4
4
16
64
% of training used Figure 3: Effect analysis of training data size, drop ratio and span size on performance of models trained with SPANDROP and Beta-SPANDDRO over SQuAD
Drop ratio
Span size
Effect of low data. To understand SPANDROP's regularizing effect when training data scarce, we study the model's generalization performance when training on only 0.1% of the training data (around 100 examples) using the entire training (around 88k examples). As can be seen in Figure B (left), both SPANDROP and Beta-SPANDROP significantly improve model performance when the amount of training data is extremely low. As the amount training data increases, this gap slowly closes but remains consistently positive. The final gap when 100% of the training data is used is still sufficient separate top-2 performing systems on this dataset.
Impact of drop ratio. We compare SPANDROP and Beta-SPANDRRO by controlling how likely each span is dropped on average (drop ratio p). Recall from our experiments on FINDCATS that larger will result in distribution drift from the original training set for SPANDROP but not Beta- SPANDROP, thus the performance tually not affected. As can be seen in Figure (middle), observation consistent with this theoretical prediction, and indicate that Beta- SaaNDRAADDDPP is a better technique for data augmentation should want increase sequence diversity by setting p larger value.
the former deteriorates as p increases while the latter is vir-
8

---

Under review as a conference paper at ICLR 2022
Impact of span size. We train the model with SPANDROP SQuAD with varying span sizes of {1,2. 4, 8, 16, 32, 64) tokens per span to understand the effect of this hyperparameter. We observe in Figure E (right) that as span size grows, the generalization performance of the model first holds roughly constant, then slowly deteriorates as span size grows too large. This suggests that the main contributors to generalization performance might have been the total number of spans in the entire sequence, which reduces with larger spans. This results in fewer potential augmented sequences for counterfactual learning, therefore lowering regularization strength. This observation is consistent with that on our synthetic data in our preliminary experiments, where we see that controlling for other factors, larger span sizes yield deteriorated generalization performance (data not shown due to space limit). This also suggests that while SPANDROP works with arbitrary span sizes, the optimal choice of spans for different tasks warrants further investigation, which we leave to future work.
5 RELATED WORK
Long Sequence Inference. Many applications require the prediction/inference over long sequences, such as multi-hop reading comprehension (Yang et al., 2018, Welbl et al.) 2018), long document summarization (Huang et al. 202T), document-level information extraction Yao et al.
2019) in
natural language processing, long sequence time-series prediction (Zoou et al. 2021a). promoter
region and chromatin- profile prediction in DNA sequence (Oubounyt et al. 2019 Zhou & Troyan- skaya 2015) in Genomics etc, where not all elements in the long sequence contribute equally to the desired output. Aside from approaches we have discussed that attempt to approximate all pair-wise interactions between elements in sequence, more recent work has also investigated compressing long sequences into shorter ones to distill the information therein for prediction or representation
learning (Rae et al. 2020 Goyal al. 2020 Kim & Cho 202
Sequence Data Augmentation. Data augmentation is an effective common technique for under- specified tasks like long sequence inference. Feng et al. (20TT) propose group common data augmentation techniques in natural language processing into three categories; 1) rule-based meth- ods Zhhag et al. 2015; Wei & Zou, 2019, Şahin & Steedman 2018), which apply a set of pre- defined operations over the raw input, such as removing, adding, shuffling and replacement; 2) example mixup-basees based methods (Guo et al. 2019 Guo 2020 Chen et al. 2020 Tindal et al. 2020).
which, inspired from Mixup in computer vision (Zang et al., 2018), perform interpolation between continuous features like word embeddings and sentence embeddings; 3) model-based methods (Xie)
et al. 2020 Sennrich et al. 2016), which use trained models to generate new examples (e.g., back
translation Xic et al. 2020
Most of existing rule-based data augmentation methods operate at the token/word level (Feng al. 2021]. such as word shuffle/replacementaddiii
(Wei & Zou 2019). Shuffle- techniques are less applicable when order information is crucial in the raw data (Lan et al. 2019 e.g., in natural language). Moreover, these operations might not be trivial in implementation over larger spans (e.g.. the phrase sentence level). For example, while replacing tokens require selecting candidates from a fixed vocabulary which can be provided by well estimated language models (Clark et al., 2019) replacing phrases or sentences is significantly more challenging since the "vocabulary" is unbounded and marginal probability difficult to estimate. In contrast, our proposed SPANDROP supports data augmentation in multiple granularity as the spans in SPANDROP can be of any length, and is able to reserve sequence order since drop operation does not change the relative order of the original input.
6 CONCLUSION
In this paper, we presented SPANDROP, simple and effective method for learning from long se- quences, which ablates parts of the sequence at random to generate counterfactual data to distill the sparse supervision signal that is predictive of the desired output. We show via theoretical analysis and carefully designed synthetic datasets that SPANDROP and its variant based on the distribution help model achieve competitive performance with a fraction of the data by introducing diverse augmented training examples, and generalize better to previously unseen data. Our exper- iments on four real- world NLP datasets demonstrate that besides these benefits, SPANDROP can further improve upon powerful pretrained Transformer models even when data is abundant.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Wei Bao, Jun Yue, and Yulei Rao. A deep learning framework for financial time series using stacked autoencoders and long-short term memory. PloS one, ((77)e0180944 2017.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long- documenttt transformer. arXiv preprint 205::2004.051500 2020.
Jiaao Chen, Zichao Yang. and Diyi Yang. MixText: Linguisticall--infomeel interpolation of hidden space for semi-supervised text classification. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 2147-2157, 2020.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with Performers. In International Conference on Learning Representations, 2020.
Kevin Clark, Minh- Thang Luong, Quoe V Le, and Christopher D Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2019.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Steven Y Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. A survey of data augmentation approaches for nlp. Findings of ACI,, 2021.
Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sab- harwal, and Ashish Verma. Power-bert: Accelerating bert inference via progressive word-vector elimination. In International Conference on Machine Learning, pp. 3690-3699. PMLR, 2020.
Hongyu Guo. Nonlinear mixup: Out-of-manifold data augmentation for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4044-4051, 2020.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Augmenting data with mixup for sentence classifi- cation: An empirical study. arXiv preprint arXiv: 08941, 2019.
Kevin Huang, Qi Peng, Guangtao Wang. Tengyu Ma, and Jing Huang. Entity and evidence guided relation extraction for docred. arXiv preprint arXiv:2008. 12283, 2020.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp- 1419-1436, 2021.
Fangkai Jiao, Yangyang Guo, Yilin Niu, Feng Ji, Feng-Lin Li, and Liqiang Nic. REPT: Bridg- ing language models and machine reading comprehensionvia retrieval-based pre-training. arXiv preprint arXiv:2 105.00 (05.04201, 2021.
Amit Jindal, Arijit Ghosh Chowdhury, Aniket Didolkar, Di Jin, Ramit Sawhney, and Rajiv Shah. Augmenting nlp models using latent feature interpolations. In Proceedings of the 28th Interna- tional Conference on Computational Linguistics, pp. 6931-6936, 2020.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Romneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature. 596(7873):583-558, 2021.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In International Conference Machine Learning. PMLR, 2020.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Look- ing beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018
10

---

Under review as a conference paper at ICLR 2022
Gyuwan Kim and Kyunghyun Cho. Length-adaptin transformer: Train once with length drop. anytime with search. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference Natural Language Processing, pp. 6501-6511, 2021.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2019.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. ALBERT: lite bert for self-supervised learning of language representations. Interna- fional Conference on Learning Representations, 2019.
Juho Lee, Saehoon Kim, Jaehong Yoon, Hae Beom Lee, Eunho Yang, and Sung Ju Hwang. Adap- tive network sparsification with dependent variational beta-bernoulli dropout. arXiv preprini 17X.: 2018.
Lei Liu, Yuhao Luo, Xu Shen, Mingzhai Sun, and Bin Li. B-dropout: A unified dropout. IEEE Access, 7:36140-36153, 2019a.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pre- training approach. arXiv preprint 1rXiv:1907.11692 2019b.
Mhaned Oubounyt, Zakaria Louadi, Hilal Tayara, and Kil To Chong. Deepromoter: robust promoter predictor using deep learning. Frontiers in genetics, 10:286, 2019.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong.
Random feature attention. In International Conference on Learning Representations, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Jack W. Rac. Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100, 000+ questions for machine comprehension of text. In EMNLP, 2016.
Gözde Gül Şahin and Mark Steedman. Data augmentation via dependency tree morphing for low- resource languages. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 5004-5009, 2018.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. 54th Annual Meeting of the Association for Computational Linguistics, pp. 86-96, 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-195: 2014.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: benchmark for efficient transformers. In International Conference Learning Representations, 2020.
Ming Tu, Kevin Huang, Guangtao Wang. Jing Huang. Xiaodong He, and Bowen Zhou. Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 9073-9080, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.

---

Under review as a conference paper at ICLR 2022
Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance text classification tasks. In Proceedings of the Conference on Empirical Methods Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing, pp. 6382-6388, 2019.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. Constructing datasets for multi-hop read- ing comprehension across documents. Transactions of the Association for Computational Lin- guistics, 6:287-302, 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's is transformers: State-of the-art natural language processing. arXiv preprint arXiv:1910.03770 2019.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems, 2020.
Benfeng Xu, Quan Wang, Yajuan Lyu, Yong Zhu, and Zhendong Mao. Entity structure within and throughout: Modeling mention dependencies for document-level relation extraction. Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 14149-14157, 2021.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Mamning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings the Conference Empirical Methods in Natural Language Pro- cessing, pp. 20692236222220 2380, 2018.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLNet: Generalized autoregressive pretraining for language understanding, Advances in Neu- ral Information Processing Systems, 2019.
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. DocRED: A large-scale document-level relation extraction dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.
764-777, 2019.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alherti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big Bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems, 2020.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David I Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018
Xiang Zhang. Junbo Zhao, and Yann LeCun. Character-leve convolutional networks for text clas- sification. In Advances in Neural Information Processing Systems, pp. 649-657, 2015.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of AAAI, 2021a.
Jian Zhou and Olga G Troyanskaya. Predicting effects of noncoding variants with deep learning- based sequence model. Nature methods, 12(10):931 1-944, 2015.
Wenxuan Zhou, Kevin Huang, Tengyu Ma. and Jing Huang. Document-leved relation extraction with adaptive thresholding and localized context pooling. In Proceedings of the AAAI Conference Artificial Intelligence, 2021b.
12


Reviews:
[{"review_id": "XoUHF965-h8", "reviewer": "Reviewer_ykje", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\nEffective data augmentation is a relevant topic.\n\nThe paper is well written.\n\nWeakness:\n\nLimited novelty.\n\nMy major concern is the lack of comparison of other baselines. In section5, the author discuss a bunch of related works on data augmentation, however, in natural language experiments these are not compared. I know that SCANDROP is easier to implement, but that does not mean you don't need to compare with other algorithms.\n\nIt's not surprising that this data augmentation is effective, especially for the catfinding task.\n\nWhile the improvement over electra-base is consistent, it's not large.\n\nBelow are detailed comments:\n\nSec1. I don't quite understand why when in data the contributing portion is small or sparse, then the problem is \"underspecified\". It's still specified by the small contributing portion I think? Are you suggesting the contributing part is so small, that for example, a classifier can not reach a decision?\n\nWhat not span-replace? If you replace a span with a random token, it gives counterfactual sequence with the same length.\nMinor comments:\n\nSec1. \"the data does not sufficiently define the goal for statistical models\", I don't understand what global is here.\n", "summary_of_the_paper": "In this paper, we instead investigate learning problems for long sequences where not all input elements contribute equally to the desired output. SCANDROP is a simple algorithm to randomly drop segments in a sequence. The authors first establish that when the number of contributing segments is sparse, the algorithm will preserve them with a relatively large probability. Then, Beta-SCANDROP is proposed to preserve the original sequence length with higher probability. In experiments, consistent improvement is shown.\n", "summary_of_the_review": "I have two major concerns (1) limited novelty. (2) The lack of comparison of other baselines. In section5, the author discuss a bunch of related works on data augmentation, however, in natural language experiments these are not compared. I know that SCANDROP is easier to implement, but that does not mean you don't need to compare with other algorithms.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FzzbMZ14-LE", "reviewer": "Reviewer_qHWi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: 1. Simple procedure to improve accuracy for long sequence input problems. 2. Mathematically sound. The authors prove the claims regarding augmented sequence lengths.\n\nWeaknesses: \n1. The procedure is fairly simple, and similar ideas have been used for regularizing models. I find the solution similar to word dropout, and would be interesting to see comparison to that.\n2. The gains on real datasets are not high. ", "summary_of_the_paper": "The paper focuses on distilling supervision signal from long sequences. They focus on cases where the input is a long sequence of length n, but the target prediction is determined by a small subset of size m of sequence fragments, where m << n. The authors propose augmenting data by randomly dropping spans from the input sequence. They first propose SpanDrop which removes each span with a probability p. This process might result in shorter sequences, resulting in shift of training data distributions. As a fix, they also propose Beta-SpanDrop where the spans are dropped using probabilities sampled from beta bernoulli distribution. Beta SpanDrop preserves the length of the original utterance with higher probability, while generating similar variety of augmented utterances.   ", "summary_of_the_review": "The augmentation procedure is very similar to word dropout. It is not clear if similar benefits can be obtained by using word dropout at training time. The experimental results on real datasets are weak. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "9kJd9LkP5k", "reviewer": "Reviewer_DeC3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n- The proposed method is simple and easy to understand. \n- The method outperforms baseline methods.\n\n\nWeaknesses  \n- Several technical details are not discussed.\n    - How to partition a long sequence into spans? For text sequences? For time series? Will the final results be sensitive to the partition of spans? \n    - If each word in NLP sequences is a span, the proposed method is the same as the word-level dropout baseline studied in previous works, e.g., [1]. What's the tech novelty of this work?\n    - While the authors focus on the \"learning problems for long sequences where not all input elements contribute equally to the desired output\", the uniqueness of this kind of problems is not clear to me. In most (if not all) real-world sequence classification and prediction problems, not all input elements contribute equally to the desired output; otherwise, the problems will become much easier. \n- Experiments need to be improved and enhanced, e.g., \n    - As reviewed in the second paragraph, there are many Transformer variants proposed for long sequences, but none of them is compared. Although these approaches aim to approximate the original pairwise interaction with lower cost and are often interested in still capturing the interactions between every pair of input elements (e.g., the long sequence benchmark proposed by Tay et al., 2020), they can still be directly applied to long sequences where not all input elements contribute equally to the desired output. \n    - The four NLP datasets used in experiments are not representative tasks for long sequences. I suggest to test on the public benchmark datasets, e.g., [2].\n\n\n[1] Soft Contextual Data Augmentation for Neural Machine Translation, ACL 2019.\n\n[2] Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations, 2020.", "summary_of_the_paper": "This work proposes SPANDROP, a simple variant of dropout, working on the spans of long sequences. SPANDROP randomly ablates parts of a sequence at a time and asks the model to perform the same task to emulate counterfactual learning and achieve input attribution. The method is tested on both toy tasks and four NLP tasks.", "summary_of_the_review": "Both the technical part and empirical evaluations can be improved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5SONjJQCFcx", "reviewer": "Reviewer_EEvq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method is simple yet effective. Experiments in the paper are well-designed to show the effectiveness of the method.\n\nThe method generally assumes that the decision of drop can be done independently. However, the positions of salient information would usually exhibit certain patterns such as appearing close to each other. Do you have any idea to incorporate those characteristics into the method or learn those patterns?\n\nAs the name of the method is SpanDrop, it would be much interesting if various selection/split methods of spans have been investigated. Of course, splitting by a sentence for passages is natural.\n\nSpanDrop is only applied at the input level. I am curious whether this drop could be applied to intermediate representations like done in LengthDrop (Kim and Cho, 2021).\n\nI wonder about the additional training cost of SpanDrop compared to the standard training. The expectation of length is the same for SpanDrop and Beta-SpanDrop. However, the expectation of length’s square would be different. Therefore, I guess their computational overhead will be also different. The sampling cost would be almost negligible. Does it require additional training steps to converge due to its regularization effect? Don’t it require modification of other regularization such as standard dropout?\n\nDo you have any thoughts about whether SpanDrop will be generalized seamlessly to other tasks than NLP?\n", "summary_of_the_paper": "This paper proposes a data augmentation method, SpanDrop (and its variant Beta-SpanDrop), for long sequence data, especially where supporting facts take small portions. They provide a theoretical background on their method and evaluate the method on a synthetic task (FindCats) and four real natural language processing tasks requiring reasoning over long texts. SpanDrop is effectively improves the accuracy in both low-resource and abundant-resource settings.\n", "summary_of_the_review": "The paper is well-written and well-structured. The proposed method is theoretically and empirically verified. Also, the paper is practically useful because it is easy to implement and plug into any tasks handling (long) sequences for better accuracy.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a data augmentation approach called SpanDrop to help to distill supervision signals from a long sequence prediction problem. The reviewers generally agree on two major drawbacks of the paper. First, the novelty of this approach. Second, the experiment results are not very convincing.

After reading the responses from the authors, I don’t think the authors convinced me of the novelty of the work, especially when comparing it to the word dropout. No matter if you treat the data or the model as a black-box, it’s effectively doing the same thing. Apart from that, the model can only be used in the setting of “underspecification” long sequence tasks, which diminishes its value in real applications.

On the experiment side, there are three issues. One, many tasks considered are not long sequence tasks. Second, the improvement is marginal in many cases. Three, more related methods should get considered as baselines. Besides these three points raised by the reviewers, I also want to raise the point that it is not (and should not) be acceptable to report ALL your language experiment results on dev sets. I understand it is more time-consuming to get the test results on tasks where the test has to be done online, e.g. SQuAD. However, it is not a good practice in reaching a conclusion merely on dev sets in general.

Based on the reviewers' comments and the reasons listed above, I recommend rejection of this paper.
Decision: Reject