Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a data augmentation method, SpanDrop (and its variant Beta-SpanDrop), for long sequence data, especially where supporting facts take small portions. They provide a theoretical background on their method and evaluate the method on a synthetic task (FindCats) and four real natural language processing tasks requiring reasoning over long texts. SpanDrop is effectively improves the accuracy in both low-resource and abundant-resource settings.

main_review: The method is simple yet effective. Experiments in the paper are well-designed to show the effectiveness of the method.

The method generally assumes that the decision of drop can be done independently. However, the positions of salient information would usually exhibit certain patterns such as appearing close to each other. Do you have any idea to incorporate those characteristics into the method or learn those patterns?

As the name of the method is SpanDrop, it would be much interesting if various selection/split methods of spans have been investigated. Of course, splitting by a sentence for passages is natural.

SpanDrop is only applied at the input level. I am curious whether this drop could be applied to intermediate representations like done in LengthDrop (Kim and Cho, 2021).

I wonder about the additional training cost of SpanDrop compared to the standard training. The expectation of length is the same for SpanDrop and Beta-SpanDrop. However, the expectation of length’s square would be different. Therefore, I guess their computational overhead will be also different. The sampling cost would be almost negligible. Does it require additional training steps to converge due to its regularization effect? Don’t it require modification of other regularization such as standard dropout?

Do you have any thoughts about whether SpanDrop will be generalized seamlessly to other tasks than NLP?

summary_of_the_review: The paper is well-written and well-structured. The proposed method is theoretically and empirically verified. Also, the paper is practically useful because it is easy to implement and plug into any tasks handling (long) sequences for better accuracy.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes SPANDROP, a simple variant of dropout, working on the spans of long sequences. SPANDROP randomly ablates parts of a sequence at a time and asks the model to perform the same task to emulate counterfactual learning and achieve input attribution. The method is tested on both toy tasks and four NLP tasks.
main_review: Strengths: 
- The proposed method is simple and easy to understand. 
- The method outperforms baseline methods.


Weaknesses  
- Several technical details are not discussed.
    - How to partition a long sequence into spans? For text sequences? For time series? Will the final results be sensitive to the partition of spans? 
    - If each word in NLP sequences is a span, the proposed method is the same as the word-level dropout baseline studied in previous works, e.g., [1]. What's the tech novelty of this work?
    - While the authors focus on the "learning problems for long sequences where not all input elements contribute equally to the desired output", the uniqueness of this kind of problems is not clear to me. In most (if not all) real-world sequence classification and prediction problems, not all input elements contribute equally to the desired output; otherwise, the problems will become much easier. 
- Experiments need to be improved and enhanced, e.g., 
    - As reviewed in the second paragraph, there are many Transformer variants proposed for long sequences, but none of them is compared. Although these approaches aim to approximate the original pairwise interaction with lower cost and are often interested in still capturing the interactions between every pair of input elements (e.g., the long sequence benchmark proposed by Tay et al., 2020), they can still be directly applied to long sequences where not all input elements contribute equally to the desired output. 
    - The four NLP datasets used in experiments are not representative tasks for long sequences. I suggest to test on the public benchmark datasets, e.g., [2].


[1] Soft Contextual Data Augmentation for Neural Machine Translation, ACL 2019.

[2] Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations, 2020.
summary_of_the_review: Both the technical part and empirical evaluations can be improved.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper focuses on distilling supervision signal from long sequences. They focus on cases where the input is a long sequence of length n, but the target prediction is determined by a small subset of size m of sequence fragments, where m << n. The authors propose augmenting data by randomly dropping spans from the input sequence. They first propose SpanDrop which removes each span with a probability p. This process might result in shorter sequences, resulting in shift of training data distributions. As a fix, they also propose Beta-SpanDrop where the spans are dropped using probabilities sampled from beta bernoulli distribution. Beta SpanDrop preserves the length of the original utterance with higher probability, while generating similar variety of augmented utterances.   
main_review: Strengths: 1. Simple procedure to improve accuracy for long sequence input problems. 2. Mathematically sound. The authors prove the claims regarding augmented sequence lengths.

Weaknesses: 
1. The procedure is fairly simple, and similar ideas have been used for regularizing models. I find the solution similar to word dropout, and would be interesting to see comparison to that.
2. The gains on real datasets are not high. 
summary_of_the_review: The augmentation procedure is very similar to word dropout. It is not clear if similar benefits can be obtained by using word dropout at training time. The experimental results on real datasets are weak.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, we instead investigate learning problems for long sequences where not all input elements contribute equally to the desired output. SCANDROP is a simple algorithm to randomly drop segments in a sequence. The authors first establish that when the number of contributing segments is sparse, the algorithm will preserve them with a relatively large probability. Then, Beta-SCANDROP is proposed to preserve the original sequence length with higher probability. In experiments, consistent improvement is shown.

main_review: Strength:

Effective data augmentation is a relevant topic.

The paper is well written.

Weakness:

Limited novelty.

My major concern is the lack of comparison of other baselines. In section5, the author discuss a bunch of related works on data augmentation, however, in natural language experiments these are not compared. I know that SCANDROP is easier to implement, but that does not mean you don't need to compare with other algorithms.

It's not surprising that this data augmentation is effective, especially for the catfinding task.

While the improvement over electra-base is consistent, it's not large.

Below are detailed comments:

Sec1. I don't quite understand why when in data the contributing portion is small or sparse, then the problem is "underspecified". It's still specified by the small contributing portion I think? Are you suggesting the contributing part is so small, that for example, a classifier can not reach a decision?

What not span-replace? If you replace a span with a random token, it gives counterfactual sequence with the same length.
Minor comments:

Sec1. "the data does not sufficiently define the goal for statistical models", I don't understand what global is here.

summary_of_the_review: I have two major concerns (1) limited novelty. (2) The lack of comparison of other baselines. In section5, the author discuss a bunch of related works on data augmentation, however, in natural language experiments these are not compared. I know that SCANDROP is easier to implement, but that does not mean you don't need to compare with other algorithms.
review_title: null

Meta-review: The paper proposes a data augmentation approach called SpanDrop to help to distill supervision signals from a long sequence prediction problem. The reviewers generally agree on two major drawbacks of the paper. First, the novelty of this approach. Second, the experiment results are not very convincing.

After reading the responses from the authors, I don’t think the authors convinced me of the novelty of the work, especially when comparing it to the word dropout. No matter if you treat the data or the model as a black-box, it’s effectively doing the same thing. Apart from that, the model can only be used in the setting of “underspecification” long sequence tasks, which diminishes its value in real applications.

On the experiment side, there are three issues. One, many tasks considered are not long sequence tasks. Second, the improvement is marginal in many cases. Three, more related methods should get considered as baselines. Besides these three points raised by the reviewers, I also want to raise the point that it is not (and should not) be acceptable to report ALL your language experiment results on dev sets. I understand it is more time-consuming to get the test results on tasks where the test has to be done online, e.g. SQuAD. However, it is not a good practice in reaching a conclusion merely on dev sets in general.

Based on the reviewers' comments and the reasons listed above, I recommend rejection of this paper.
Decision: Reject