Investigating Positional Influence of Intricate Training Instances on Neural NLP Models
Authors: Simran Gupta, DEBAJYOTI ROY, Sandipan Sikdar, Koustav Rudra


Reviewer paper summaries: This paper investigates how the order of training data points affects the performance of neural NLP models, specifically identifying that placing intricate data points at the start or end of training sequences significantly degrades model performance. The study shows that contaminating the labels of these intricate data points can mitigate these vulnerabilities.
Summary of strengths: 1. The paper is well-written and easy to understand.
2. The investigation of the data points' order in SFT is interesting, and many post-training researchers might refer to this paper to tune their data points during training.
Summary of weaknesses: 1. I am curious about the fine-tuning epochs. How many epochs do you train the models on the downstream tasks? If you train for multiple epochs, I think we can mitigate this issue by shuffling the data points at the beginning of each epoch, instead of proposing a more advanced model structure to address it (as you mentioned in the conclusion). (This is likely why people shuffle the dataset at each epoch).

2. I do not disagree with proposing a new model to mitigate this issue, but before starting that work, I think we should explore more existing model structures, such as various transformer-based variants including GPT, LLaMA, and Mistral (with or without MOE). This paper seems to only investigate BERT and does not cover enough other LLMs.

3. Have you analyzed whether pre-trained and non-pre-trained models with the same structure are affected by data points to different degrees during the fine-tuning stage?

4. If they are affected to different degrees and the pre-trained models can better mitigate the effect, is it possible that larger pre-trained LLMs (or those pre-trained with more text context) might not be affected by the intricate data points?

5. I wonder if your work can also benefit existing SFT, where their labels might be a longer context instead of a single word or token.
Comments, suggestions and typos: Refer to the Summary Of Weaknesses part
Overall assessment scores: 3