Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Learning to Split (ls), a novel algorithm that automatically detects a potential bias in datasets. More specifically, ls consists of a Splitter and a Predictor. The Splitter learns to divide the dataset into a train split and a test split. Then, the Predictor is trained using the divided train split and measures the generalization error with the test split. The Splitter is then trained in the direction in which the generalization error increases. That is, ls find a train split that cannot generalize well in the test split and then observe the bias existing in the dataset. The authors validated that ls behaves as expected on datasets with bias annotation. Finally, when ls is combined with debiasing methods that require bias annotation during training, such as GroupDRO, these models were trained successfully without bias annotation
strength_and_weaknesses: Strength

1. The paper is logically well-written, including a discussion of the limitations of ls. In particular, additional experiments related to label noise further enhanced the reliability of the paper.

2. Several experiments have well validated the effect and applicability of ls. In particular, the experimental results using GropuDRO with ls are impressive. It would be interesting to conduct experiments with ls applied to other methods using bias annotations in the validation phase.

3. This research tackles an important question, and I believe it is a helpful research direction for the community.

Weakness

1. I think there should be further experiments and discussion on how the \delta value affects the results of ls. Therefore, during the review period, I hope the authors validate that the model works robustly even with various \delta values. Also, it would be helpful to understand ls if the authors address how the model behaves when the percentage of minority groups varies rather than 25%.

2. As the authors mentioned in Section 5, ls seems to have issues with scalability. To further discuss this concern, I think the training time of other debiasing methods should be provided in Table 2 of the Appendix.

3. In Table 1, the source of the results brought from the previous paper is omitted. That information should be included to determine whether the experimental settings of ls are the same as those of the baselines.



clarity,_quality,_novelty_and_reproducibility: The manuscript is logically well-written and easy to follow. Also, to the best of my knowledge, this is the first paper to successfully deal with detecting biases in datasets without bias annotations.
summary_of_the_review: The main contribution of this paper is that potential biases present in datasets can be detected without bias annotations. Although there are concerns regarding scalability, I believe it is valuable as the first paper that automatically detects potential biases.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies a very interesting and novel problem setting: how to split the data in a way that when a model is trained on one partition cannot generalize to the second partition: while this task does not make realistic sense, careful studying of it will surely reveal many other properties that are important in the machine learning community. On the other hand, I feel like the actual algorithm proposed by this paper used for this new task is still in its fairly preliminary stage, with multiple questions unanswered. 
strength_and_weaknesses: - strengths: 
   - the paper studies a very new and interesting problem
   - figures/visualizations are stunning
   - the writings are very clear (although might be too detailed)
   
- weakness:
   - as a new study, many new questions are left unanswered for the community to follow in the later stage
   - while the visualizations are stunning, they look more like belong to brochures instead of academic papers, many figures do not necessarily need such visualization to explain. A simple table will do the job and then save a lot of space for more detailed discussions on other aspects. 
clarity,_quality,_novelty_and_reproducibility: - clarity: good
- novelty as a task: great
- novelty as a technique: 
     - the algorithm to partition the data is quite limited as it is a fairly straightfoward search through the dataset. 
          - the two constraints at Eq 1 seem to be an over-complicated paraphrase of simple ideas (sufficient training samples and label balance)
          - line 7 and line 8 at algorithm 1 are random samples (that can probably be much improved with more heuristics, e.g., whether searched dataset will update the later search strategies)

- quality
     - it seems weird that eq.3 does not involve any hyperparameters to balance the weight between the main objectives and the regularization loss. 
          - if this is a typo, then detailed discussion will be needed on the choice of the hyperparamters
          - otherwise, then probably more discussions will be needed on why such terms are not needed
    - the fact the group-DRO works on the new partition data seems to offer a piece of evidence that the method doesn't work that well. 
         - group-DRO works by leveraging the minority of the samples during the training, however, if the algorithm successfully achieves its goal as depicted in Figure 1, there probably do not exist any minor samples in the training set for DRO to use
         - probably it's more convincing to test other algorithms that explicitly account for the bias, as can be useful without the usage of the minor samples. 
    - it might be better to use the performances of these more advanced algorithms to evaluate the performance of the partition algorithm (and its variants, as ablation studies), then the detailed setup of all these algorithms will become essential. 

- reproducibility:
     - given the limitation of the novelty of the method, it is probably necessary to discuss computing loads in details in the main manuscript (any evidence a more efficient algorithm is not even needed?)

- minor
     - the authors offer a good summary of papers explicitly discussing about biases, there are many other relevant papers following the debiasing strategy but do not use these keywords as this main ideas fall into a greater scope of ML robustness. It's probably better to expand the literature scope to a broader scope, e.g. (each of these represents a whole branch, instead of the listed individual paper)
          - Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering 
          - Learning robust representations by projecting superficial statistics out 
          - Invariant risk minimization
summary_of_the_review: Overall, I think this is a greatly interesting paper with a lot of potentials, but probably a bit preliminary at this moment.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes using an adversarial bi-level optimization to find hard train/test splits of a dataset. More specifically, the inner loop corresponds to normal ERM training on that train/test split, and the outer loop learns a Splitter that changes the train/test split to make generalization in the inner loop difficult. The authors use two regularizers to avoid degeneracies: first, constraining the ratio of train/test dataset sizes, and second, constraining the label ratio in the two datasets (to avoid all labels being in the train split and all of the others being in the val split). This automatic splitting can then be used to identify biases in the dataset, like spurious correlations, and you can combine it with group DRO to get robust models.
strength_and_weaknesses: Strengths:
- Interesting and novel idea and approach, particularly the idea of using this bi-level optimization to identify biases in datasets.
- The paper is clearly written and provides a thorough empirical analysis of the proposed method. 

Weaknesses:
- The algorithm can be computationally expensive, since it needs to retrain erm each time in the inner loop. Maybe it doesn't need to be trained to convergence? It would be interesting to see what would happen if each inner loop isn't trained fully to convergence. Another interesting experiment to see would be to alter the algorithm to just train the last layer in the inner loop (e.g. by pretraining the earlier layers with an initial ERM run), to reduce the computation requirements. 

Not a weakness of this paper, but I would also be curious to see if the splitting could be used in other settings, such as OOD detection. 
clarity,_quality,_novelty_and_reproducibility: I think this paper proposes a clever novel idea and method and is written clearly. 
summary_of_the_review: The proposed idea is novel and interesting, so I recommend accepting the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors focus on the problem of identifying potential generalization gaps that can be caused by selectively choosing train a test sets in supervised learning problems.  The authors propose an algorithm called Learning to Split (LS) that iteratively trains a model, uses a three part objective to train a "splitter" model to assign samples train and test sets, and then again retrains a model until the gap between train and test performance ceases to increase.  They show empirically that LS is able to find splits that have greater generalization gaps between train and test performance than randomly splitting the data.  Further, they show that methods that attempt to debias models work well when labels indicating subpopulations are given, but when not provided such labels, debiasing methods perform no better than standard empirical risk minimization.
strength_and_weaknesses: Strengths
1. The problem of automatically identifying biases in data sets is clearly one of major practical importance to the use of machine learning techniques in real-world setting.
2. Their technique is conceptually simple, and seems rather easy to implement.
3. The results reported clearly show LS is able to split data to increase the generalization gap compared to random sampling on a large number of varied data sets.

Weaknesses
1. I think the problem as set up in the paper may lack some practical utility.  I do not see the direct connection bween intentionally biasing train and test splits and measuring the generalization gap and bias that can arise when training a model on the data set.  What is not clear is that if a normal training procedure which randomly splits train and test data results in a sufficiently accurate model on the subpopulation discovered by LS.  If so, then LS is measuring a worst case generalization performance that may not be ever realized.  I think a more compelling set up would be if a training data is representative of a standard split, and a subpopulation can be found within a test set for which a model performs poorly.  This is much more representative of how data is used to train a model.
2. I think the experiments could benefit from other baseline lines than random splits.  For instance, in the data sets where auxiliary attributes are available (like CelebA), you could split the train/test data across attributes and measure the generalization gap.  Doing this can result in different “oracle” baselines that better explain generalization gap relative to clear biases.  Further, I feel a conceptually much simpler and heuristic approach can be taken based on the same principle as (2).  At each outer iteration, simply choose 25% of the samples, accounting for label distributions, that the predictor performed the worst on.  This doesn’t require a complex splitter model, can exactly satisfy the desired constraints, and is in the same spirit as LS.
3. The regularization approach seems inexact for a sampling procedure, and the constraints attempted to be imposed by the regularization terms are not satisfied in the experiments (~82% of the examples are in the train set for the “Beer Look” experiment).  It’s not clear how important this is to be exact but if it is important, then stronger enforcement of constraints needs to be considered.
4. Overall the paper doesn’t provide much of principled framing of the problem being solved nor the proposed technique.  For instance, since label distributions are controlled for in LS, it would seem the cause of generalization gap is due to either concept drift (p(y|x) changing) or covariate shift (p(x) changing).  Without identifying the underlying formal definition of bias, it is difficult to understand what is being modeled in a principled manner.  Similarly, it is difficult to reason about how much LS is guided by distributional shifts in the data or the predictive power of the predictor (i.e. Is the generalization gap found by LS a function or the data, the model, the training procedure, or all of the above?).
5. The justification for the \delta parameter is taken out of context from Centola et al. 2018  Their work focuses on what proportion of a subpopulation is needed to tip a majority opinion to a minority opinion.  This has little to do with the problem that is the focus of this work.

Minor Points/Questions:
1. (2) can use some further elaboration.  As I understand it, the second term is and indicator if the predictor predicts the true label. This should be mentioned 
2. There are no details on how (3) is optimized.  The free variables z_i are in {0,1}, which makes this not an direct application of gradient based methods as written.  How is it done?
clarity,_quality,_novelty_and_reproducibility: Clarity - Overall the writing is clear.  The notation, however, could use elaboration. Specifically the equations in 3.3 and 3.4 could use some elaboration about specifically is meant by their notation.

Quality - Beyond the strengths and weaknesses above, I have no further notes on the quality of the work.

Novelty - Much of the proposed methodology is rather straight-forward and does not introduce many novel concepts. The main source of novelty of the work is in the posing of the problem itself: The task of splitting a data set so that a model achieves high generalization gap between test and train sets.  While i have not encountered work performing this exact task, a lot of work has focused on sources of bias.  Also, there is another parallel line of work on test and evaluation of ML models that aims to uncover biases present in data collections.

Reproducability - Algorithm 1 lacks any level of specificity in terms of how the sampler is used to split data and what optimization procedures are used to update the sampler or predictor.  As such, I feel it is very difficult to reproduce this work. 
summary_of_the_review: Overall, I feel there are three major weaknesses of the paper that influenced my decision.
1.  I do not believe the specific problem formulation considered in this paper has much practical significance.  I am not convinced that intentionally splitting whole data sets to maximize a generalization gap necessarily means that training a model on the data set using standard data splits and procedures will be biased.
2. Due to lack of principled discussion and justification, I am not convinced that the generalization gap can entirely be explained by bias in the data and not biases from the specific predictor model or training procedure.
3. The experiments are quite limited in that there are baselines that should be used to justify the proposed approach.

(see rebuttal discussion below for more details on final scoring)
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a procedure to split a dataset to make generalization from one to the other difficult.
The reviewers appreciated that the proposal was simple and that some of the ideas were novel and is an interesting first work to identify biases in the dataset.
However, during the virtual discussions, some of the reviewers agreed that while the paper has some meaningful contributions, the paper would need significant work to (1) better highlight the relevant contributions and its impact, which was only a small part in the paper submission, and to (2) simplify the presentation of the paper as it seemed to some of the reviewers that some key parts were intentionally over-complicated.
justification_for_why_not_higher_score: Presentation did not put the emphasis or expansion on the actual valuable contribution, and some parts were over-complicated-- significant work would be needed to make this suitable for publication.
Decision: Reject