Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a novel concept of sample robustness, which is used to find tailored training(sub)sets and hyperparameters depending on the robustness distribution of the test(sub)sets for boosting model performance. The motivation is straight forward and the proposed concept of sample robustness is inspiring. However, the works in this paper do not match with the announced contribution. Hence, I vote to reject.

Reasons for scores:
1，this paper announces that the novel concept of sample robustness can be used to choose training subsets and boost the model performance. In addition, some simple theoretical analysis about sample robustness are also given. But, this paper just gives some simple results of theoretical analysis and have no any explanation about those analytical results. More significantly, those analysis have little relationship with the goal of this paper. So this paper lacks some key theoretical analysis about sample robustness for its advantages.

2，The main works of this paper is based on empirical studies. Because of the lack of necessary theoretical analysis, more datasets are needed for verifying the behavior of model based on the concept of sample robustness. However, the datasets in this paper are too few to guarantee the good behavior of proposed technique in orther datasets.

3，this paper does not analyze the pros and cons of proposed technique based on the concept of sample robustness.

4，Some irregular use of mathematical symbols and grammatical errors.

Overall, the works in this paper do not support the announced contribution

review_title: The works in this paper do not match with the announced contribution 

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:   

This work introduces the concept of sample robustness – based on computing the pointwise Lipschitz constant of a data point – and use it to empirically analyze the effects of training on least and most robust training subsets on the performance for different models. This is done for both classification and regression setups. It is shown that the model performance can be sometimes improved by choosing particular training subsets and hyperparameters depending on the robustness distribution of the test (sub)sets.

-------------------------------------------------------------------
Pros:

-	The problem studied is interesting and is very relevant from a practical point of view.
-	The paper is structured well, and the problem has been motivated well in the introduction.
-	The related work and experiment results sections are quite thorough, and the empirical results are interesting.

------------------------------------------------------------------
Cons:

-	My main concern is that the definition of the sample robustness (in Section 3) seems overtly simplistic and ad-hoc. I understand the underlying reasoning behind the choice, but one could also consider many other definitions. It would have been more satisfactory had some supporting theoretical results been provided for the proposed notion.
-	The grammar and quality of writing can be improved at many places.
-	In Table 3, it seems that training on the complete training set gives the best loss/accuracy when tested on the complete test set (compared to when we train on subsets of the data). But then, what is the benefit here of training on a (least/most robust) subset of the training data? No explanation is provided in this regard.
-	In Table 4 as well, downsizing x drastically increases the loss on the complete test t. However, in Table 6 (KNN), choosing the 40,000 most robust training samples gives the best accuracy on t. There is no explanation provided as to why this is the case.

----------------------------------------------------------------------
Further remarks:

-	In Section 4.2: 50.000 --> 50,000 (same for 10.000).
-	It will be helpful for the reader if Algorithm 2 were in the main text.
-	In the experiments, it is not clear to me why the least and most robust test subsets are considered. In practice, we do not have access to the test data so we cannot process it.  

--------------------------------------------------
Post rebuttal:
I have read the authors response and appreciate the effort made to improve the paper. But I think the results still need additional work, especially from the theoretical front. So my original score is unchanged.
review_title: Recommendation to reject

final_score: rating: 2: Strong rejection
review_content: This work investigates how to choose the right training set and hyper-parameter for a test set. The authors claim that they introduce a concept of sample robustness based on the Lipschitz constant of the label map. The authors then empirically evaluate the robustness distribution of two datasets and investigate the model performance when using training subsets from different parts of the robustness distribution.

Pros:
- The idea of robustness distribution for a dataset seems interesting. It might be used to compare different datasets.
- The authors provide extensive experiments using different models. It is nice to see the details for each model and the robust samples for each dataset.

Cons:
- The novelty of this work is somewhat limited. The notion of sample robustness that the authors propose is just a minor modification of Lipschitz constant. Moreover, there is no theoretical or empirical justification regarding why this definition of sample robustness is useful.

- Despite that I really appreciate the efforts that the authors have put on the experimental section, I think that the contribution to the community is somewhat weak. This paper just uses the so called sample robustness to select a subset of the training sample and then reports the performance of model trained with that subset on some test data. But the authors do not provide any insights on how it works. In fact, from the results, it is still not clear how sample robustness can help choose the right training set and improve the model performance. 

- There are some typos. For example, in the nine line of abstract, “be regarded as an intrinsic” should be “be regarded as being intrinsic”; in the related work, when citing multiple work, it is better use the command “\citep{work1, work2}”.
review_title: Official Blind Review #4

final_score: rating: 4: Ok but not good enough - rejection
review_content: Overview: 

The authors introduce sample robustness, a pointwise measure of the sensitivity of the label map to perturbations in the feature space. They do so by taking the pointwise Lipschitz constant of the label map and normalizing it by the label norm. The authors show that training on samples that are / are not sensitive helps on a test set that is / is not sensitive. 

Positives:

The authors do extensive experimentation to evaluate the effect of training with various data subsets constructed by taking the most or least sensitive samples, and testing on other subsets of sensitive / not sensitive samples.

The results generally show that performance on non-sensitive subsets improves when training with non-sensitive data.

Negatives:

I'm not terribly convinced by the experimental results. It seems like the conclusion of the paper is that if the goal is to get good performance on the standard test set, you should just use the original data without taking into account sample robustness.

There are gains for specific subsets, but the paper doesn't really do a good job of convincing me why I should care about this. It seems like the end goal should be performance on whatever test set I originally collected, and not a subset where you remove particular hard or easy examples. This concern is exacerbated by the fact that the train and test data selection procedures are matched (i.e. you select points based on sample robustness and evaluate on similar selection criteria). 

Finally, I'm not sure if this notion of sample robustness is conceptually novel compared to methods like adversarial pruning (Yang 2019). The authors could have done more in linking the effects of sample robustness to the margin or generalization properties of the resulting classifier. For an example of such discussion see the recent line of work on r-consistency and nonparametric methods for adversarial robustness.

Finally, the writing style is generally convoluted and requires substantial effort to parse. The abstract sentence starting with "In this work" and first sentence of the intro are both examples of this. The problem is too pervasive for me to give exhaustive pointers, but the paper requires substantial copyediting before it is accepted or published.

Minor / trivial: 

Abstract - "In this work ..." has way too many commas.
Please use proper left quotes for Latex.
review_title: Extensive analysis, but concerns about their relevance and value beyond existing work.

Meta-review: The paper proposes sample robustness (a data-dependent measure) which is essentially a point-wise Lipschitz constant of the label map. The measure is used to choose a subset of training data for training and it measures how small of a perturbation is required to cause a label change w.r.t. label map.
initially, the paper lacked theoretical motivation and backing and the empirical studies were limited to be convincing enough. The authors added additional theoretical explanations. There were some mathematical mistakes that were fixed in the revision. 
However, that is not enough to justify the proposal fully. Therefore, I suggest the authors improve the theoretical explanation. The paper would also benefit from more empirical analysis and discussion. As is, the paper has limited significance to the community since the conclusion is not convincing enough.


The paper writing quality although improved from the original version, still has room for improvement. 

The proposed measure is simple, which can be a plus. but that means that we are also missing on some relationships and interactions between samples impact on training. Therefore, The paper will benefit from clearly discussing pros and cons of the proposed method. Moreover, discussing how this definition works in choosing the best subset of samples will improve the paper. 

i thank the authors for their effort and improving the paper in response to the reviews. However, given that myself and reviewers find the modifications enough for addressing all the concerns, I vote for rejecting the paper. Please improve the paper and resubmit to a future venue.
Decision: Reject