Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This work tries to build a sub-pile of the test data to save the testing time with minimum effect on the test adequacy and the output distribution. In this paper, the work is done by adding a test-sample search algorithm on top of the HGS algorithm to balance the output distribution.

However, the novelty of the proposed work is limited, and there is no evidence to show the proposed algorithms can be applied to other related works. Furthermore, the result does not present a strong success: the error of output distribution is much worse than the compared work.

Other comments to the proposed manuscript are:

1. In Definition 1 the authors declare that the goal is to satisfy f(T,M)=f(T’M) and g(T,M)=g(T’M), and then in the following paragraphs they change it to f(T,M)≈f(T’M) and g(T,M)≈g(T’M) with no justification. More explanation is needed.

2. In Table 2, the authors try to compare the output distribution. To better demonstrate the change between raw testing set and proposed subset, I think that it could be better to present the metrics of distribution or the accuracy of each class instead.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper develops methods to reduce the test data size while maintaining the coverage and the effectiveness on large test data. The paper proposes  a two-phase reduction approach to select representative samples based on heuristics. Extensive experiments have shown the proposed methods can reduce 95% test samples while still obtaining similar measurements. 

The paper targets a very important problem in practice. Effectively selecting small, representative test sets can save many computational resources and greatly accelerate the research and development. Although the developed technique is quite simple, they are meaningful in practice. Overall, the work can be much improved if a theoretical framework is proposed. 


review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The paper presents a new approach to create subsets of the testing examples that are representative of the entire test set so that the model can be tested quickly during the training and leaving the check on the full test set only at the end to validate its validity.

The key idea is to create the smaller possible subset with the same or similar coverage (in the paper the neurons coverage is considered) and output distribution, maintaining the difference below a (small) threshold.

In particular, the output distribution is approximated by dividing the output range of each neuron into K intervals. In this way, an estimate of the output distribution is considered during the extraction of the representative subset of the testing data.

The whole process is divided into two phases, the first is to create a first subset using HGS, the second refines this subset in order to achieve the desired precision in the output distribution.

The paper also presents a good experimental campaign that shows good performances.

The paper is really well written and enjoyable, clear in its description and in the objectives it aims to achieve. To improve the readability a bit further, I would suggest trying to move equation 1 after or close to its reference, or at least to describe before it what KL is.
Moreover, a similar problem is present for TI, which is introduced in Algorithm 1 and described later. I would add, for example in the input section of the algorithm, a sentence stating that TI is used in getCandidate and described later.

The first sentence on page 5 seems to be incomplete as it is written. I would suggest rephrasing the sentence.

In Algorithm 2, about the two "for" cycles for i in 1,m and foreach k in 2,K, I suggest unifying them and use the same cycle (only for reasons of readability). Moreover, I think that using braces instead of parentheses would be more correct in these cycles.

It is interesting to note that in the first line for VGG19, last column of Table 3, the accuracy is lower with KL < 0.001 than with KL < 0.005. I would expect monotonicity here. Do the authors have any idea of the reasons for this?

Finally, it would be interesting to know the runtime to obtain the subsets of the test data of Table 2 required by the considered systems.

Typos:
On page 6, penultimate paragraph, the word "the" is repeated twice in the parentheses.
On page 8, at the beginning of the first sentence after Table 3, there is a comma that seems to be useless after the word "that".
On page 8, "When the termination crite gets stricter", crite should be corrected in criterion.
There is a typo in the README of the github project linked in the paper: "coveraeg data" instead of "coverage data".
review_title: Official Blind Review #1

Meta-review: This paper presents a method which creates a representative subset of testing examples so that the model can be tested quickly during the training. The procedure makes use of the famous HGS selection algorithm which identifies and then eliminates the redundant and obsolete test cases based on two criteria: (1) structural coverage as measured by the number of neurons activated beyond a certain threshold, and (2) distribution mismatch (as measured by KL divergence) of the last layer activations. The algorithm has two-phases: (1) a greedy subset selection based on the coverage, and (2) an iterative phase were additional test examples are added until the KL divergence (as defined above) falls below some threshold. 
This approach is incremental in nature -- the resulting multi-objective optimisation problem is not a significant improvement over BOT. After the discussion phase, we believe that the advantages over BOT were not clearly demonstrated and that the main drawback of BOT (requiring the number of samples) is not hindering practical applications. Finally, the empirical evaluation is performed on very small data sets and I do not see an efficient way to apply it to larger data sets where this reduction could be significant. Hence, I will recommend the rejection of this paper. To merit acceptance to ICLR the authors need to provide a cleaner presentation (especially of the algorithms), with a focus on the incremental improvements over BOT, an empirical analysis on larger datasets, and a detailed look into the computational aspects of the proposed approach.

Decision: Reject