Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers combining multi-objective optimization with multiple-hypothesis testing. More specifically, they consider optimizing some objectives Q_i's while applying risk control on multiple other objectives such that Q_j\le c with high probability. They consider reducing the dimension of the high dimensional vector of candidates of configurations by only restricting themselves to the Pareto frontier. Effectiveness of their method is considered on large-scale transformer models in NLP.

Main method is based on existing work fixed sequential testing, by adapted it to the Pareto frontier setting.


strength_and_weaknesses: Strength. Reducing high-dim configurations is an interesting topic. Experiments are comprehensive.

Weakness:
1. Writing is not clear.  For example, is equation 3, should the optimization objective be of subscript c+1 instead of c? The discussion of c+1 and c+k is not clear enough, constantly switching between the two cases.
2. Lack of innovation. The main technique is based on FST. The only contribution is to suggest using the Pareto frontier to reduce the dimensionality of (tau_1, \cdots, tau_n)'s need to be considered, which is quite natural. Also, there is a discrepancy between the empirical Pareto frontier based on Q.hat and based on the real Q. The one based on real Q is the desired frontier. 
3. Notation sloppiness. In equation 4, argmin is mis-used, which is also pointed out by the author themsevle -- it is possible that no uniformly optimal solution exists.

I suggest the authors significantly polish this writing and consider more extensions.
clarity,_quality,_novelty_and_reproducibility: Not clear enough. See comments above.
summary_of_the_review: See comments above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work is built upon the results of Learn Then Test framework by introducing Pareto Testing.  The authors develop an efficient method for calibrating models such that their predictions provably satisfy multiple explicit and simultaneous statistical guarantees (e.g., uppper-bound error rates), while also optimizing any number of additional, unconstrained objectives.

This method contains two stages. Stage1: solving an unstrained  multi-objective optimization problem to recover an approximate set of Pareto-optimal configurations; Stage2: performing rigorous sequential testing over the recovered set to yield tight control of the desired risks.

The main idea of this proposed method is that  to perform testing ONLY over the Pareto optimal set. And the effectiveness is demonstrated for reliably accelerating Transformers 

strength_and_weaknesses: Strength:

This work is a great attempt by combining the combination of multi-objective optimization and "Learn and Test" for risk-control.  Intuitively, it is obvious that applying FST only on Pareto Front will be efficient (because the size of configuration is much smaller). 

Weaknesses:

The setting discussed in this paper is as:  for all c+k objectives,  the first c objectives are controlled by c user-specified risk bounds, and the rest k objectives are free to optimize.  If k =0, i.e., all the risks are controlled, how to decide the final configuration?

Other minor comments:

(1) Equation(3),  minimizing the remaining objective function $Q_c$? ( I guess it should be $Q_{c+1}$)

(2) Algorithm 1 line 11, when k>1, how to get a set of configurations representing different possible trade-offs?

(3) Figure 2 is a bit confusing, from the path showed, Q1 is not controlled under $\alpha$, instead, the configuration selected with $Q_1>\alpha$.
clarity,_quality,_novelty_and_reproducibility: This paper is well-written, and the framework is well-organized. 
summary_of_the_review: Overall, this paper is marginally above the acceptance threshold.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose Pareto Testing, a multiple hypothesis testing strategy based on Learn then test (LTT) for statistically controlling multiple risks. In contrast to prior work, the method uses an additional split of the data to identify hyper-parameter configurations that are Pareto optimal (i.e., on the Pareto frontier). This improves statistical efficiency as less hypothesis tests are required.
strength_and_weaknesses: Strengths:
- Writing, while very dense, is generally well structured and includes the main points to follow the paper despite building on extensive work in the Learn Then Test paper.
- Related work is covered thoroughly and differentiates the contributions of the paper well.
- The method generally addresses an important problem and improves over previous work by decreasing the number of hypotheses to test.

Weaknesses:
Some comments on writing:
-- Because the paper is very dense, it would occasionally be worth resorting to easier sentences. The fourth paragraph in the introduction is a good example for this.
-- I found the intro figure a bit misleading as it puts too much focus on the architecture. I understand that the experiments use architectural hyper-parameters for risk control. But the figure does not really tell me anything about the risk control itself. It is not a bad figure, but I feel it misses the main contributions.
-- The notation of minimizing a set of is a bit unclear – I assume the authors want to emphasize that we jointly want to minimize different risks. But I feel this should be made clearer – see Eq. (4) and (6) and algorithm 1.
-- The computation of the p-values – which is an integral part of the method – could be made clearer in the main paper. I could only find that in algorithm F.1 or related work and I feel the paper would benefit from making that clearer for readers less familiar with these approaches (e.g., conformal p-values are computed quite differently compared to p-values based on these bounds).

Method:
- How exactly is the Pareto frontier found in practice. The experiments include one paragraph on this, but it is actually unclear what the default method of optimization is? Is it right that exhaustive optimization over the grid is used?

Experiments:
- In general, I feel that the main contribution is improving statistical efficiency and reducing the number of hypotheses to test. I would appreciate some more concrete experiments surrounding this:
-- More explicitly showcases to what extent Pareto Testing reduces the hypotheses space – i.e., how big is the Pareto optimal set, how many hypotheses are tested compared to LTT?
-- How does the D_opt set size impact that and influence the variation or the tightness of the risk control in the end?
- In plots, I assume the are is the standard deviation, or are the min/max/X% confidence interval? I am trying to understand whether Pareto Testing has some influence on this variation.
clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: I think the paper is a good contribution for ICLR. Due to the density, I feel that writing can be improved at places and I’d appreciate comments on my other questions regarding experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a two-stage approach to flexibly optimize and test to find
hyperparameter settings that have provable guarantees with respect to the
optimized objectives. The authors describe their framework and evaluate it
empirically on a Transformer model. The presented results demonstrate the
promise of the proposed method.
strength_and_weaknesses: + important and interesting problem
+ novel angle of analysis

- approach not intuitive to understand
- performance in practice not clear
clarity,_quality,_novelty_and_reproducibility: Novel as far as I can tell and the quality seems to be high, but the paper is not very clear.
summary_of_the_review: The paper studies an interesting and important problem with real-world
consequences. It is well written, although the density of math makes it
challenging to understand. Intuitively, it is difficult to follow the presented
technique -- the paper seems to imply that I can guarantee e.g. 100% accuracy
with 100% probability, which does not make sense in practice. A detailed example
to illustrate this would be helpful, in particular as the risks for the
application are not defined in terms of absolute, but relative performance.

The empirical results are not easy to interpret either as all results are shown
dependent on a specific range of alpha values (chosen how?) and it is unclear
how this affects other approaches that are not controlled. If I decided to use
the proposed method, what results can I expect compared to other hyperparameter
optimizers, e.g. SMAC, when given the same resources?

The authors denote hyperparameters with tau, whereas most of the literature uses
lambda. It would be nice to be consistent with the literature.

In the "Model configuration" section on page 2, the example given for
performance guarantees should be _minimal_ error rates. The labels in most
figures are too small.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: I Summary

- I.1 Investigated Problem:
To overcome issues caused by the changing nature of ML frameworks objectives as well as the unreliability and sub-optimality of their corresponding solutions, the paper proposes Pareto Testing.

- I.2 Proposed Solution:
The presented solution is a multiple-hypothesis testing strategy based on the Learn Then Test (LTT) framework for statistically controlling multiple risks. The presented solution calibrates models such that their predictions are proved to satisfy multiple and simultaneous statistical guarantees while it optimizes an arbitrary number of auxiliary objectives. Pareto testing is a two-stage process where it constructs at first an approximate set of Pareto-optimal configurations which is recovered by solving an unconstrained multi-objective optimization problem. Then, rigorous sequential testing over the recovered set, which yields tight control of the desired risks, is performed.

- I.3 Validity Proof of the Proposed Solution:
Empirical evidence is provided to support the validity of the proposed solution. The authors demonstrate the computational effectiveness of the method as it accelerates the execution of large-scale Transformer models in natural language processing (NLP) applications. 

II Strengths

- II.1 From a structural point of view: The paper is organized and well-structured. It is important to mention that prior to the rebuttal, the reviewers raised concerns about the dense aspect of the submission despite its well-structured nature. The authors addressed positively this issue in the rebuttal as well as issues related to the clarity and consistency of the notation.

- II.2 From an analytical point of view:  Reviewers pointed out the interesting nature of the problem investigated by the authors in this paper and its practical benefit. It offers a new angle of analysis using Pareto testing which reduces the number of hypotheses tested and therefore provides statistical efficiency improvement. More precisely. Reviewers appreciated: 
 . The interesting nature of the addressed problem and improvement upon existing work;
 . The Learn Then Test (LTT) framework is leveraged in a meaningful way.
 . The proposed method has the potential to be of great benefit to the ML community from a practical perspective.

- II.3 From a perspective of soundness (development, unity, and coherence) and completeness (correctness): The strength points mentioned above are sufficient evidence of the soundness and completeness of the paper. The authors build their solution upon existing work. The investigated problem was already faced by one of the reviewers who enjoyed reading the submission. This reinforces the soundness aspect of the presented solution.

An additional point reinforcing the strengths mentioned above is the active interaction of the authors during the rebuttal period and their openness to concerns and questions raised by the reviewers.

III What can be thought of as weaknesses:  

- III.1 All reviewers during the AC-reviewer meeting pointed out that the work is less innovative from a theoretical perspective given that the paper is presented from a theoretical background. As a theory paper, more contribution is expected from this side. This does not diminish the practical potential of the proposed solution and its benefit to the community of machine learning.

- III.2 One of the reviewers mentioned during the AC-reviewer meeting that discrepancy in sample complexity is not taken into account.

Most of the points that could be thought of as weaknesses have been addressed.

IV. Potential of the paper:

- IV.1 From a Potential perspective (Potential of the paper to the community): Application-wise, and as it has already been underlined in the strengths section, the proposed solution has the potential to be of great benefit to the ML community. Once the decision of acceptance is announced, providing the source code could be of great benefit to the researchers interested in multi-objective and Pareto settings. We thank the authors for providing a detailed pseudo-code in section F of the paper. Providing source code would make the submission even more impactful in the long term and provide it more openness and accessibility.

justification_for_why_not_higher_score: The paper had a very good rating (6.25) with an average confidence of 3.5. The paper was considered borderline as its grade is in the margin of 4.5 and 6.25. This being said, the paper has practical benefits for the community.
justification_for_why_not_lower_score: Good submission with practical benefit to the community.
Decision: Accept: poster