Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 2: Strong rejection
review_content: Summary
The paper addresses the proposes a solution for multi-task multicriteria hyperparameter optimization. The solution hinges around Pareto optimality. The technique appears to do an exhaustive enumeration in the space of hyperparameters (Equation 4). Experimental results are provided for two scenarios with two and three hyperparameters. 

Pros
The paper addresses an important problem.

Cons
1.	The paper appears to use exhaustive enumeration before finding Pateto frontier. Exhaustive enumeration is computationally expensive.
2.	The experimental results are sketchy. The final performance of the models as a result of hyper-parameter selection is not provided.
3.	Comparison with baselines is missing.

Clarifications needed
The paper can be improved by adding the following information.

1.	Section 4.1: Provide details of the tasks.
2.	Section 4.1: “Neural networks trained on ten TPUs v2, which took several days.”: How many days?
3.	Section 4.2: “sample mean/variance of the epoch number at which convergence is achieved in the test sample.”: What is meant by convergence on test samples?

review_title: Relevant topic, paper appears a bit preliminary

final_score: rating: 3: Clear rejection
review_content: This paper proposes MTMC, a method that solves the pareto optimization problem for multiple tasks and multiple criteria.

Pro:
1, concise.
2, solution seems interesting from a technical perspective.

Con:
1, No explanation on why this proposed method should work well.
2, No comparison to state-of-the-art method (or any baseline methods). Several related methods were discussed in related work, but why not compare to them, e.g. (Igel, 2005)?
3, Experiments are not convincing. Please include descriptions of this problem in (Akhmetzyanov & Yuzhakov, 2019) and add more realistic real-world problems. Please include specific description of the metric to evaluate different methods including the proposed one.

Some minor points:
1, It's unclear what N_combination is exactly and why it's different from N_parameter.
2, Typo: in Eq (4), x_sise -> x_size

Overall I think this paper is half developed and can benefit from some intuitive explanations, theoretical analysis of the proposed method and more convincing experiment.
review_title: Half-baked, needs more work

final_score: rating: 2: Strong rejection
review_content: 
This paper proposes a multi-task multicriteria hyperparameter optimization method. Experiments fail to demonstrate the performance of the described method. 

Strong points:

- The multi-objective selection procedure appears sensible
- Experiments run on state of the art models

Weak points: 

- Experiments don't say what dataset they used, don't make me look at the citation when you have about 4 pages left of space
- Real values for objectives never provided
- No baseline / comparison 
- No justification for method choice

Justification: 

The paper makes little sense to me. A lot of things are poorly explained, although somewhere in there is the core of a good idea. The whole paper needs to be rewritten with an emphasis on properly defining terms, objectives and research questions.

I strongly recommend that this paper be rejected. 

Some suggestions: 
 - I am not sure if hyperparameter optimization is the right term to describe this method. One cannot speak of hyperparameter optimization, as there is no hyperparameter optimization process taking place. In fact, this is more akin to a selection procedure.
 - Equation 9 is introduced as the vector of the optimal solution, whereas it is one of many optimal solutions. It is incorrect to say that the optimal solution is putting an equal weight on all objectives, as far as multi-objective optimization goes. 
 - Add some figures explaining the whole procedure (from hyperparameter evaluation to multi-objective selection)
 - Evaluate against baselines
 - Provide some justification for the steps of the method (i.e. in 3.2)
 - Have someone not familiar with the work revise the paper before submitting.


review_title: clear reject

final_score: rating: 3: Clear rejection
review_content: This article bases hyperparameter optimization over multi-objective by aggregating them with weights. An illustration is provided on a grid search to select Pareto optimal solutions.

Aggregation of objectives to get scalar values has been studied at length in multi-objective optimization, so it is hardly novel. See, e.g.,  Miettinen, K., Nonlinear multiobjective optimization,  Springer, 1999, 12.
Besides , choosing weights is known to be quite difficult in practice.

Concerning optimization of hyperparameters, there are many works applicable that would be much more efficient than a crude grid search. In the case of Bayesian optimization, there are works like, e.g.,:
-Swersky, K.; Snoek, J. & Adams, R. P., Multi-task bayesian optimization, Advances in neural information processing systems, 2013, 2004-2012
- Paria, B., Kandasamy, K., & Póczos, B. (2020, August). A flexible framework for multi-objective Bayesian optimization using random scalarizations. In Uncertainty in Artificial Intelligence (pp. 766-776). PMLR.
- Hernández-Lobato, D., Hernandez-Lobato, J., Shah, A., & Adams, R. (2016, June). Predictive entropy search for multi-objective bayesian optimization. In International Conference on Machine Learning (pp. 1492-1501).

Since no comparison is provided with methods from the literature, that the proposed method is not novel, I thus recommend rejection.

Typos:
Eq. 4: sise → size
P2: the nearest Pareto front to the origin?
review_title: Review of Multi-Task Multicriteria Hyperparameter Optimization

Meta-review: The paper has been discussed by the reviewers that have acknowledged the rebuttal and the authors’ responses. However, the reviewers still had the following weaknesses and concerns (not solved post rebuttal):

* Expensive procedure (e.g., exhaustive enumeration before finding Pareto frontier)
* The experiments should be more rigorous, with more realistic real-world problems.
* Missing comparison with baselines (unanimously acknowledged by the reviewers).
* No explanations and insights provided as to why the method should work well
* Clarity of the presentation

As a result, the paper is recommended for rejection. The detailed comments of the reviewers provide an actionable list of items to improve the paper for a future resubmission.

Decision: Reject