Multi-Task Multicriteria Hyperparameter Optimization
Authors: Kirill Akhmetzyanov
Alexander Yuzhakov
Venue:  (2021)
Keywords: hyperparameter optimization
machine learning
neural network

Abstract: We present a new method for searching optimal hyperparameters among several tasks and several criteria. Multi-Task Multi Criteria method (MTMC) provides several Pareto-optimal solutions, among which one solution is selected with given criteria significance coefficients. The article begins with a mathematical formulation of the problem of choosing optimal hyperparameters. Then, the steps of the MTMC method that solves this problem are described. The proposed method is evaluated on the image classification problem using a convolutional neural network. The article presents optimal hyperparameters for various criteria significance coefficients. 

Under review as a conference paper at ICLR 2021
MULTI-TASK MULTICRITERIA OPTIMIZATION
HYPERPARAMETER
Anonymous authors Paper under double-bling review
ABSTRACT
We present a new method for searching optimal hyperparameters among several tasks and several criteria. Multi-Task Multi Criteria method (MTMC) provides several Pareto-optimal solutions, among which one solution selected with given criteria significance coefficients. The article begins with mathematical formula- tion the problem choosing optimal hyperparameters. Then, the steps of the MTMC method that solves this problem described. The proposed method is evaluated on the image classification problem using a convolutional neural net- work. The article presents optimal hyperparameters for various criteria signifi- cance coefficients.
INTRODUCTION
Hyperparameter optimization (Hutter al. 2009) is important component in the implementa- tion of machine learning models (for example, logistic regression, neural networks. SVM, gradient boosting, etc.) in solving various tasks, such classification, regression, ranking, The problem is how to choose the optimal parameters when a trained model is evaluated using several sets and several criteria.
This article describes method to solving the above problem. We will present the results of ex- periments on the selection of hyperparameters obtained using the proposed approach (MTMC) with various criteria significance coefficients.
The article organized as follows. First, we discuss related work in Section Section describes the proposed method. Section 4 presents the results of experiments on the selection of optimal hyperparameters. Section contains the conclusion and future work.
2 RELATED WORK
Hyperparameter optimization is applied to solve various problems such as computer vision (Bergstra)
et al. 2013 Dong al.) 2019). robotics (Mahmood et al. 2018 Tran et al. 2020). natural language
processing Wang et al. 2015 Dernoncourt & Lee 2016) and speech synthesis (Koriyama et al. 2014).
The problem of choosing optimal hyperparameters has long been known. Existing solutions can be considered for the following features:
Number of optimal solutions.
2. Number of tasks to be solved.
Number of criteria for choosing the optimal solution.
In this article, task means set of images with number of classes Nelasses and number of images Nimages- There are examples the same classes between tasks, the difference is how the images made (different lighting, background and used cameras). Criteria is quantitative characteristic of training / evaluation a neural network on a task (e.g. accuracy, latency or epoch of training convergence).

---

Under review as a conference paper at ICLR 2021
In (Sener & Koltun 2018 Pareto optimization method is proposed, in which the optimal solution is given for several problems simultancously. This method consists in minimizing the weighted sum of loss functions for each task. (Fliege & Svaiter 2000) describes the Pareto optimization method, which gives an optimal solution according to several criteria based on gradient descent, and this op- timization is also carried out in the learning process. In (Igel 2005), the search for Pareto-optimal solution is carried out according to several criteria. (Miettinen (0012) gives several methods for multiobjective Pareto-optimization. The method in (Bengio 2000) gives optimal hyperparameters using back propagation through the Cholesky decomposition. In (Bergstra et al. 2011). optimiza- tion performed using a random choice of hyperparameters based on the expected improvement criterion. (Bergstra & Bengio 2012) proposes method of hyperparametric optimization based on random search the space of hyperparameters.
(Snoek et al., 2012), search for optimal hyper- parameters is carried out using Bayesian optimization. (Swersky et al. (013) proposes a method for finding optimal hyperparameterss using multi-task Bayesian optimization. (Paria et al., 20200 Hemández-Lobato al. 2016) describe the Bayesian methods for multi-objective optimization.
The proposed method based on Pareto optimization. Pareto optimality means that is impossible to improve the Pareto optimal solution by any criteria without worsening it by at least other criteria. Thus, for a certain set of values of the criteria, the Pareto-selected solutions are optimal. The closest criteria the given criteria are determined by finding the minimum weighted sum of Pareto solutions (where the weight is the inverse criteria).
The novelty of MTMC method:
1. Optimization is carried out simultaneously according to several criteria and several tasks with setting the significance of the criteria.
2. The choice of optimal hyperparameters is provided after training and evaluation, which eliminates the need re-train the model.
3. The proposed method does not need to be trained.
3 THE PROPOSED METHOD
First, we describe the mathematical problem that MTMC solves, then we present the steps performed in MTMC.
3.1 FORMALIZATION OF THE PROBLEM
In the proposed method, the model is evaluated several test sets (tasks) T. The problem of finding a minimum for tasks is known as minimizing the expected value of empirical risk (Vapnik 1992).
The choosing optimal hyperparameters is formalized follows:
0 argminE, Φ)]
(1)
where θ is the set of all hyperparameters, 0 is the selected optimal hyperparameters, ø is the vector of significance coefficients of the criteria, C(-) is the estimation function of the model with the given hyperparameters 0 and the coefficients á is the task for which optimization is performed.
The developed method gives a solution to the problem (1).
3.2 DESCRIPTION OF MTMC
According to (1), the developed method should fulfill the following requirements:
1) the method should solve the minimization problem;
2) the significance of each criterion is determined by the vector of coefficients & (the higher the coefficient, the more important the corresponding criterion).
We denote the test sample of the task 7:
D.i i
Nusk
2

---

Under review as a conference paper at ICLR 2021
where æ' is the th test set has the distribution D, Nusk is the number of tasks.
Before choosing hyperparameters, for model M we obtain an evaluation matrix for the test set xf and the given evaluation criteria:
VM
(3)
M (x :: (IR R. Mpooneterr N Nasetiinatio)) + R Noontication X. Nerberia
(4)
where MM() is the model function that transforms the given set x² and with the given hyperparam- eters ④ into the evaluation matrix V. Neriteria is the number of criteria, Jsize is the dimension of the test set, N parameter is the number of hyperparameters, N combination is the number of hyperparameter
combinations.
Then, the function C is calculated for each set æ", which is formally described as follows:
f θ Φ E(V
(5)
(V; (R Menueni TR Nenini ) + R
(6)
(a)
(b)
(c)
(d)
(e)
(f)
Figure 1: Illustration of the steps of our proposed method MTMC. (a) Evaluation for all combi-
nations of hyperparameters in the criteria space. (b) Getting the evaluation boundary. (c) Getting Pareto optimal solutions. (d) Defining the criteria vector. (c) Projecting Pareto optimal solutions onto the criteria vector. (f) Finding the closest projection the origin of coordinates.
Figure [ shows the steps of MTMC to obtain the optimal solution for the given criteria. MTMC method gives Pareto optimal solutions which the following steps are performed:
1. The vectors from the evaluation V (the number of such vectors is Neriteria) is in the space of given criteria.
2. Pareto gives solutions that closer optimal by some criterion, and farther from optimal by other criteria. We get Pareto optimal solutions V C V the nearest Pareto front to the origin of the criteria space coordinates:
ParetoFront (V), V. R Nog Noitai
where Nop is the number of Pareto optimal solutions.
3. The optimal solutions V are scaled according to each criterion the interval [0; 1]:
V. Emin Unin E Neniens Tmin E IR Naranaa ; = 1. ......
Vicated
N
(8)
Umax
Unin
3

---

Under review as a conference paper at ICLR 2021
where U maz is the vector of maximum values of V for each criterion, Umin is the vector of minimum values of V for each criterion.
Thus, the optimal solution is the solution closest to the origin of coordinates, and if any solution ū E V is the origin of coordinates, then it is optimal for any d.
The vector Φ space criteria defined.
We introduce the vector of the optimal solution, which is the middle of the segment [0; 1] in the axes
of the criteria space:
2)
Popt
Vi: do
= @Norttrra 2
(9)
Conditions for Φ are:
- Popt if Φ; =0 Φ E [0; 1], otherwise.
(10)
5. It is necessary to determine which hyperparameter in the criteria space is closer to the given criterion. Project the vectors from the matrix V Vecated onto the vector 6:
VT Vacaled Φ |0l
Vpros
R-ope
(11)
From (9) and (11) follows that if the vectors and &opt are collinear, then:
YY : X Popt
Vpros Σ
1
Vecated:
Vecated,
V
Vocuted
(12)
10l
That is, in the case equality of all elements of o, the minimization problem reduces to finding the minimum L1- Z1-norm Vecated-
From (11) it also follows that if some component of the vector o is equal zero, then the corre- sponding criterion will not affect the choice the optimal hyperparameter. all criteria are equal zero, except for one, then only the criterion with nonzero component of the vector will affect the choice of optimal hyperparameters.
We find hyperparameters which the minimum the vector Vproy is reached, which is equiv- alent to finding the minimum weighted sum of vector values:
argmin proj
(13)
Figure shows an example solution using MTMC for random numbers in the three-dimensional space.
4 CONDUCTING EXPERIMENTS
First, the evaluation matrix V for the selected model M is obtained. Then, for various combinations of components of á, optimal hyperparameters are selected using MTMC.
4.1 OBTAINING THE EVALUATION MATRIX
The developed MTMC method is applied solve the problem of image classification. The problem we are solving is described in the article (Akhmetzyanov Yuzhakov 2019). The problem is the choice of such hyperparameters, which achieve the highest classification accuracy among several tasks. Each task consists of several images of one of two objects such as a plastic bottle and other object. The tasks differ in how the images made, namely lighting, background and used cameras.
In (Akhmetzyanov & Yuzhakov 2018), we selected the MobileNet neural network architecture (Howard et 2017) as a mathematical model for image processing.
4

---

Under review as a conference paper at ICLR 2021
1.0
0.8
0.6
0.4
0.2
0.0
18.8
8
4 0
6.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 2: Example of solution given by MTMC, green points denote Pareto optimal solutions, blue vector is the vector é, yellow point denotes the optimal solution given by MTMC for a given ø.
The search for optimal hyperparameters changing the learning speed based on the epoch Ir base Ir Ir decay (where base Ir is the initial learning rate, Ir decay is the coefficient of decreasing learning rate, epoch is the number of epochs) and cyclical learning (Smith) 2017). In cyclic learning, there are three ways to change the learning rate:
carried out among two popular training methods:
1. triangular is fixed initial learning rate (base_Ir), maximum fixed learning rate (max_Ir), learning rate increases from base Ir to max and decreases from max_Ir to base linearly.
2. Iriangular2 is fixed initial learning rate (base Jr), maximum learning rate (max_Ir), learning rate, in triangular, varies linearly, but max_/r in the learning process is halved.
3. eangr is fixed initial learning rate (base_Ir). maximum learning rate (max_Ir), learning rate also changes linearly, but max in the learning process decreases exponentially.
In the first learning method, the hyperparameters are the value of the initial learning rate (base_in and the coefficient of decreasing learning rate (Ir_decay). In the second method, hyperparameters way to change the learning rate (cyclic_mode), the value of the initial learning rate (base_lr) and maximum learning rate (max ir).
For each hyperparameter, range of change and a constant step of change within the range were selected. For training, Grid search was used among Ncombination 100 combinations of hyperpa- rameters.
For each combination of hyperparameters, training was carried using cross-validation k-fold (Stone 1974) with 10 folds. For training, Keras (Chollet et al. 2015) and TensorFlow (Abadi et al.) 2015) were used, The training lasted 15 epochs; the test was carried out on Niast different test sets. That is, 100 10 1000 is number of different neural networks, 15 1000 15000 neural networks evaluations are conducted, 15000 5 = 75000 evaluation results are obtained. Neural networks trained ten TPUs v2, which took several days.
Among all epochs, for each fold and for each test set, the maximum accuracy is selected, as well the number of the epoch at which the maximum accuracy is achieved. The following values calculated for each test set among the folds: the expected value and the variance of the classifica- tion error, the expected value and the variance of the epoch number at which convergence the

---

Under review as a conference paper at ICLR 2021
test set is achieved. We have obtained evaluation matrix among all neural networks with their hyperparameters and among all test samples.
4.2 PROCESSING THE EVALUATION MATRIX
Based on (I), for each criterion, among all the samples, the expected value is considered. That is, for all test sets, the criteria: (i) the sample mean of the classification error. (ii) the sample variance of the classification error, (iii) the sample mean and (iv) sample variance of the epoch number which convergence on the test set is achieved. These values are the criteria for evaluating hyperparameters for a certain test set (matrix V from (3)) with the number of criteria Nariteria
V is calculated from (7), the number of Pareto optimal solutions obtained is Nopt 25. Optimal hyperparameters, i..., V, are presented in Appendix A.
The vector the optimal solution according to (9) for Nersteria = 4 Popt = {0.5; 0.5; 0.5). Next, calculations carried out according to (8) and (11), and for various optimal solutions chosen according to (5). These optimal solutions are presented in Appendix B. Also, Appendix C shows the learning curve of the neural network for each test set and each epoch for the obtained optimal hyperparameters.
5 CONCLUSION
In this work, we proposed a new method for hyperparameter optimization among several tasks and several criteria. We trained several neural networks with various hyperparameters solve the im- age classification problem. Then, for these neural networks, evaluation matrices were obtained on several tasks. We applied MTMC to these matrices and got optimal solutions with different signifi- cance coefficients. In the future, we will work to create meta-learning method that solves the same problem as the method described this article, but optimization will be performed among various models.
ACKNOWLEDGMENTS
The reported study was partially supported by the Government of Perm Krai, research project No. C-26/174.6.
REFERENCES
Martín Abadi, Ashish Agarwal, al. TensorFlow: Large-scale machine learning heterogeneous systems, 2015. URL http:/ttt///eeeoofolooooooo
Software available from tensorflow.org.
Kirill Akhmetzyanov and Alexander Yuzhakov. Convolutional neural networks comparison for waste sorting tasks. Izvestiya SPŁGETU "LETI", (6):27, 2018.
Kirill Akhmetzyanov and Alexander Yuzhakov. Waste sorting neural network architecture opti- mization. In 2019 International Russian Automation Conference (RusAutoCon), pp. 1-5. IEEE, 2019.
Yoshua Bengio. Gradient-based optimization of hyperparameters. Neural computation, 12(8):1889 1900, 2000.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb):281-30) 2012.
James Bergstra, Daniel Yamins, and David Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International conference machine learning, pp. 115-123, 2013.
James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. In Advances in neural information processing systems, pp. 2546-2554 2011.
François Chollet et al. Keras. https://keras: 2015.

---

Under review as a conference paper at ICLR 2021
F. Demoncourt and J. Y. Lee. Optimizing neural network hyperparameters with gaussian processes for dialog act classification. 2016 IEEE Spoken Language Technology Workshop (SLT), pp- 403,413, 2016. doi: 10. 0..1199/LTT2016.84666999
X. Dong, J. Shen, W. Wang, L. Shao, H. Ling, and F. Porikli. Dynamical hyperparameter opti- mization via deep reinforcement learning tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 2019. doi: 10.1 1111099PAAM...2222296666...
Jörg Fliege and Benar Fux Svaiter. Steepest descent methods for multicriteria optimization. Mathe- matical Methods of Operations Research, 4193):479-494, 2000. 2000.
Daniel Hernández-Lobato Jose Hernandez-Lobato, Amar Shah, and Ryan Adams. Predictive en- tropy search for multi-objective bayesian optimization. In International Conference on Machine Learning. pp. 1492-1500 1501, 2016.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 170.:0/774.048//// 2017.
Frank Hutter, Holger H Hoos, Kevin Leyton-Brown. Brown, and Thomas Stützle. Paramils: an automatic al- gorithm configuration framework. Journal of Artificial Intelligence Research, 36:267-306, 2009.
Christian Igel. Multi-objectivv model selection for support vector machines. In International Con- ference Evolutionary Multi- Crttti-Crtteoonn Optimization, pp. 536. Springer, 2005.
T. Koriyama, T. Nose, and T. Kobayashi. Parametric speech synthesis based on gaussian process regression using global variance and hyperparameter optimization. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3834-3834 2014. doi: 10. N..100//CASSP.20144 4.6553119.
A Rupam Mahmood, Dmytro Korenkevych, Gautham Vasan, William Ma, and James Bergstra. Benchmarking reinforcement learning algorithms on real-world robots. In Conference on Robot Learning, pp. 561-591, 2018.
Kaisa Miettinen. Nonlinear multiohjective optimization, volume 12. Springer Science & Business Media, 2012.
Biswajit Paria, Kirthevasan Kandasamy, and Barnabás Póczos. A flexible framework for multi- objective bayesian optimization using random scalarizations. volume 115 of Proceedings of Ma- chine Learning Research, pp. 766-776. PMLR, 22-25 Jul 2020.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Advances in Neural Information Processing Systems, pp. 527-538, 2018.
Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE Winter Confer- ence on Applications of Computer Vision (WACV), pp. 464-472. IEEE, 2017.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pp. 2951-2959, 2012.
Mervyn Stone.
Cross-validaton choice and assessment of statistical predictions. Journal of the Royal Statistical Society: Series B (Methodological), 36(2):11--33, 1974.
Kevin Swersky, Jasper Snoek, and Ryan Prescott Adams. Multi-task bayesian optimization. In NIPS. pp. 2004-2011 2012, 2013.
P. Tran, G. N. Nguyen, and V. Hoang. Hyperparameter optimization for improving recognition efficiency of an adaptive learning system. IEEE Access, 11160569-160580, 2020. doi: 10.1
Vladimir Vapnik. Principles of risk minimization for learning theory. In Advances in neural infor-
mation processing systems, pp. 831-838, 1992.
Lidan Wang. Minwei Feng, Bowen Zhou, Bing Xiang, and Sridhar Mahadevan. Efficient hyper- parameter optimization for nlp applications. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2112-2117, 2015.
7

---

Under review as a conference paper at ICLR 2021
Table 1: Pareto optimal solutions for the first learning method
base Ir decay
0.001
0.75
0.001
0.8
0.005
0.75
0.01
0.9
0.01
0.95
Table 2: Pareto optimal solutions for the second learning method
base Ir
max Jr
cyclic.mode
0.0001
0.005
exp-range
0.0001
0.005
triangular2
0.0005
0.001
exp-range
0.0005
0.005
triangular2
0.001
0.0001
triangular2
0.001
0.0005
triangular
0.001
0.001
exp_range
0.001
0.005
triangular2
0.005
0.0001
triangular
0.005
0.005
triangular
0.01
0.0001
triangular
0.01
0.0001
triangular2
0.01
0.005
triangular
0.01
0.005
triangular2
0.01
0.01
triangular
0.0001
0,0001
triangular
0.0005
0.001
triangular
0.0005
0.01
triangular2
0.001
0.005
triangular
0.005
0.01
triangular
A PARETO OPTIMAL HYPERPARAMETERS
Table and Table 2 show the Pareto optimal hyperparameters for the learning methods.
8

---

Under review as a conference paper at ICLR 2021
B MTMC OPTIMAL HYPERPARAMETERS
Table 3 shows the optimal hyperparameters 9 obtained using MTMC method for given significance coefficients of the criteria
C ACCURACY OF MTMC OPTIMAL SOLUTIONS
The figures 3-7 show optimal solutions chosen by MTMC method: 95% confidence intervals of the dependence of accuracy on the fold / epoch and maximum accuracy for each of the folds (left) and box plot with maximum accuracy for all test sets (right).

---

Under review as a conference paper at ICLR 2021
Table 3: Optimal hyperparameters for relevant criteria significance coefficients
Φ1
Ø
0
0.5
0.5
0.5
0.5
1r0001,
max_Ir=0.01,
cyclic.mode-trianguaaa
0.0
0.5
0.5
0.5
base_Ir=0.01.
max_Ir=0.01,
cyclic modc-triangular base.1r=0.01.
1.0
0.5
0.5
0.5
max
cyclic.mode-trianglaar base_Ir=0.005.
0.5
0.0
0.5
0.5
max_Ir=0.0001.
cyclic mode-triangular base.Ir=0.01.
0.5
1.0
0.5
0.5
max Ir0.00
cyclic.mode-triangulaar base_Ir=0.01,
0.5
0.5
0.0
0.5
max_1r=0.01,
cyclic mode-triangil
0.5
0.5
1.0
0.5
basc.Ir=0.01.
max 1r=0001, cyclic traeerr
0.5
0.5
0.5
0.0
basc.Ir00.0001,
max max_ir=0.005.
cyclec.mode-triangulaa2 base.Ir=0.01.
0.5
0.5
0.5
1.0
max.Ir0.01,,
cyclic mode-tiaannula max_Ir=0.005.
0.0
0.0
0.5
0.5
1r.decay=0.75
1.0
1.0
0.5
0.5
base_Ir=0.01,
max_Ir=0.01,
cyclic.mode-trangularr
0.5
0.5
0.0
0.0
base_1r=000001,
max.1r=0.005.
pyciccmode=ttringulla22
0.5
0.5
1.0
base.Ir=0.01.
max_Ir00.01,
cydlic.mode-triangllar
1.0
0.0
0.0
0.0
base_Ir=0.0001,
max_Ir=0.005.
cyclic
0.0
1.0
0.0
0.0
base.1r=0.0001,
max.Ir=0.005.
cyclic.node-ttiingulla22
0.0
0.0
1.0
0.0
max_Ir=0.005.
1r.decay=0.75
0.0
0.0
0.0
1.0
base_lr=0.0000,
max_Ir=0.00,,
cyclic.nod==exp-raage
10

---

Under review as a conference paper at ICLR 2021
ncc by opoch locd)
Acc by spoch (tost soc,
oep4m
Acc tecaaas
by upuch (tssl_ass_2)
240
Deporh
told tecct
380
R53
Acc Tull (io.
Acc fold eoc
6.1
135
Acc by 1old (besi, acc
Acc fold cost acc 4)
Hax ant by fold (are))
Hax act by hold treet treat 90
0300
E.%%
by Cast 200_
by ocss_
6090
030
8450)
84355
Figure 3: base 1r=0.01, max Ir=000,, cyclic mode=triangular
11

---

Under review as a conference paper at ICLR 2021
Acc spach (nec)
Acc by epach Itest soc, 00
by (teet 13
Daal
24
Rcpech
by epoch thant, acc 3)
epoch (ast acc
23
Acc by told lscc)
by fuld (ted, auc,
075
0953
315
M
696
ore by fiid (wer acc_
ore by fill (twer scc
33350e1
acc_0l
Max oct hold then acc 1)
Max nct by fold Dent arc: 21
0348
Accumce
345
+34
Figure 4: Ir=e.000 max 0r=000011, cyclic.mode-trinnulatt
12

---

Under review as a conference paper at ICLR 2021
Acc by spoch Coet 11
by sparh (Oee aec
67
by spach (omd ecc.3)
spach (oea,
67
by acc
Ace by fold (ren jaze.
Acc by fold beal
05
02
Max not by Ower))
Max nee frin Orwet aor
toid (bac)
Mac teld gled,
638
ocd sot_2)
Max scc
C055
Figure 5: 1r=00001000 max max.1r=0.003 weeleccceeeeeeeeeeeeeeeee mode=triangular22
13

---

Under review as a conference paper at ICLR 2021
Acc by opoch (aCC)
spech (ost_acc_
0:1
850
050
845
Doet wor_4)
640
Acc loccl
noc by fold aect_ 01
Socnecy
2000
Asc by lold (est_
Acc by fuld [teA_
0.00
07
Reonect
04
Acc by fold (ectLaac,. aoc_.)
by foid [teA_,cc_44
09675
soece
Inoi
Max doe by are, 31
Mix act by fold thes poc,
855
Figure 6: mrx.00=0.005, Ir.decay=0.75
14

---

Under review as a conference paper at ICLR 2021
laoc)
0.00
839
#spach
(aee))
0.10-
6.22
6700
Acc hold acc.
thet
Acc fold
to lhet
Max fold tacel
Max fots thst snc
6.00
Eume
643
K0GG
855
(tast,
Max tots thstt anc.
6.10
855
853
(tect
Mact tots ttet
Figure 7: base mose_Ir0.0005.. max max_Ir=0.001, cyclic_
15


Reviews:
[{"review_id": "ui3-TRi0OnZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThe paper addresses the proposes a solution for multi-task multicriteria hyperparameter optimization. The solution hinges around Pareto optimality. The technique appears to do an exhaustive enumeration in the space of hyperparameters (Equation 4). Experimental results are provided for two scenarios with two and three hyperparameters. \n\nPros\nThe paper addresses an important problem.\n\nCons\n1.\tThe paper appears to use exhaustive enumeration before finding Pateto frontier. Exhaustive enumeration is computationally expensive.\n2.\tThe experimental results are sketchy. The final performance of the models as a result of hyper-parameter selection is not provided.\n3.\tComparison with baselines is missing.\n\nClarifications needed\nThe paper can be improved by adding the following information.\n\n1.\tSection 4.1: Provide details of the tasks.\n2.\tSection 4.1: “Neural networks trained on ten TPUs v2, which took several days.”: How many days?\n3.\tSection 4.2: “sample mean/variance of the epoch number at which convergence is achieved in the test sample.”: What is meant by convergence on test samples?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Zha6jGCMcp6", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This article bases hyperparameter optimization over multi-objective by aggregating them with weights. An illustration is provided on a grid search to select Pareto optimal solutions.\n\nAggregation of objectives to get scalar values has been studied at length in multi-objective optimization, so it is hardly novel. See, e.g.,  Miettinen, K., Nonlinear multiobjective optimization,  Springer, 1999, 12.\nBesides , choosing weights is known to be quite difficult in practice.\n\nConcerning optimization of hyperparameters, there are many works applicable that would be much more efficient than a crude grid search. In the case of Bayesian optimization, there are works like, e.g.,:\n-Swersky, K.; Snoek, J. & Adams, R. P., Multi-task bayesian optimization, Advances in neural information processing systems, 2013, 2004-2012\n- Paria, B., Kandasamy, K., & Póczos, B. (2020, August). A flexible framework for multi-objective Bayesian optimization using random scalarizations. In Uncertainty in Artificial Intelligence (pp. 766-776). PMLR.\n- Hernández-Lobato, D., Hernandez-Lobato, J., Shah, A., & Adams, R. (2016, June). Predictive entropy search for multi-objective bayesian optimization. In International Conference on Machine Learning (pp. 1492-1501).\n\nSince no comparison is provided with methods from the literature, that the proposed method is not novel, I thus recommend rejection.\n\nTypos:\nEq. 4: sise → size\nP2: the nearest Pareto front to the origin?", "extra_scores": {}, "extra_text": {}}, {"review_id": "V-h0kHyB6Np", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes MTMC, a method that solves the pareto optimization problem for multiple tasks and multiple criteria.\n\nPro:\n1, concise.\n2, solution seems interesting from a technical perspective.\n\nCon:\n1, No explanation on why this proposed method should work well.\n2, No comparison to state-of-the-art method (or any baseline methods). Several related methods were discussed in related work, but why not compare to them, e.g. (Igel, 2005)?\n3, Experiments are not convincing. Please include descriptions of this problem in (Akhmetzyanov & Yuzhakov, 2019) and add more realistic real-world problems. Please include specific description of the metric to evaluate different methods including the proposed one.\n\nSome minor points:\n1, It's unclear what N_combination is exactly and why it's different from N_parameter.\n2, Typo: in Eq (4), x_sise -> x_size\n\nOverall I think this paper is half developed and can benefit from some intuitive explanations, theoretical analysis of the proposed method and more convincing experiment.", "extra_scores": {}, "extra_text": {}}, {"review_id": "FOBpG5pask", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposes a multi-task multicriteria hyperparameter optimization method. Experiments fail to demonstrate the performance of the described method. \n\nStrong points:\n\n- The multi-objective selection procedure appears sensible\n- Experiments run on state of the art models\n\nWeak points: \n\n- Experiments don't say what dataset they used, don't make me look at the citation when you have about 4 pages left of space\n- Real values for objectives never provided\n- No baseline / comparison \n- No justification for method choice\n\nJustification: \n\nThe paper makes little sense to me. A lot of things are poorly explained, although somewhere in there is the core of a good idea. The whole paper needs to be rewritten with an emphasis on properly defining terms, objectives and research questions.\n\nI strongly recommend that this paper be rejected. \n\nSome suggestions: \n - I am not sure if hyperparameter optimization is the right term to describe this method. One cannot speak of hyperparameter optimization, as there is no hyperparameter optimization process taking place. In fact, this is more akin to a selection procedure.\n - Equation 9 is introduced as the vector of the optimal solution, whereas it is one of many optimal solutions. It is incorrect to say that the optimal solution is putting an equal weight on all objectives, as far as multi-objective optimization goes. \n - Add some figures explaining the whole procedure (from hyperparameter evaluation to multi-objective selection)\n - Evaluate against baselines\n - Provide some justification for the steps of the method (i.e. in 3.2)\n - Have someone not familiar with the work revise the paper before submitting.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has been discussed by the reviewers that have acknowledged the rebuttal and the authors’ responses. However, the reviewers still had the following weaknesses and concerns (not solved post rebuttal):

* Expensive procedure (e.g., exhaustive enumeration before finding Pareto frontier)
* The experiments should be more rigorous, with more realistic real-world problems.
* Missing comparison with baselines (unanimously acknowledged by the reviewers).
* No explanations and insights provided as to why the method should work well
* Clarity of the presentation

As a result, the paper is recommended for rejection. The detailed comments of the reviewers provide an actionable list of items to improve the paper for a future resubmission.

Decision: Reject