Efficient Hyperparameter Optimization Through Tensor Completion
Authors: Aaman Rebello
Kriton Konstantinidis
Yao Lei Xu
Danilo Mandic
Venue: Submitted to ICLR 2023 (2023)
Keywords: hyperparameter optimization
tensor completion

Abstract: Hyperparameter optimization is a prerequisite for state-of-the-art performance in machine learning, with current strategies including Bayesian optimisation, Hyperband, and evolutionary methods. Whereas such methods have been shown to improve performance, none of these is designed to explicitly take advantage of the underlying data structure.  To this end, we introduce a completely different approach for hyperaparameter optimization, based on low-rank tensor completion. This is achieved by first forming a multi-dimensional tensor which comprises performance scores for different combinations of hyperparameters. Based on the realistic underlying assumption that the so-formed tensor has a low rank structure, this then allows for reliable estimates of the unobserved validation scores of combinations of hyperparameters to be obtained through tensor completion, and from only a fraction of known elements. Through extensive experimentation on various datasets and learning models, the proposed method is shown to exhibit competitive or superior performance to the state-of-the-art hyperparameter optimization strategies. Distinctive advantages of the proposed method include its ability to simultaneously handle any hyperparameter type (e.g., kind of optimizer, number of neurons,  number of layer, etc.), its relative simplicity compared to the competing methods, as well as the ability to suggest multiple optimal combinations of hyperparameters.



Under review as a conference paper at ICLR 2023
EFFICIENT HYPERPARAMETER THROUGH TENSOR COMPLETION
OPTIMISATION
Anonymous authors Paper under double-blind review
ABSTRACT
Hyperparameter optimisation is prerequisite for state-of-the-art performance in machine learning, with current strategies including Bayesian optimisation, hy- perband, and evolutionary methods. While such methods have been shown improve performance, none of these is designed to explicitly take advantage of the underlying data structure. this end, we introduce completely different approach for hyperparameter optimisation, based on low-rank tensor completion. This achieved by first forming a multi-dimensional tensor which comprises per- formance scores for diffferent combinations of hyperparameters. Based on the realistic underlying assumption that the so-formed tensor has a low-rank struc- ture, reliable estimates of the unobserved validation scores of combinations of hyperparameters can be obtained through tensor completion, from knowing only fraction of the elements in the tensor. Through extensive experimentation on various datasets and learning models, the proposed method is shown to exhibit competitive or superior performance tion strategies. Distinctive advantages of the proposed method include its ability to simultancously handle any hyperparameter type (c.g.. kind of optimiser, number of neurons, number of layer, etc.). its relative simplicity compared to competing methods, as well the ability suggest multiple optimal combinations of hyper- parameters.
state-of-the-an hyperparameter optimisa-
INTRODUCTION
Machine learning (ML) applications have been steadily growing in number and scope over recent years, especially in Computer Vision and Natural Language Processing. This growth is mainly attributed to the ability of large deep learning (DL) models to learn very complex functions. Con- sequently, the performance of such models (but even of simpler models that perform less computa- tionally heavy tasks) is critically dependent on fine-tuning of their internal hyperparameters. Given the importance of hyperparameter optimisation and the ever increasing complexity of training mod- ML models. efficient tuning of hyperparameters has become area of utmost importance, not only in obtaining state-of-the-ad performance, but also in alleviating the complexity the training process. It is therefore not surprising that large effort of the machine learning community has been focused on developing efficient hyperparameter optimisation methods. Commonly used methods in- clude grid search, random search, methods based on Bayesian optimisation, multi-fidelity optimisa- tion and evolutionary strategies. For a comprehensive review of currently available hyperparameter optimisation methods we refer the reader Yu & Zhu (222)).
Despite success, current approaches are either rather heuristic or depend on strong underlying as- sumptions. To this end, we introduce a radically different approach based on exploiting the low-rank structure of the hyperparameter space. For example, we expect that given optimiser with its learn- ing rate set to
will most likely yield similar validation loss performance to a learning rate of 9.9e - Extending this argument to multiple dimensions, so as to reflect multiple hyperparameters, we aim to identify highly promising subspaces in the vast space hyperparameter combinations by evaluating only small fraction of these combinations. More specifically, the proposed method models the set of all possible hyperparameter combinations as a multidimensional tensor. Each en- try in this tensor corresponds to score indicating relative suitability of particular hyperparameter combination with respect to others (c.g., validation loss). By evaluating a subset of these combi- nations, construct incomplete tensor with only fraction of its elements known. Assuming

---

Under review as a conference paper at ICLR 2023
the complete tensor is of low rank, the unknown entries be estimated using low-rank tensor completion techniques. This makes it possible to predict the relative performance of different hy- perparameter configurations without the need for their explicit evaluation. The best hyperparameter configurations can then be found by searching the so completed tensor.
To take full advantage of the tensor completion framework, propose sequential tensor comple- tion algorithm which narrows down the hyperparameter search space based on identified promising subspaces from previous tensor completion cycles. Furthermore, for each cycle, we employ the Cross method (Zhang 2019)). a tensor sampling scheme which ensures that as few hyperparameter evaluations as possible are required for accurate tensor completion. We show that such an approach results in a speed of optimisation that is highly competitive with, and often surpassing, other state- of-the-art hyperparameter optimisation techniques. Comprehensive numerical results and extensive experimentation illustrate the potential of the proposed framework as competitive and intuitive, yet physically meaningful, alternative option for efficient hyperparameter optimisation.
The rest of the paper is organised as follows. Section 2 discusses related work. After briefly dis- cussing key tensor preliminary concepts in Section 3 we present a validation the assumed low- rank property of the hyperparameter tensor in and the proposed tensor completion algorithm for hyperparameter optimisation in Section 5 Next, comprehensive experimental results are presented in Section 6 followed by the Conclusion in Section
2 RELATED WORK
Deng Xiao (2022) provide a meta-learning approach based on low-rank tensor completion which allows the optimal hyperparameter configuration in new machine learning problem to be predicted based on the optimal configurations found other related problems. Our work crucially different since it does not require the results of related problems any other prior knowledge to optimise given machine learning problem. A plethora of hyperparameter optimisation techniques exist that also do not require prior knowledge of the problem e.g. random search Bayesian optimisation. However, to the best of our knowledge, there is no such hyperparameter optimisation method based on low-rank tensor completion. Furthermore, we have been unable find any work hypothesising that the performance distribution of various hyperparameter combinations has an underlying low- rank structure, even though this is a natural assumption for any physically meaningful data structure.
3 TENSOR PRELIMINARIES
Low-rank tensor completion is based on the premise that elements in low rank tensor exhibit certain interrelationships. This makes it possible infer values of unknown elements based on the elements whose values are known. Research abounds with algorithms for tensor completion e.g., Bengua al. (2017). Song et al. (2018), Liu et al. (2014) or Acar et al. (201T); however, most of these algorithms assume there is no prior knowledge of which elements of the true tensor are known. In the strategy proposed by this paper, one able "choose" which elements of the tensor are known by choosing which hyperparameter combinations to evaluate. Since each hyperparameter evaluation consumnnnu,ing, is important to achieve accurate tensor completion with as few known elements as possible; this capability is provided by the Cross technique for efficient low rank tensor completion from Zhang (2019).
To explain this technique, it is necessary to explain two important prerequisites: Tucker decompo- sition and Tucker rank. Any N-dimensional tensor X' can be expressed as a Tucker decomposition, consisting of an N-dimensional core tensor 9 and N factor matrices (m),, one for each dimension. The relationship between the tensor X' and its Tucker decomposition is
X = 9 x 40) X2 42)
XN XN ((N)
The operation x, denotes the mode-n product. Consider a tensor X' E Rixxl...xly and matrix A E IR such that S N. The n-mode product of X' and A is tensor whose elements
2

---

Under review as a conference paper at ICLR 2023
(X Xn Al 1ditttinds Tii fa 1 datiidd Ojit
where r and are respectively elements of X and A with indices given by their subscripts. Based on the Tucker decomposition, the Tucker rank of tensor X' is list containing the smallest possible dimensions of g in a Tucker decomposition that accurately reconstructs X. Tucker rank is the definition of tensor rank used throughout this paper. For more details, refer to/Cichocki al. (2015).
Let the tensor X' to be completed have dimensions I, x Iz...IN. The Cross technique requires the Tucker rank of the complete tensor be assumed beforehand let this assumed rank be [rı, r... It provides a heuristic to accurately estimate the complete tensor by sampling Y1 x '2...-N + v1 x (I1 - (1) + r2 (I2 r x (IN rN) elements the authors prove this is the minimum number of elements required for accurate tensor completion at given assumed rank. The heuristic samples body tensor of dimensions 71 r2...NN: starting from Xo.o.... and rn arm vectors for each nº tensor dimension that stretch along the entire dimension and must intersect with the body. The authors of ohangg (2019) provide 2 algorithms to generate a complete tensor estimate from these samples; throughout this paper we use "Noisy Tensor Completion with Cross Measurements", which accounts for the presence of "noise" that prevents the tensor from being perfectly low-rank.
4 INVESTIGATION OF THE Low RANK PROPERTY OF HYPERPARAMETER TENSORS
Before introducing the proposed hyperparameter optimisation algorithm, it is necessary to provide validation to the assumption that tensor constructed from validation losses of hyperparameter com- binations, when constructed in particular manner, is of low (Tucker) rank. This section aims to illustrate this through experiments on widely used machine learning setups.
4.1 MAPPING THE HYPERPARAMETER SEARCH SPACE TO THE TENSOR
The first step is to discuss how construct the tensor T based on the sets of possible values i.e., search spaces for the hyperparameters. Firstly, there are N hyperparameters, the tensor T would be N-dimensional. Each hyperparameter H corresponds straint on which dimension this can be. In experiments, we consider three kinds of hyperparameters based their search spaces: categorical: set of category- -aaed values; uniform inleger: uni- formly spaced sequence of integers and uniform real a uniformly spaced sequence of real numbers.
dimension of T there is no
The search space for a categorical hyperparameter, H is specified as list: consider the activation function in neural networks as an example: ("ReLU", "tank", "sigmoid"). If H corresponds to the 1th dimension of 'T then, based on the example provided, elements of T with index 0 for the 1th dimension correspond to all hyperparameter combinations with Heat = ='ReLU". Similarly, indices and correspond to "tanh" and sigmood', respectively. The search space for uniform integer or real hyperparameter is specified through three values: start s, resolution interval r, and end e. For uniform integer hyperparameters, r is an integer, while for uniform real r is real. The search space is then {8, + r, + 2r, ..., min(e, [(e s)/r] x r)}. Let Hani be uniform integer real hyperparameter on the m dimension of T. The elements T whose index for the mh dimension is i correspond to all hyperparameter combinations having Huni
XTT.
There is thus a mapping between hyperparameter combinations and tensor indices, generated based on the set of search spaces for the hyperparameters. Each tensor element represents the validation loss of the hyperparameter combination corresponding to that index. We hypothesize that a tensor constructed in this way is (at least approximately) of low rank.
4.2 SETUP FOR TENSOR COMPLETION EXPERIMENTS
Experiments were performed with the following machine learning problems: support-vector ma- chine (SVM) with polynomial kernel in binary classification of the iris data set from Fisher (1936) (abbreviated SVM-P-iris): K-NN regression on the diabetes data set from Efron et al. (004)) (KNN- R-diab), and random forest binary classification of the wine data set (RF-wine). The wine data set
3

---

Under review as a conference paper at ICLR 2023
of size 178 was modified for binary classification by retaining only samples from classes 0 and to give a data set of size 130.
For SVM-P-iris, hyperparameters were used to construct the (4-dimensional) tensor: the SVM regularisation hyperparameter, and the degree, constant term and scaling factor the polynomial kernel. For KNN-R-diab, 3 hyperparameters K-NN algorithm, the order of Minkowski norm (used to calculate distances between points) and the weighting heuristic (uniform or distance-based) for values of the K neighbours when making pre- diction at any data point. For RF-wine, 5 hyperparameters were optimised: the number of decision trees in the forest; maximum depth of any tree; minimum number of data samples needed split tree node; number of data features to consider when splitting a tree node, and a Boolean categorical hyperparameter determining whether subset or the entire training data is used to construct each tree. The search spaces for all these hyperparameters are provided in Appendix [A.] For each of these problems p. a tensor Tp was generated based on the mapping discussed in part/4.1 Tp was then sampled and estimated through tensor completion based the Cross heuristic. In all problems, val- idation loss was calculated using cross-validation with folds predefined for each data set. The loss metric used calculate validation loss varied across problems: hinge loss for SVM-P-iris, logcosh loss for KNN-R-diab and Kullback- Leibler divergence for RF-wine.
optimised: the number of neighbours K of the
4.3 TENSOR COMPLETION RESULTS
Table represents the result of tensor completion using assumed Tucker rank of for each dimension i.e., Fi = '2...'' = 1. Table representt the best result obtained when using any Tucker rank different from the used in Table | The columns represent metries devised compare the predicted tensor from tensor completion Tp with the true tensor Tp. NND stands for "normalised norm difference", and is given by:
NND 1 Tpll 1|T/11
(3)
where || is the norm of a tensor ||-'I|, given by the square root of the sum of squares of all its elements. A lower NND indicates better accuracy, although NND = 1 is still poor: equivalent to all elements of Tp being O. CE10% is the percentage of tensor indices of the top 10% elements in T by lowest validation loss, in common with the top 10% of Typ This metric provides an indication of whether tensor completion can identify the best hyperparameter combinations. Note that the number of elements in Tp is the same in tables] and 2
we found it convenient to display in table
Metric
SVM-P-iris
KNN-R-diab
RF-wine
Elements in Tp
111,600
20,000
4,500
Elements sampled
92
200
27
NND
0.09
0.09
0.27
CE10%
7.5%
14.6%
2%
Table
Tensor completion results using assumed Tucker rank of [1, 1...
Metric
SVM-P-iris
KNN-R-diab
RF-wine
Tucker rank
[2,2,2,2]
[2,2.4]
[2,2,2
Elements sampled
190
596
65
NND range
[0.97, 1.37]
[0.14, 1.00]
[0.92, 1.55] 1.16
NND mean
1.07
0.47
CE10% range
[0, 46.21% 19.9%
[0. 85.1
[0, 8]%
CE10% mean
68%
1.8%
Table Tensor completion results using best assumed Tucker rank * [1, 1...1.
4

---

Under review as a conference paper at ICLR 2023
Due to the inherent randommess of the Cross heuristic, the set of elements of Tp sampled when the Tucker rank is not [1, 1...1] varies over trials hence the range and mean of each metric over 10 trials is provided in Table 2 As seen in table I tensor completion with the Cross technique consistently provides a degree of accuracy in approximation (NND < 1) when the Tucker rank is [1, 1...1. This is in spite of the fact that across all problems, the proportion of tensor elements being sampled is always < 1%. However, the CE10% value is always < 15%, indicating that, while some of the best validation loss elements are identified, many are not. When higher rank values are used as in Table the NND worsens to the extent that NND ~ 1 but the CE10% performance improves. These results suggest that Tucker rank of for each dimension is able to capture general trends in the variation of validation loss values throughout the tensor this explains why it can consistently produce low
NND values. This can only be true if the validation loss values can be roughly described by a low- rank structure of rank [1, 1...1.. However, this Tucker rank is less effective at capturing specific local variations in the values, which is why its CE10% performance is lower. On the other hand, using a higher Tucker rank may at times be able to capture local variations accurately hence the higher CE10% - tends to overfit these variations i.e., emphasise the variations over the underlying low rank structure. This results in more variable performance and poorer NND. For higher Tucker ranks to be able to capture the (approximate) rank [1, 1...] structure underlying the validation loss values, a larger number of samples of T, would be required.
For hyperparameter optimisation, we decided that while in some cases, a higher rank may prove more effective at identifying the best hyperparameter combinations, using a Tucker rank of [I, L...]] is the best default choice as it can capture the structure of different Tp, while ensuring the best time performance by evaluating the lowest number of hyperparameter combinations.
5 TENSOR COMPLETION FOR HYPERPARAMETER OPTIMISATION
In this section, we present a technique for hyperparameter optimisation based on tensor construction and completion as performed in Section Throughout this paper, we refer to this technique as "Hyperparameter optimisation through tensor completion". abbreviated HOTC. From the results of Section 4 it is apparent that low-rank tensor completion can approximate the global distribution of validation losses over hyperparameter combinations, but may miss local variations. Accounting for these variations is, however, crucial to obtaining the optimal hyperparameter combination. There- fore, we decided a suitable approach would be to use tensor completion to predict the general region of the search space most likely to hold the optimal combination(s), and then focus the optimisation on this region for another round of tensor completion. This focusing of the optimisation can be repeatedly applied until the region being searched is small enough for an exhaustive i.e., grid search.
5.1 THE PROPOSED ALGORITHM
Algorithm describes HOTC. Note that the search space for each hyperparameter H is as defined in part 4.1 In each tensor completion cycle, the function cenerate_tenson wtw.rateeeeesssrrrssooooooeooeee coopoennssss first evaluates different hyperparameter combinations i.c., samples the true validation loss ten- according to the Cross sampling scheme. It returns the sampled elements in the form of Cross measurements B,J and A which are respectively the body tensor, array of joint matrici- sations and array of arm matricisations see Zhang (019) for more information. The function generate_complett somplete_tenso then applies the noisy tensor completion algorithm from Zhang (2019) to estimate the complete tensor. In this, B B,J,A A are used to generate the core and factor matrices of a Tucker decomposition of the rank T estimate of the true tensor T. An overall illustration of how true tensor samples can form a Tucker decomposition of the estimate is illustrated in figure [
The hyperparameter combination h with the lowest validation loss T is then found by the func- tion find best combination, which converts the tensor index to its corresponding hyperparameter combination. The function narrow_scan wwwssscrrrsppoee search spaces centred around the values in the hyperparameter combination, h. The search space for each hyperparameter is narrowed around its corresponding value in h; assume this value to be h..... for a uniform integerireal hyperparameter Huni- Let the start, resolution interval, and end (see part 4.1 for definitions) of the original search space before narrowing for Hans be 7 and respectively.
then generates new version of S with smaller
The range of values searched in this space is G = [ec s)/r| x r; the range of the narrowed space G:41 would be at most [G./2r] x r + 1 with hir... at the centre of the new sequence of
5

---

Under review as a conference paper at ICLR 2023
Algorithm
The HOTC Algorithm
Input
C
Number of tensor completion cycles
M
Maximum number of hyperparameter combinations for grid search
Tmin
Minimum search space resolution interval for real-valued hyperparameters
T
Tucker rank assumed for the completed tensor Function to generate validation loss for a hyperparameter combination Map of each hyperparameter to its search space
h + Φ
for cycle_num in 1,2...C do
B, J.J.A + generale.tenson www.ss_coooppeeccc Sompssoommmmeenss.SS..1..
h t find best combination(T,55 (B.J.A)
number of elements in T S M then
h t- grid grid_ssercc((f.)
Exit loop
end if
cycle.num < C then
+ narron.saareh.spccess. h, Fminn)
end end for Output
Selected hyperparameter combination
Incomplete Tensor
Tensor Decomposition
Completed Tensor
Antottenn (Rebl) tool Sgpoode)
Actration: (Rall, toekk slgmald)
lBeli tonk, sigmold)
Autratton::
Lagene [1.A3%44]
Layers [1,2344
R
Lambda:
[U.S..N. 1.5.
Rg
Lambda:
Lamida:
1.5.22255
[05, 25,22
Figure 1: Illustration of a tensor completion approach, based on Tucker decomposition, for hyper- parameter optimisation. Missing values the incomplete tensor are designated grey.
values. The new start and end of the search space hence become max (hH... [(G./4r)] x r, s)
and min (hr... + [(G./Ar) x., e) respectively. The new resolution interval max([r/22) Tmmnn when Huni is uniform real, or max((r [r/2], 1) for uniform integer hyperparameters.
Some categorical hyperparameters may take numerical category values, where closer numbers in- dicate a closer relationship. To narrow the search space of such a hyperparameter, Heats the list of numerical values representing the search space is first sorted in an ascending order. The nar- rowed search space then the sequence of values from the th to the 5th element (both inclu- sive) of the sorted list, where round_fo wowll/oooottee___e T., /4), T.). Here, the length of the original list is T.; and hH... is the zth el- ement in the sorted list. The size L of the new search space list is at most [L/2]] + 1, with h roughly in the middle. The search spaces of categorical hyperparameters taking unrelated non-numerical values ReLU", 'tanh', 'sigmoid"), J, are not narrowed down, and left they
max (i
1) and b min (i +
Once the search spaces have been sufficiently narrowed that the tensor estimate has < M elc- ments, grid search performs exhaustive grid search by evaluating every hyperparameter com-

---

Under review as a conference paper at ICLR 2023
bination in S to find the lowest validation loss. Note that M can be set to 0, in which case the predictions would be purely from tensor completion eycles without any grid search.
5.2 HEURISTICS TO ENHANCE PERFORMANCE
important to mention heuristics to obtain the best performance from Algorithm [ which has numerous configuration parameters. These heuristics were followed to generate the results in Sec- tion 6 Firstly, it is advisable to define large resolution intervals for the initial search spaces in S. For example, if uniform integer hyperparameter has start and end of and 101 respectively, good choice for the resolution interval would be 10 or 20, so that 10 or 5 values are being searched the completion cycle. As discussed in part 4.3 Tucker rank of 1 for every dimension is the best default choice. With these settings, the algorithm should narrow the search space, with each tensor completion cycle, on areas likely to contain the optimal hyperparameter combinations. The resolution intervals would automatically decrease from their large values to Tmin-
Once the initial search spaces are defined, the next step to run the algorithm for one completion cycle, keeping the grid search limit M as zero (i.e., disabling grid search) and to observe the perfor- mance. The number of cycles, C, can be gradually increased until no improvement in performance is obtained on running the algorithm, or the maximum amount of time acceptable for optimisation is reached. At this stage, performance may be further improved by setting M > O, enabling a grid search- at the cost of more execution time, this ensures that the best possible validation loss in the area of the search space being examined has been obtained.
It is possible to predict how much time the algorithm will take, if one knows the time required to evaluate one hyperparameter combination; let this be L. Consider a tensor T of dimensions I I2...N.. To estimate T using the Cross technique, with assumed Tucker rank (ri, [ri, (T.,.N..... is necessary to sample samp ri X T2...'' + r1 x (11 r1) + x (I2 T2)...IN x (IN TN) elements of T under the Cross scheme. In the context of algorithm [ for the first completion cycle, In is the number of values in the initial search space for the n hyperparameter, Estimating the completed tensor based on the sampled elements involves N n-mode tensor multiplications that take negligible time compared evaluating samp hyperparameter combinations. Thus, the first completion cycle takes roughly somp x t in time. subsequent completion cycles, the tensor to be completed has smaller dimensions due to narrowing of the search space thus sump x is an upper bound for the time taken by these cycles. The maximum time taken by the completion eycles is thus C X samp x t. If a grid search is performed at the end, the maximum time is Ro (C x samp) x L + M.
EVALUATION OF HOTC AGAINST ALTERNATIVE HYPERPARAMETER OPTIMISATION TECHNIQUES
The HOTC technique was compared with six existing hyperparameter optimisation techniques: ran- dom search (RS) (Bergstra & Bengio (2012); Bayesian optimisation with Gaussian process (BO- GP) (Frazier 2018)); Bayesian optimisation with Tree-Parzen estimator (BO-TPE) (Feurer& & Hut- ter 2019)): covariance matrix adaptation evolution strategy (CMA-ES) (Hansen 2016)): hyperband (HB) (Li ct al. 2018), and Bayesian optimisation hyperband (BOHB) (Falkner et al. (2018).
The techniques were compared across five benchmark machine learning problems: K-nearest neigh- bours (K-NN) binary classification on the wine data set (abbreviated KNN-C-Wine); K-NN re- gression on the California Housing data set (KNN-R-Calb); random forest binary classification on the Forest Covertype data set (RF-FC); binary classification with 3-layer convolutional neural network (CNN) on the MNIST data set (3I.C-MNIST), and transfer-learning the VGG16 CNN architecture in multi-class classification of the CIFARIO data set (VGG-CIF10)
For each hyperparameter in each problem, the limits (maximum and minimum) of the range of values to search were kept identical for each hyperparameter optimisation technique. In KNN-C-Wine, K-N-R-Calh and RF-FC, each hyperparameter combination was evaluated using cross-vas vallidaalo loss over 5 fold pre-defined for each data set. This ensures consistent loss value is obtained over multiple trials. For 3LC-MNIST and VGG-CIF10, separate training and validation data sets, in size ratio 80 20, were used instead, enable faster calculation of validation loss.

---

Under review as a conference paper at ICLR 2023
6.1 IMPLEMENTATION DETAILS
In KNN-C-Wine, 3 hyperparameters were optimised: the number of neighbours K of the K-NN al- gorithm, the order of Minkowski norm (used to calculate distances between points) and the weight- ing heuristic (uniform or distance-based) for values of the K neighbours when making a prediction at any data point. The wine data set of size 178 was modified for binary classification to only tain samples from classes O and 1. The resulting set was of size 130. The fraction of misclassified validation data samples i.e., misclassification loss was the metric used represent the validation loss. In KNN-R-Calh Raahh, 2 hyperparameters were optimised: the number of neighbours K and the order of Minkowski norm. Only the uniform wighting heuristic could be used here, as the Califor- nia Housing data set had coinciding elements that resulted in infinite distance- logcosh metric was used to calculate validation loss. In RF-FC, 5 hyperparameters were optimised: the number of decision trees in the random forest; maximum depth of any tree; minimum number of data samples needed to split a tree node; number of data features to consider when splitting a tree node, and boolean hyperparameter determining whether subset or the entire training data is used to construct each tree. The Forest Covertype data set, of initial size 581, 012 was too large to be processed in RAM. Hence, it was modified for binary classification by retaining the first 10, 000 samples of classes 1 and each. This gave an "evenly balanced" data set of size 20, 000. The truncation also made the data set faster to train on. In 3LC--MNIST, the CNN had 2 convolutional layers and 1 fully-connected output layer. 11 hyperparameters were optimised, the most among all the benchmark problems. These were: the numbers of neurons, activation functions, dimensions of max pooling and dimensions of stride of the first and second convolutional layers, as well as the neural network optimiser, learning rate and learning rate decay rate. To evaluate each hyperparame- ter combination, the CNN was trained for 5 epochs. The transfer learning problem in VGG-CIF10 involved freezing its 16 convolutional layers and training its fully connected output layers. These layers were of size 4096, 4096 and 10 (to classify 10 classes) resulting in 18, 923, 530 trainable parameters. 7 hyperparameters were optimised: the optimiser, learning rate, learning rate decay, number of epochs to train the network, batch size and dropout probabilities of the fully connected layers of size 4096. It should be noted that the 32 x 32 CIFARIO were resized to 224 x 224 to be accepted by the VGG16 network.
weights. The
The search spaces for each of these hyperparameters are described in Appendix A.2 The computing hardware, software implementations of the different hyperparameter optimisation techniques, and the configurations of these techniques are described in Appendix B
6.2 NUMERICAL. RESULTS
The code for HOTC and our benchmarks is publicly available on GitHub. Figure 2 illustrates the results of the experiments. The graphs were generated for RS, BO-TPE, CMA-ES, BO-GP, HB and BOHB by sampling the best obtained validation loss at the timestamp of every trial. For HOTC, this sampling done after each completion cycle and after the final grid search, if any. For KNN-C- Wine, KNN-R-Calh and RF-FC, the HOTC approach is remarkably faster than the other techniques and also obtains the lowest value of validation loss. It is not possible to HOTC for longer than is seen on these graphs, as the completion cycles have already converged to very small region in the tensor that is trivial to exhaustively search. Note that in the diagram for KNN-C-Wine, the graphs of BO-GP, CMA-ES, RS and BOHB overlap each other, as do those of BO-TPE and HB.
In the diagrams for 3LC-MNIST and VGG-CIF10, the ends of the completion cycles for HOTC have been indicated by numbered markers. Grid search was not enabled in either problem, as it would consume too much time. In these problems with more hyperparameters, each combination of which takes longer to evaluate (neural networks take longer to train), HOTC does not outperform the other techniques. One has to wait 1500 seconds for the first suggestion of hyperparameter combination from HOTC in 3LC-MNIST and z 2000 seconds in VGG-CIF10 this is because a minimum number of tensor elements must be known in order for tensor completion to be possi- ble. However, the effectiveness of the tensor completion can be seen as the suggested combination improves over subsequent completion cycles in both cases, the final validation loss is compara- ble to the best results observed. Overall, while HOTC is not outperforming its competitors in all situations, it is still able to consistently find optimal or npar-optimal combinations across machine learning paradigms.
8

---

Under review as a conference paper at ICLR 2023
KNW Classification an Wine Data Set
KNN Regression on California Housing Data Set
65
50-
(MAES
CMA-ES
Vallat en Los Mocdsssft.Mnnnnn
80-5P
0.92
50-GP
HD2C
00115
VM (e40
0.00
HDTC
D0H10
0.68
0:30
0.65
8.00
0.64
0.82
25
100
125
150
175
200
200
400
600
800
1020
Opmisstion time (veconds)
Opim salion time (seconds)
Random Forest Forest
3-layer CNN
0.200
0.375
Va (S 100000000
0.75
0.150
0.325
0.325
0.309
0.30
0.059
0250
0.809
800
1020
2500
Opmixation time tuecondel
Leaming
1,40
Variatonn Looe (Culggrrcl Coae Goasertass
50-1P
1.30
1,20
1.10
1,00
Figure 2: Graphs of validation loss against optimisation time for the different hyperparameter opti- misation algorithms on the benchmark machine learning problems.
7 CONCLUSION
We have introduced the concept of tensor completion in the hyperparameter optimisation paradigm. Through sequential completion cycles that are able to identify the most promising subspaces, the proposed method has been shown be highly competitive and often superior wide range of state-of-the-ate and commonly used frameworks over diverse set of benchmarks and algorithms. our hope that the presented intuitive yet efficient approach to hyperparameter optimisation will spur the interest of the machine learning community, and we envision tensor completion becoming a strong alternative to current approaches. Regarding future work, we aim to work on a procedure to automatically determine optimal values for the algorithm inputs, such as resolution of hyperpa- rameter ranges or number of tensor completion cycles. Furthermore, we plan to investigate whether combining the current approach of focusing optimal regions with random exploration of the space will improve the overall performance. Finally, an interesting direction we aim to pursue the investigation of alternative tensorisation methods of the hyperparameter space.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Evrim Acar, Daniel M. Dunlavy, Tamara G. Kolda, and Morten Mørup. Scalable tensor fac- torizations for incomplete data. Chemometrics and Intelligent Laboratory Systems, 106(1):
41-56, Mar 2011. ISSN 0169-7439. doi: 10. ch0ll//cccmollaa2200000000 7/dx.doi.org 1016/ whw.://lll...cmmlbbbbbooooo www.chwwwww...c..oo.ooooooopaaee 004
URL http:
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework, arxiv 1907 1et1199779.0,,,,, 2019.
AutoML. Hpbandster distributed hyperband implementation steroids, 2019. URL https:
//github. com/ aut om1/ HHtp///BBaadttrrrre
Johann A. Bengua, Ho N. Phien, Hoang Duong Tuan, and Minh N. Do. Efficient tensor completion for color image and video recovery: Low-rank tensor train. IEEE Transactions on Image Pro-
cessing, 26(5):2466 (44,))2466-277, May 2017. ISSN 1941-0042, doi: 10.11099ip.2011::2774599 URL
http:// doi org/ 10. 1109/TIF
0117.2672333
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(10))281-305, 2012. URL
v13/bergstral22 htmml]
Andrzej Cichocki, Danilo Mandic, Lieven De Lathauwer, Guoxu Zhou, Qibin Zhao, Cesar Caiafa, and Huy Anh Phan. Tensor decompositions for signal processing applications: From two-way to multiway component analysis. IEEE Signal Processing Magazine, 32(2): 145-161,, 2015. doi: 10. N0.1100/MSSP.2013.2222279
Liping Deng and Mingqing Xiao. A new automatic hyperparameter recommendation approach un- der low-rank tensor completion framework. JEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-13, 2022. doi: 10. (0.11097PAAMM.2222211955
Bradley Efron, Trevor Hastie, Iain Johnstone, and Robert Tibshirani. Least angle regression. The Annals of Statistics, 32(2), Apr 2004. doi: 10. 0001214000055334400000007 URL https::/ doi www.org/110 11w.000.22222200003666000000
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter opti- mization at scale. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp.
1437-1446. PMLR, 10-15 Jul 2018. URL https::/prrceeddingg ml www.asssssssseesse.emmpmmm.m www.sreessvvve vBC/ falkner18a html
Matthias Feurer and Frank Hutter. Hyperparameter Optimization, pp. 3-33. Springer International Publishing, Cham, 2019. ISBN 978-3-030-05318-5 doi: 30079978--0000053188 URL
https://don
wwt.:oo.ogg...... httt::::/doi.or//10. 100. 3-300-5---00--81-----
R. A. Fisher. The use of multiple measurements in taxonomic problems. Annals of
Eugenics, 7(2)8179-188, 1936. doi:
URL
1936.tb02133
Peter I. Frazier. tutorial on bayesian optimization, Mixiv1807.028.11 2018.
Nikolaus Hansen. The ema evolution strategy: tutorial, Mxxiv.104.000772 2016. URL https:
7/arxiv. wtwpxi..or//bb//500000 /ttp//obbb/550000022
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband:
A novel bandit-based approach to hyperparameter optimization, arxiv16003.06560, 2018.
Yuanyuan Liu, Fanhua Shang, Hong Cheng, James Cheng, and Hanghang Tong. Factor Matrix
Trace Norm Minimization for Low-Rank Tensor Completion, pp. 876-874. 04 2014. ISBN 978- 1-61197-344-0. doi: 13.1137/1.971111 1973440.99.
Fernando Nogueira. Bayesian Optimization: Open source constrained global optimization tool for Python, 2014. URL UR. htt.:htts://gggg http://tttwt..tcooeeoooo/////aae anOptimm i
10

---

Under review as a conference paper at ICLR 2023
Qingquan Song, Hancheng Ge, James Caverlee, and Xia Hu. Tensor completion algorithms in big data analytics, 2018.
Tong Yu and Hong Zhu. optiiiiaaantterrooiimiiiiiooooi 2020. DtA.Aa/.e.///aaa.....////////.00
optimization: A review of algorithms and applications,
Anru Zhang. Cross: Efficient low-rank tensor completion The Anmals of Statistics, 47(2):936 964, 2019. doi: 10.1214/18AASS6622 (tt.atppee//aa..rrr//002222//2/3
A HYPERPARAMETER SEARCH SPACES
A.1 TENSOR COMPLETION EXPERIMENTS
In these experiments, the hyperparameter search spaces are set order to generate complete tensor of validation loss values. In the tables, they are presented in the for- mat (start, resolution_intervad. end) for uniform integer or real hyperparameters, or [iteml, item2, ...] for categorical hyperparameters.
SVM-P-iris
Refer to table 3
Hyperparameter
Search Space
Regularisation Hyperparameter
(0.1, 0.1,
Polynomial Kernel Degree
(0.0, 0.1, 3,0)
Polynomial Kernel Scaling Factor
(0.1, 0.1, 3.0)
Polynomial Kernel Constant Term
(0.0, 0.1, 3.0)
Table 3: Search spaces for SVM-P-iris.
KNN-R-diab
Refer to table 4
Hyperparameter
Search Space
No. of Neighbours Minkowski Norm Order
(1, 100)
(1, 1, 100) ['uniform', 'distance")
Weighting Heuristic
Table 4: Search spaces for KNN-R-diab.
RF-wine
Refer to table
Hyperparameter
Search Space
No. of Decision Trees
[1, 10, 20, 30, 40] [1, 10, 15, 20] (2,1,10) (1,1,10) [True, False]
Maximum Tree Depth
Minimum Samples to Split Node
Data Features to Split Node
Subset/Entire Training Data to Construct Tree
Table 5: Search spaces for RF-wine.
A.2 HYPERPARAMETER OPTIMISATION EXPERIMENTS
In these experiments, the limits of the search spaces for each hyperparameter were kept common for all the tensor completion techniques. The search space resolution interval is only defined for HOTC that it construct tensor representing the search spaces. For all other techniques, integer hyperparameters always have resolution and the search spaces for real hyperparameters continuous. The advantage of this is that it ensures these techniques have access to the entire set

---

Under review as a conference paper at ICLR 2023
of possible values, although the downside is that the set of combinations that can be tested becomes larger, especially with real hyperparameters.
In the tables, search spaces presented either in the format of (start, end) for real or integer hyperparameters, or [iteml, item? ...] for categorical hyperparameters. Fuethermore, if the log column value is "True", this means the hyperparameter value varies between start and end expo- nentially rather than uniformly. Note that while it was stated that HOTC only supports uniformly distributed hyperparameter values, is easy to accomodate exponentially varying values by consid- ering the exponent as uniformly varying hyperparameter over a constant base.
KNN-C-Wine
Refer to table
Hyperparameter
Search Space
HOTC Inital
Log
Resolution Interval
No. of Neighbours Minkowski Norm Order
(1, 100)
10
False
(1, 100)
10
False
Weighting Heuristic
('uniform', ddsstance']
N/A
N/A
Table 6: Search spaces for KNN-C-Wine.
KNN-R-Calh
Refer to table 7
Hyperparameter
Search Space
HOTC Inital Resolution Interval
Log
No. of Neighbours Minkowski Norm Order
(1, 100)
10
False
(1, 100)
10
False
Table Search spaces for KNN-R-Calh.
RF-FC
Refer to table
Hyperparameter
Search Space
HOTC Inital
Log
Resolution Interval
No. of Decision Trees
(1, 40)
False
Maximum Tree Depth
(1, 20)
5
False
Minimum Samples to Split Node Data Features to Split Node
(2, 11)
2
False
(1, 11)
False
Subset/Enttre Training Data to Construct Tree
[True, False]
N/A
N/A
Table 8: Search spaces for RF-FC.
3LC-MNIST
Refer table 9
12

---

Under review as a conference paper at ICLR 2023
Hyperparameter
Search Space
HOTC Inital
Log
Resolution Interval
No. of Layer Neurons
(5, 40)
False
No. of layer 2 Neurons
(5, 40)
5
False
Layer Activation Function
['relu', 'tanh', "sigmoid']
N/A
N/A
Layer Activation Function
['relu', 'tanh', "sigmoid")
N/A
N/A
Layer Pooling Dimension
(1, 5)
False
Layer 2 Pooling Dimension
(I, 5)
False
Layer Stride Dimension
(1,
False
Layer 2 Pooling Dimension
(1, ['RMSProp's 'adam', 'sgd']
False
Optimiser
N/A
N/A
Learning Rate
(10 10 2)
True
Learning Rate Decay
(10 8,-8.00 10
1
True
Table 9: Search spaces for 3LC-MNIST.
VGG-CIF10
Refer to table 10 Note that dropout probability is the probability of the input to neuron being turned off, not turned on.
Hyperparameter
Search Space
HOTC Inital
Log
Resolution Interval
Optimiser
('RMSProp', 'adam', 'sgd']
N/A
N/A
Learning Rate
(10 10 10-2)
True
Learning Rate Decay
(10-8 8, 10-4)
True
Epochs Batch Size
(1, (2 28)
False
True
Dropout Probability of First
(0.0, 0.9)
0.15
False
Fully-Connected Layer
Dropout Probability of Second
(0.0, 0.9)
0.15
False
Fully-Comnected Layer
Table 10: Search spaces for VGG-CIF10.
B SOFTWARE AND HARDWARE DETAILS
All experiments were evaluated on Google Colab: the Google Compute Engine backend consisted of two single-core Intel Xeon CPUs of clock frequency 2.2 GHz, each accommodating two threads. The total RAM available was 12.68 GB. For experiments with 3LC-MNIST and VGG-CI110 the environment was accelerated with an NVIDIA Tesla T4 GPU. While HOTC was implemented in Python by us. open-source Python libraries used for the other techniques: Optuna (Akiba al. 2019 version 2.10 for RS, BO-TPE, CMA-ES and HB; Bayesianoprimissaton (Nogueira (2014)) version 1.2 for BO-GP. and hpbandster (AutoML (2199) version 0.7 for BOHB.
The configuration parameters of HOTC were set for each problem according to the heuristics de- scribed in part 5.2] and are listed in table [T] For RS, BO-GP, BO-TPE and CMA-ES only the number of trials were varied according to the time for which optimisation was to be performed. The time of optimisation for each problem can be seen the graph of figure A For multi-fidelin techniques: HB and BOHB, the number of trials, the minimum and maximum budget values, and step size with which the minimum budget was increased the maximum were set to reasonable values based on the data achieve good performance these values are described in table 12 Apart from this, all other configurations were left with the default values from their respective library implementations.
For HB and BOHB, the multi-fidelity budget was the fraction of the training data used to train the machine learning model in all problems except KNN-C-Wine. Since the size of the modified wine dataset, 130, was small compared to the maximum searched value of i.c., 100, the budget was defined as the fraction of the total number of data features (which was 13) used in training.
13

---

Under review as a conference paper at ICLR 2023
Problem
Assumed Tucker
Maximum Number of Completion
Minimum Number of Elements Grid Search
Minimum Resolution for
Rank
Cycles
Hyperparameters
KNN-C-Wine
[1,1,1]
5
51
N/A
KNN-R-Calh
[1,1]
51
N/A
RF-FC
[1,1,1,1,1]
4
51
N/A
3LC-MNIST
[1] x11
0
0.25
VGG-CIF10
[1] x7
2
0
0.05
Table 11: Configuration parameters of HOTC in different machine learning problems.
Problem
Minimum Budget
Maximum Budget
Step Size
KNN-C- -inn
2/13
13/13
1/T3
KNN-R-Calh
0.2
1.0
0.2
RF-FC
0.2
1.0
0.2
3LC-MNIST
0.2
1.0
0.2
VGG-CIF10
0.2
1.0
0.2
Table 12: Multi-fidelity budget limits used for HB and BOHB in the different machine learning problems.
14


Reviews:
[{"review_id": "ouF9RHjaSR", "reviewer": "Reviewer_1YoG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper applied tensor completion to search for the optimal hyperparameters for machine learning methods. Precisely, the combinations of parameter candidates are mapped on the tensor values with multiple modes. By the efficient sampling trick (Zhang, 2019) followed by tensor completion, the optimal parameters are selected from the minimum (or maximum) values of the completed tensor.", "summary_of_the_review": "I agree that tensor completion methods might be useful for quickly predicting the optimal combinations of hyperparameters. But as mentioned, the usefulness is held if the low-rank assumption exists in practice in the incompleted tensor. However, this point is not clearly discussed and proved from either theoretical or empirical side.", "strength_and_weaknesses": "**Strength:**\n\n1. As a new application of tensor completion, this work may be the first to apply tensor completion to NAS (as claimed in the paper), which might inspire more studies on this problem in the tensor community.\n2. Some tricks, such as the mappings from hyperparameters to tensor values and the ones for performance improvement, is interesting and might be applied to broader ranges of tensor applications.\n\n**Weakness**\n\n1. Several crucial problems are not convincingly discussed in the paper.\n2. The experimental results cannot sufficiently support that the proposed completion method outperforms SOTAs in practice since only several simple architectures are used for evaluation in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oS1Crqlxpc", "reviewer": "Reviewer_PJTn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a new hyper parameter optimization method using tensor completion algorithm. The new algorithm consider a N dimensional tensor with its dimension representing tunable parameters and values representing feedbacks (e.g. validation loss). A Tucker factorization is used to for tensor completion in this paper.\n\nEvaluation with machine learning algorithm, including SVM, KNN, Random Forest, Neural Network shows a better selected parameters for optimize the validation losses in the given execution time, as compared to other Bayesian Optimization approach. ", "summary_of_the_review": "Overall, I feel the idea of using tensor completion to improve hyper-parameter tuning is an interesting and innovative idea. \n\nHowever, due the concerns from \n[1] The motivation of the paper seems weak and little evidence has been provided to prove the low rank property holds, let along why Tucker rank is selected in the paper.\n[2] There are some concerns on the main algorithm of the paper, particular from sampling patter and sampling ratio to ensure tensor completion converges. Some minor concerns are from the rank selection as well.\n[3] In sufficient numerical results to demonstrate the advantage of the new algorithms. \n\nThus, I feel the paper can be improved. ", "strength_and_weaknesses": "Strength\n[+] Using tensor completion to improve hyper parameter optimization is a new and innovative idea.\n\n\nWeakness\n[-] The motivation of assuming the optimized metrics satisfies the low rank properties seems weak.\nAlthough heuristically sound reasonable. Could the author help to provide more evidence on reasoning of the low rank assumption? Although in the introduction, the paper mentions close parameter provides similar performance, it does not seem to motivate the low rank assumption. Is it possible to have search space with all evaluated loss, and then using tucker factorization to quantify if the low rank assumption holds?\n\n[-]  Not all tensor completion algorithm can converge, which in general depends on the sampling pattern and sampling ratio from the tensor. In this paper, the sampling pattern is directly related to S (set of search spaces) in algorithm 1. Could the author help to clarify how to select  the search spaces to ensure the success of tensor completion algorithm here?\n\n[-] The numerical results seem weak. With only the final searching results, it is difficult to quantify the performance of the algorithm. Could the author provide the convergence of plot of the parameter optimization process to demonstrate either fast or better convergence?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Yi90VQ0fVUv", "reviewer": "Reviewer_BwZ7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new way to perform hyper-parameter search based on tensor completion. The combinatorial space of hyper-parameters is formulated as a multi-way tensor, where the loss value of a performed trial is registered as tensor entries. By decomposing and reconstructing the highly sparse tensor, one could predict the loss value of unseen hyper-parameter settings. On a set of experiments, the proposed method has shown comparable results with SOTA algorithms. ", "summary_of_the_review": "In general this is a very interesting idea but the presentation could have been better, especially with respect to points 2 and 3 under weakness. I would like to see the work published but after a revision. It would be much more valuable if the community could have a sound and thorough analysis of this idea, even if it turns out to underperform some SOTAs, than to have some discrete numbers that are supposed to demonstrate the superiority of the proposed approach. ", "strength_and_weaknesses": "This idea is quite intriguing and I’ve been expecting some work in this direction for some time. \nMy major concerns are the following: \n\nWeaknesses: \n1. The step of narrowing the search space may result in finding a local optimum. In other words, the very first search is just crucial for the final result.\n2. As the first work of using tensor completion for hyper-parameter search, I think it is important to perform a more thorough study on the quality of the completion. For instance, one could simulate a full tensor as ground truth and show how well the proposed algorithm can find the global minimum. \n3. The presentation of the results: the results could have been more convincing if the authors could provide results with more than 2 choices of execution time. For instance, consider visualizing the loss values in y-axis against execution time in x-axis. \n4. I would also be interested in the memory consumption of the proposed approach in comparison with SOTA. \n5. Comparing different tensor decomposition methods could also be an interesting contribution. \n6. How does this work compare to Deng and Xiao: \"A New Automatic Hyperparameter Recommendation Approach Under Low-Rank Tensor Completion Framework\"? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Cxyb7A5zjn-", "reviewer": "Reviewer_C2WJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a HPO method using tensor decomposition. It uses tensor decomposition to predict hyperparameter performance, which is a very novel idea. The best hyperparameter after tensor completion is recommended as the next hyperparameters to try and the algorithm continues until a predefined budget exhausts. Experiments compared to common HPO methods on tuning 4 algorithms showed some potential of the methods.", "summary_of_the_review": "The paper is not matching the ICLR standard due to the weakness that I have mentioned above.", "strength_and_weaknesses": "On the strength, the work proposed a very novel route for HPO through tensor completion and I found it very interesting.\n\n\nOn the weakness, I have major concerns on the technical quality and clarity, as well as in the experiments set up. I think many questions should be answered to show tensor completion is a promising route for HPO. The questions follows:\n\nOn the technical quality.\n* Why is the low rank assumption true? Any motivating examples? For example, on some small scale problem, evaluate all the HP configurations and show that the low rank approximation has good accuracy.\n* How good is the predicted accuracy based on tensor decomposition, giving more and more hyperparameter evaluations? Tensor completion with only a few entries could not end up with a good prediction. How does it compare to GP that is used in BO? This is not studied but critical for a convincing story in HPO.\n* The current algorithm only does exploitation but no exploration. This is surprising and why did the author design it in this way?\n\nOn the experiment set up.\n* In Section 5.2, the author wrote “For HOTC, T was achieved by varying the number of\ntensor completion cycles, resolution of the initial search space, maximum number of elements for grid-search and minimum resolution for real-valued hyperparameters.” This is problematic because the search spaces (min, max, resolution) should be exactly the same for all the methods.\n* The narrowing down search space part to me seems a bit arbitrary. Do you have experiment results without it? I would highly recommend taking this component out and comparing the proposed algorithm  to other methods.\n* It can be hard to know the low rank number “T” in advance. Did you try some other values in the experiments and some tip on how to set T?\n\nOn the clarity:\n* If the tensor decomposition used in the paper is “noisy tensor completion with Cross measure-\nments (Algorithm 2, Zhang (2019))”, then I would expect the authors to explain at least the core ideas.\n* Is it a hard requirement for the uniformness of the integer and real hyperparameters (Section 4.1)?\n* After reading Section 4.3, I understand why the grid search is used in Algorithm 1. It is because when the search space is small enough, one could afford a grid search and not miss any important HPs. But it will be very helpful to explain that while describing the algorithm, or at least mention that it will be explained later on.\n* The language needs to be improved for correctness and smoothness.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject