Probable Dataset Searching Method with Uncertain Dataset Information in Adjusting Architecture Hyper Parameter
Authors: Chen Yang
Jingyuan Wang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Different types of tasks with uncertain dataset information are studied because different parts of data may have different difficulties to achieve. For example, in unsupervised learning and domain adaptation, datasets are provided without label information because of the cost of human annotation.  In deep learning, adjusting architecture hyper parameters is important for the model performance and is also time consuming, so we try to adjust hyper parameters in two types of uncertain dataset information:1, dataset labels are postponed to be obtained so hyper parameters need to be adjusted without complete dataset information. 2, hyper parameters are adjusted with a subset training dataset since training models with complete training dataset is time consuming. Here, we propose several loss functions to search for probable dataset when the complete dataset information is not obtained. The experiments on 9 real world data demonstrate the performance of our method.

Under review as a conference paper at ICLR 2023
PROBABLE DATASET SEARCHING METHOD WITH UN- CERTAIN DATASET INFORMATION IN ADJUSTING AR- CHITECTURE HYPER PARAMETER
Anonymous authors Paper under double-blind review
ABSTRACT
Different types of tasks with uncertain dataset information are studied because different parts of data may have different difficulties to achieve. For example, in unsupervised learning and domain adaptation, datasets are provided without label information because of the cost of human annotation. In deep learning, adjusting architecture hyper parameters is important for the model performance and is also time consuming, we try to adjust hyper parameters in two types of uncertain dataset information dataset labels are postponed to be obtained so hyper parameters need to be adjusted without complete dataset information. 2, hyper parameters are adjusted with subset training dataset since training models with complete training dataset is time consuming. Here, we propose several loss functions to search for probable dataset when the complete dataset information is not obtained. The experiments on 9 real world data demonstrate the performance of our method.
INTRODUCTION
In deep learning, most regression data can be represented with the form (X, Y), where X is the data input and Y is the label. However, different parts of the data may have different difficulties to achieve. For example, in unsupervised learning [arlow (1989) and domain adaptation Wang & Deng 2018), the label of dataset is assumed hard to be obtained since label usually needs human annotation. These situations could be viewed as making decisions when part of dataset information is uncertain. Another probable situation is that input sample X can be obtained much earlier than label Y is obtained, because human annotation Y is time consuming or the exact task target of Y is not determined when the input samples are collected. In such situation, the computing resources are assumed to be abundant before the label is obtained. In deep learning, architecture hyper parameter setting is an important factor for the performance of model. Then question is whether the architecture hyper parameters, corresponding to different network architectures, can be compared and adjusted only with input sample information X. Selecting hyper parameter only with input sample information X could save the time to try different hyper parameters when the label Y is obtained. Input sample information usually takes more memory space than label information Y. indicating that input sample information X contains more information than label information Y, so predicting the architecture comparison only with input sample information X seems probable.
To deal with the uncertain information in a dataset, we propose probable dataset searching method to predict architecture comparison, where the dataset representation is inspired by the dataset definitions and assumptions in recent neural network convergence works Kohler & Langer 202T); Bauer & Kohler (2019); (Schiidt-Hiebe (2020); [uzuki (2018); Farrell et al. (202T). ). Our method could search probable datasets with provided dataset information such as input sample information X. Concretely, the comparison of two hyper parameters can be predicted by searching for the existence neural
of probable dataset that architecture is better worse than another. Here, network to approximate the dataset regression function and apply several loss functions to search for the probable dataset that trained architecture is better than another in testing dataset.
An assumption in our method is that the compared architectures should have competitive performance on searched dataset. Empirically, the compared architectures selected because they perform well

---

Under review as a conference paper at ICLR 2023
on the data domain due to former experience, so we search the dataset from the situations that at least one compared architecture could perform well in the dataset. In implementation, the neural network to approximate the regression function has the same architecture compared architecture.
Probable dataset searching method could also help analyze the characteristics irrelevant to the concrete dataset information of X. For example, sometimes training models with complete dataset costs too much time so a small subset can be used to approximately adjust architecture hyper parameters Klein al. (017)); Elsken et al. (2019). However, this approximation is correct sometimes and wrong at other times. When comparing two given hyper parameters, probable dataset searching method could figure out the conditions that the approximate comparison is correct.
2 METHOD
2.1 DATASET REPRESENTATION
When a sample in a regression dataset denoted by a random variable vector with the form (X, Y). the relationship between X and Y can be represented as a regression function fu( E X
2). where E is the expectation. In recent deep neural network convergence theorem Kohler Langer (202T): [Bauer& & Kohler (2019); Scheidt-Hieber (2020); [Suzuki (2018); Farrell al. (2021), it is proved that fo could be converged by multi-layer fully connected neural networks trained with enough samples, where fo satisfies some requirements and assumptions. Here, we assume that the convergence theorem is also true with other types of deep neural networks. We then use neural network approximate a regression function with two reasons: as a function, neural network also satisfies the requirements and assumptions of regression function. 2, there is always neural network g that the difference between fo and g is smaller than given positive value because of the definition of convergence.
As result, regression dataset can expressed by
(1) where the neural network g has small enough similarity to fo. is a random disturbance of regression function with 0 expectation. Then, there are three components influencing the regression dataset, the regression function g. X distribution and random disturbance distribution e. e should be much smaller than g in the dataset so the relation between inputs and outputs could be easily distinguished from random disturbance.
Y gX)) +.
There is another additional assumption of the regression dataset: The regression function should not be too complex that compared architecture can perform well in the comparison. It also assumes that the compared architectures in the searching method are selected with empirical knowledge to provide competitive performance in an uncertain dataset with enough training samples. Practically, the assumption is satisfied by setting the regression function g with the same architecture as a compared architecture
Formally, to train a neural network 9AR-A with its parameters ARRA with a training dataset (1,, error value Eir is introduced smaller the better value to evaluate the performance of model in the dataset a, Then training gAR-A with a training dataset on aims to find parameters OAR-A (ai) that
( arg min Err(o AAR-A (:, AR-A))
0RRA
where smaller error value means better model performance in training dataset ai, In regression task, Root Mean Square Error (RMSE) a widely used error value: RMSE = / * l yIß. where || ||2 is the 1-2 norm, y is the true value, á is the model output and K is the dimension of y.
The difference of the error value between two architectures on a testing dataset a2 can be used to compare the performance of two architectures AR-A and AR-B:
C(AR-A, AR-B, a, a2) Err(az. A (-, AAR (a1))) Err(a,, (: (ar))), (3)
where models are trained by training dataset aj and tested by testing dataset ar2-
In this paper, try compare the performance of two architectures when part of dataset information uncertain, which aims to find the probable situations that AR-A (or AR-B) performs better, corresponding to small C(AR-A, AR-B, 0n 02) (or C(AR-B, AR-A, 01, 02,)2) value.
2

---

Under review as a conference paper at ICLR 2023
Soarch Function gAR.D. Distribution X,
Luz Lg(AR-A, (ARR-BB.
Iteration
Train
K
Test
(X1= Vi (X Y2)
(X)
AR-B
Generric
Test
CIAR-A,AR--
Genernte
Y = BAR-D
Data Representation
Iteration
Training
Trained Network
Testing Datuset
Testing Error Comparison
Dataset R
Figure I: Probable dataset searching procedure, where models are trained in aj and tested in 0g. The testing difference of two architectures are used to search probable dataset.
2.2 CALCULATION AND Loss FUNCTION
We start with the illustration of loss functions, which are important to search for dataset parameters in backpropagation. As shown in Fig. I by optimizing loss functions, we could search for the probable dataset that architecture AR-A performs better than AR-B. Then, the comparison score can be used to test the performance of a searched dataset.
In regression task, L2 mean square error loss is a widely used loss function to train deep learning models, where small L2 loss means a small error value. The L2 loss function train an architecture AR-A with dataset can be denoted by:
L2(AR-A, a) = nK Σ A OAR.A) - Y:
(4)
where (X,, Yi) E a. dataset a contains samples, K is the dimension of Y. || 1l2 denotes the 1-2 norm and gAR-A is neural network with architecture AR-A and parameters ARRA
When we deal with the situation that the dataset do not have label Y so the information of regression function is uncertain, we use neural network with architecture AR-D to approximate the regression
function in Eq. equation] I Then, when representing the dataset with the form of Eq. equation [ L2 loss becomes (AR-A, a K Σ A (X,, AR-A) - (gAR-D( (x, AR-D) + <,) 112, where L2 can also be used to train the variables that contain dataset information, including the parameters AR-DD the distribution of X, and c,
To compare the performance, two architectures AR-A and AR-B first need be trained with the same training dataset a with Lg loss:
min Lm = min L2 0n) + min L2(AR-B,OO) 01):
(5)
BAR-A
BAR
where the parameters of AR-A and AR-B reach aARA and ORR with minimum L value.
If we want to find probable dataset where AR-A is better than AR-B, we need to let the comparison score C'AR-A, AR-B, a1, a1,02) a2) small. Since small L2 loss means small error value, searching dataset for small comparison score equals to search AAR-D. X and c with loss
min L R2 AR-A, AR-B) = min (L2(ARAA. o2) - Lz(AR-B, OAR.D.X.c
(6)
where the parameters of AR-A and AR-B are trained by L and not influenced by L As illustrated in the additional assumption, let the dataset regression function architecture AR-D be the same with AR-A in Lmz since we want to search for the dataset that AR-A performs well and a dataset with architecture AR-A should best fit the AR-A model.
3

---

Under review as a conference paper at ICLR 2023
When dataset information needs to be searched, both a1 and ag obey the relation described in Eq. equation so the dataset searched with LR2 loss function also influence 01 in LR1 loss. Although L2 is differentiable with the respect to the parameters related to probable dataset, the L loss only influence the parameters in AR-A and AR-B when training models. Besides, no parameters in AR-A and AR-B are influenced by the Lre loss when searching dataset. The combination of I. and R2 losses means searching the dataset that a trained AR-A model performs better than a trained AR-B model in testing dataset. We discuss more details of optimizing L in Appendix.
X distribution, regression function JAR-D and are three unknown variables in Eq. equation | that can be searched in our method. In this paper, the dataset searching method can be applied different types of dataset information missing. For example, when the dataset only does not have label Y. the X; loss functions uses the known dataset input information and only parameters ARRDD and e needs to be searched in Eq. equation G If the dataset does not have label Y and complete X distribution, X distribution also needs to be searched in Eq. equation [6] Concretely, the parameters ARRD in regression function gAR-D can be trained by backpropagation algorithm since R is differentiable with the respect to ARRDD in Og. If X distribution needs to be searched, we use a set of discrete input samples 3x as approximation to represent the distribution of X, where the elements of 3x obey the distribution of X. The size of set Bx should be a large number (100,000 in our experiment) for fine approximation. Then input samples X; in datasets o 02 can be generated by randomly selecting inputs X from Bx and corresponding Y can be calculated with Eq. equation [ where the size of 01 set to a fixed number to discover the relationship between train size and architecture difference. The distribution of X can be searched by adjusting the input samples X; in set 8x through the backpropagation of input samples X, in dataset ag with L loss, where ag is generated from the set fx repeatedly. Similarly, the distribution of e can be trained by adjusting the value in a finite set as an approximation.
When training the distribution of X in 9x. we need an additional normalization procedure since current calculation environment can not deal with number with large out of range value (for example, single or double precision floating point format both have maximum value). As a result, when searching X distribution, we apply a normalization method, which let each dimension in X has O mean and 1 standard deviation. The normalization equals to a linear operation to each dimension in X. For distribution of c, we normalize e with 0 mean and a small standard deviation since the expectation of should be 0 and we aim study the situation that the regression function has the main influence on the value of Y.
Similarly, the searched output value of regression function should not out of range, so we normalize the searched regression function JAR.D with loss
LRN ||mean(AR-D. (AR-D, a2) 0 - ||std(AR-D,.. a²) - 1
where mean(AR- (AR-D, 022) (or std(AR-D. (22)) means calculating the mean (or standard deviation) vector of each dimension in JAR-D on all input samples X; in dataset 022- TRN aims to normalize
the output of regression function with mean and standard deviation to avoid out of range large number in searching process.
In implementation, we search dataset through loss LR1s L and LaN with several iterations, where in each iteration, the parameters in architectures AR-A and AR-B are randomly initialized and trained. The searched dataset will be tested by the score C(AR-A, AR-B, 01, ar2) for evaluation.
3 EXPERIMENTS
3.1 ADJUSTING HYPER PARAMETER WITH INPUT INFORMATION
Here, we want to evaluate several methods in comparing two architecture hyper parameters without knowing complete data information. These methods aim to save the time of adjusting hyper parameters after obtaining the complete data information. When comparing two hyper parameters denoted by architectures AR-A and AR-B with uncertain dataset information, different comparison predictions lead to different training preferences after complete data information is obtained. There are three types of preferences: 1, only training AR-A with complete data. 2, only training AR-B. 3, training both and selecting the better. Three actually training preferences correspond to different comparison prediction explanations: Corresponding type (or 2) preference, architecture AR-A (or AR-B) is
4

---

Under review as a conference paper at ICLR 2023
always better than similar to another in probable datasets, only one architecture needs to be actually trained. Corresponding to type: preference, architecture AR-A is sometimes better and sometimes worse than AR-B in probable datasets. Type 1 and 2 comparisons adjust and select hyper parameters before the complete data information is obtained while type 3 comparison is the most conservative prediction that adjusts hyper parameters after obtaining the complete data information.
There are two metrics in evaluating the performance of a comparison prediction, correct (or wrong) situation and actually training times. Here, if architecture AR-A actually performs better than another in a complete dataset and AR-A will (or will not) be actually trained in a comparison prediction, the comparison prediction is correct (or wrong). If an architecture AR-A actually performs similar with another, any actually trained preference is acceptable so any comparison prediction is correct. Here, if two architectures do not have statistical significance in a dataset with complete information, we treat them as similar architectures, where we apply a 5% significance level t-test score with 10 experiments for significance test.
When considering two metrics, correct prediction times is larger the better and actually training times smaller the better since we want to train models with smaller time consumption of adjusting hyper parameters after obtaining complete dataset information. We apply an Average Evaluation Score (AES) to consider two metrics together when making multiple comparison predictions:
AES- N
(Correct -2Wrong- 0.5Training),
(8)
where N comparison predictions number. Correct (or Wrong) refers to the correct (or wrong) prediction times. Training refers to the total actually training times in the comparison predictions. In AES, the weight of training times is half of the correct prediction times to let the most conservative type 3 prediction has 0 evaluation score. The weight of wrong prediction times is twice of the correct prediction times since a wrong comparison prediction means selecting significantly worse architecture. Then, it needs more time and computing resources to discover and rectify wrong prediction.
3.1.1 DATASETS AND HYPER PARAMETER CANDIDATES
We test the performance of different comparison prediction methods on time series forecasting datasets, where the time series value future time step predicted by a multidimensional time series sequence. In time series forecasting, time step values of different time intervals can be predicted by the same sequence, for example, a sequence from time step 1 to 100 can be used to predict the value of time step 101, 102, 103. Then, using the series to predict the value of different time intervals are the datasets with the same input X and different label Y.
Then, the experiments aim to predict the selection of different Long Short Term Memory (LSTM) hidden dimension numbers, where LSTM Hochreiter & Schmidhuber (1997) is a commonly used architecture for time series forecasting datasets and the hidden dimension number is a commonly adjusted hyper parameters in LSTM network. Probably the best hidden dimension numbers are related to the input dimension, we study the series with a fixed dimension and 100 series length. The candidate hidden dimension numbers are 4, 32, 256, which correspond to the hidden dimension similar to the time series dimension, the hidden dimension a few times larger than the time series dimension, the hidden dimension much larger than the time series dimension. We use the last hidden state of LSTM to predict regression output by 2-layer fully connected neural network with 50 hidden units and ReLU activation function. There are two comparison prediction tasks, where the first makes hyper parameter selection between hidden dimension number 32 and 256, the second makes selection between dimension number 4 and 32.
The experiments are made on nine different time series data with three data domains. Air: Three air quality data contain time series of daily air quality indexes collected from I Jan. 2014 to Mar. 2022 in the Shanghai, Beijing and Shenzhen cities of " WormMotion: Three organism motion data correspond to three mutant types (wild, goa-1. unc-38) ) of time series extracted from EigenWorms dataset in UEA time series classification archive [Banaal et al.] (2018). HandMEG: Three brain activity data contain the Magnetoenceppaalogaaphh (MEG) time series with three types of hand and wrist movement (left, right, up) extracted from HandMovementDirectionn dataset in UEA time series
https://agicn.com wwwpa/iiinn.oggaaaapaaaloooooooe
5

---

Under review as a conference paper at ICLR 2023
classification archive, where we select first 6 dimensions from 10 for forecasting to since want to keep the ratio between data dimension number and candidate LSTM hidden dimension numbers consistent in different data. For each data, forecasting the time series with 1,2.3 and 4 intervals corresponds to four regression datasets with the same X and different Y. More training samples usually has positive influence on model performance, but whether larger training dataset size could provide more concise architecture comparison prediction is uncertain. So we make experiments on various train size, from 200, 400, 800 to 1600. The validation and testing dataset size are 500, As result, there are total 144 datasets in the experiments.
3.1.2 BASELINES
CP: Conservative Prediction (CP) does not make any preference between two architectures, so conservative prediction always make type 3 prediction. CP does not require any dataset information.
SDP Similar Distribution Prediction (SDP 1) comes from the simple idea that the hyper parameter performs well in a dataset will also performs well in similar dataset. Here, we apply SDP I to predict the hyper parameter of a dataset named dataset 2 based on another dataset I, where two datasets have exactly the same X and different label Y (here dataset 1 and 2 are datasets with different forecasting intervals extracted by the same time series data). SDP 1 predicts hyper parameter of dataset based on statistical significance on dataset which makes type (or type 2) prediction on dataset if a hyper parameter named AR-A performs significantly better (or worse) on data 1. If two hyper parameters do not have significantly difference dataset 1, SDP 1 will make type prediction on data 2.
Similar Distribution Prediction 2 (SDP 2) needs the same dataset information as SDP and makes bold prediction based on eтror difference on data 1. SDP 2 makes type I (or type 2) prediction on dataset if AR-A performs better (or worse) on data 1, no matter whether the performance is significant or not.
SDP 2:
In our method, architecture comparison prediction can be made by searching probable datasets with loss in Eq. equation 6 When comparing the architecture hyper parameters denoted by AR-A and AR-B, two types of probable datasets need be searched, including the datasets that AR-A performs better and the datasets that AR-B performs better. Then the comparison prediction is based on the searching result: a searched probable dataset shows statistically significantly better performance of AR-A in testing dataset (after trained and validated in training and validation dataset) and no searched probable dataset shows statistically significantly better performance of AR-B, the method makes type i prediction. Similarly, if probable dataset that AR-B is significantly better exists and probable dataset that AR-A is significantly better exists, the method makes type 2 prediction. If the dataset that AR-A is significantly better and the dataset that AR-B is significantly better both can be searched, the method makes type 3 prediction. Here, we apply the probable dataset searching method to predict architecture comparison with different levels of uncertain dataset information.
PDS O: Probable Dataset Searching O (PDS 0) method searches dataset without any dataset information, so all parts of dataset information need be searched in PDS O.
PDS 1: Probable Dataset Searching 1 (PDS 1) method searches dataset without any information of label Y. so the regression function needs to be searched in the situation. Besides, PDS I has the input samples X information in the dataset but the saamiing/aaldaation/ssttn split of input samples X is unknown and shuffled in PDS In time series forecasting task, the true rraining/validttion/e split of input samples X is an important information since the input samples in each split are actually similar: time series forecasting, input samples are extracted by a sliding window and adjacent input samples are usually put in the same split since training set should not contain the forecasting target information of validation and testing set. Then the situation that input samples in each training/validationn/ www.vatiiontttttiiiiii set are similar contains more information than the situation that samples in each
set are shuffled.
Probable Dataset Searching 2 (PDS 2) method searches dataset regression function with the input samples X information, including vaaniin//allidaiion/essit split. Similar to PDS 1, the method does not require any information of label Y.
PDS 2:
Here, different methods require different levels of dataset information, which can be ordered as follows: CP = PDS 0 PDS 1 PDS 2 SDP = SDP 2. SDP and SDP 2 require most detailed
6

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison prediction AES of several methods.
Comparison Task
Datasct Domains
Train Size
CP
SDP
PDS
PDS
PDS
200
0
0.083
0.167
0
0.5
400
0.25
0.167
0
800
0
0.25
0.333
0
0
0.5
32 Dim vs 256 Dim
WormMotion
400
0.375
0.333
0
0.333
800
0.292
0.333
0.333
1600
0.208
0.417
0.333
200
-0.125
-0.333
HandMEG
400
0
-0.5
-0.75
0
800
-0.5
-0.833
1600
-0.125
-0.167
0.167
200
0.083
0.25
400
0
0.083
0.333
0
0.5
0.5
Air
1600
0.125
0.167
0.5
0.5
200
0.333
0.333
0.5
0.5
4 Dim Dim
WormMotion
400
0.375
0.333
0.5
800
1600
0.5
0.5
0.5
0.5
200
0,208
-0.167
0
0
0.5
HandMEG
400
0.25
-0.25
0.5
800
0.458
0.5
0.5
1600
0.5
0.5
Total AES
0
0.186
0.17
0
0.167
0.396
Avg. Correct
0.968
Avg. Wrong
0,032
0.11
0
0
Avg. Training Times
1.434
2
1.667
1.208
dataset information including the input samples X information and the label Y information on similar dataset. Probable dataset searching methods do not require such label Y information since that information usually can not be easily obtained in real world situation. Besides, CP and PDS requires specific information of the dataset.
3.1.3 RESULTS
Table 1 shows the AES results of three data domains on two hyper parameter comparison tasks, with the summarized information listed in the bottom 4 lines, including average AES, average correct times, average wrong times and average actually training times. The AES result of each data domain averaged by the AES of three data the domain, and each data corresponds four datasets with different forecasting intervals. For each dataset, there are other three datasets share the same input samples X and have different Y, so SDP 1 and 2 methods could make three comparison predictions with each dataset. The best AES score is highlighted in bold.
Firstly, it can be found that the probable dataset searching method 2 (PDS achieves the best AES score while it requires less dataset information than baselines SDP and 2. The predictions made by PDS 2 could make high level of correct prediction times with small actually training times.
Secondly, the performance of PDS is the same with the performance of CP, because when no dataset information is provided, each candidate hyper parameter may perform better due to the datasets searched by the method. As a result, in the experiments, PDS 0 always makes type 3 prediction for each dataset as the CP method does.
Thirdly, the difference of AES score among PDS 0, PDS I and PDS shows that with more dataset information, the probable dataset searching method could provide better comparison prediction between hyper parameters.
Fourthly, SDP and perform better than CP, PDS and PDS since they consider more dataset information. However, SDP and 2 not practical methods because similar dataset with the same input samples information is hard to obtain. Besides, the predictions made by SDP 2 decrease the actually training times but increase the wrong prediction times comparing SDP 1. Since wrong
7

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison prediction by a small training subset to select hidden dimension between 32 and 256, where 32 (or 256) means 32 (or 256) performs better in RMSE error value. Sig. (or No Sig.) means the difference is statistically significant (or not). The underlined number means the corresponding situation could be searched by PDS, for example, 6 in row 32 1 & Sig. means PDS method could find the dataset where 32 dimension significantly better in both 200 and 1600 train size.
Situation Happeued Times
Prodiction Methods
Train Size 1600
Correat/raining Times/AES
32 1 & Sig.
256 f Sig.
No Sig.
CP
SDPI
SDP2
PDS
32 1 & Sig.
1/22//
0.818/111-00045
0.811/11100045
1/22/0
Train 200
256 1 & Sig.
1/10/0
1/5/0.5
1/5/0.5
1/5/0.5
32 & No Sig.
1/22/0
1/22/0
0.909/11/0.227
1/22/0
256 1 & No Sig.
1/180)
1/18/0
0.556/9/-.833
1/18/0
Total AFS
0.056
-0.083
0.069
prediction could lead to severe consequence in hyper parameter selection, SDP 2 performs worse than SDP in AES score.
Finally, increasing the training dataset size could not apparently improve the hyper parameter comparison prediction performance, although larger training dataset size means more information for SDP SDP 2, PDS 1 and PDS 2 methods. We provide some additional illustration in Appendix.
3.2
ADJUSTING HYPER PARAMETER WITH A TRAINING SUBSET
Another situation of selecting hyper parameters without complete dataset is using a subset of training dataset to adjust hyper parameter. Sometimes the training dataset is large so using a subset to adjust hyper parameters an efficient approximation. Since the approximation may be wrong sometimes, apply the probable dataset searching method check the conditions when the approximate comparison is correct. The probable dataset searching method in this part does not require any information of input samples X so the procedure find the conditions do not influence the efficiency of the approximate comparison. As a result, probable dataset searching method here is actually PDS where all parts of dataset information need to be searched.
In this part, we apply the probable dataset searching method to find whether two architectures have different performances in different train sizes. Denote the training in the same dataset with two train sizes by a1 and ai. Then, training architectures in different training sets correspond to optimize the loss LST LRi (AR-A, AR-B, (AR-B/AR-B" are neural networks with the same architecture and different parameters O. AR-A/AR- AVARRB/AR-B' are tested in the same testing set a:2.
L Ri (AR-A', AR-B', af), where AR-A/AR-A'
To search for different probable situations of architecture comparisons in two train sizes, we needs different kinds of loss functions. For example, to optimize situation that AR-A performs better than AR-B in training set On while they performs similar in training set a'1, the loss function is T.51 Tzz AR-A, AR-B) + nk Σi= ||gAR. (X,, ARRN) - gAR B (x,, 0AR [2 where X; is the input sample in Q22 is the total test size, K is the dimension of g.
Another probable situation AR-A performs better than AR-B in training set aj while AR-B performs better in training set a', which corresponds to the loss Ls2 LRa(AR-A AR-B) AR-B) LARRAR-B' AR-A'). For the situation that AR-A performs better than AR-B in training set an and AR-B").
a1. the loss is Lgs Lga (AR-A, AR-B) +
3.2. RESULTS
Table 2 shows the experimental results, where we use 1600 and 200 train size pairs of all the datasets in the former experiment, which contains totally 36 dataset pairs. In this part, different methods use the training dataset with 200 size to predict the hyper parameter selection in train size 1600. We list the happened times of different situations in dataset pairs in the left part of the table, for example, 6 dataset pairs satisfy the situation that 32 hidden dimension is significantly better than 256 hidden dimension with both 200 and 1600 train size. The performance of four methods in different situations
8

---

Under review as a conference paper at ICLR 2023
is listed in the right part of the table, for example, for the 11 dataset pairs that 32 hidden dimension is significantly better with 200 train size, the average AES of SDP 1 is -0.045.
Here, similar to the former experiments, there are also three types of comparison predictions. Con- servative Prediction (CP) always makes type 3 prediction. SDP 1 predicts comparison based on statistical significance with train size 200, where SDP 1 make type prediction if no significantly difference is observed with train size 200. SDP 2 predicts comparison based on error difference with train size 200. PDS makes prediction based on the architecture statistical difference with train size 200 and the searched probable situations with train size 1600. For example, when 32 hidden dimension is significantly better with 200 train size, both 32 and 256 hidden dimension may be significantly better in PDS (6 and 2 are underlined in the table), so PDS makes type prediction.
The result shows that PDS performs best among baselines including CP, SDP and SDP In most situations, SDP performs similar to PDS, which means SDP is still good approximation comparison method. SDP method performs worse than PDS since it makes more wrong predictions when 32 hidden dimension is significantly better with 200 train size, which also shows that selecting hyper parameter from a subset training dataset is not always correct.
4 RELATED WORKS
Uncertain Dataset Information. In application, different parts of dataset information have different difficulties to obtain, which leads to several types of task with uncertain dataset information. In unsupervised learning Barlow [ Creswell et al. 2202] Baevski et al. (2021); ) Yu et al. (2021) Choudhury al. 12 data labels are assumed hard to be obtained so models are expected to work without data labels. In domain adaptation Wang & Deng (2018): Dong et al. (2021): Zellinger et al.)
(202I); Chen & Chao (202I); Lv et al. (2021); Rostami (2021), complete dataset information in source data domain and unlabeled dataset in target data domain are used to train models in target data domain. In few-shot learning Wang et al.) (2020); Snell et al. (2017); Wang et al. (2021); Sendera et al.) (202T): Cao et al. (202T), few train size dataset is used to train models with additional information of similar dataset. In this paper, we discuss the adjustment of architecture hyper parameters with two kinds of uncertain dataset information: 1, dataset labels are postponed be obtained so hyper parameters need be adjusted without complete dataset information. 2, adjusting hyper parameters with subset training dataset since training models with complete training dataset is time consuming.
Regression Dataset Representation. Recent works Kohler & Langer (2021) ; Bauer & Kohler (2019); Sccmidtt (2020); Suzuki (2018); Farrell et al. (2021) show that if regression func- tion fo satisfies some requirements, it can be converged by a layer fully connected neural network. Although the requirements ments (SantnstSiiitHHeeeee wide range of function types these requirements tion in real world functions. Our searching method use a neural network to approximate function fo and assume that the convergence is also true in other types of neural network. Our regression dataset representation further assumes that at least compared architecture has competitive performance when the compared architectures worth for application.
diffferent in different works (such as Hölder function require- (020) and (p.C)) (smmooht requirements Bauer & Kohler (2019), they include a assumed to be satisfied by the regression func-
5 CONCLUSION
In this paper, we discuss adjusting hyper parameters in two types of uncertain dataset information, including the postponed dataset labels and only using a subset training dataset. To infer the probable complete dataset information, we propose a probable dataset searching method which searches the variables in our dataset representation. We test the performance of our method in 9 real world data with two types of uncertain dataset information.
REFERENCES
Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. Unsupervised speech recognition. Advances in Neural Information Processing Systems, 34, 2021.

---

Under review as a conference paper at ICLR 2023
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. 2018.
Horace B Barlow. Unsupervised learning. Neural computation, 13)):295-3 -311, 1989.
Benedikt Bauer and Michael Kohler. On deep learning as a remedy for the curse of dimensionality in nonparametric regression. The Annals of Statistics, 47(4):2261-2285 2019.
Yuhang Cao, Jiaqi Wang, Ying Jin, Tong Wu, Kai Chen, Ziwei Liu, and Dahua Lin. Few-shot object detection via association and discrimination. Advances in Neural Information Processing Systems, 34, 2021.
Houg-You Chen and Wei-Lun Chao. Gradual domain adaptation without indexed intermediate domains. Advances in Neural Information Processing Systems, 34, 2021.
Subhabrata Choudhury, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Unsupervised part discovery from contrastive reconstruction. Advances in Neural Information Processing Systems, 34, 2021.
Antonia Creswell, Rishabh Kabra, Chris Burgess, and Murray Shanaban. Unsupervised object-based transition models for 3d partially observable environments. Advances in Neural Information Processing Systems, 34, 2021.
Jiahua Dong, Zhen Fang, Anjin Liu, Gan Sun, and Tongliang Liu. Confident anchor-induced multi-source free domain adaptation. Advances in Neural Information Processing Systems, 34, 2021.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, The Journal of Machine Learning Research, 29(1):1977-2017, 2019.
Max H Farrell, Tengyuan Liang, and Sanjog Misra. Deep neural networks for estimation and inference. Econometrica, 89(1):181-214 2021.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()):
1735-1780, 1997.
Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast bayesian optimization of machine learning hyperparameters large datasets. In Artificial intelligence and statistics, pp. 528-536. PMLR, 2017.
Michael Kohler and Sophie Langer. On the rate of convergence of fully connected deep neural network regression estimates. The Annals of Statistics, 49(4):2231-22444 2021.
Fangrui Lv, Jian Liang, Kaixiong Gong, Shuang Li, Chi Harold Liu, Han Li, Di Liu, and Guoren Wang. Pareto domain adaptation. arXiv preprint 2rXiv:2112.04137,, 2021.
Mohammad Rostami, Lifelong domain adaptation via consolidated internal distribution. Advances in Neural Information Processing Systems, 34, 2021.
Johannes Schmidt-Hieber Nonparametric regression using deep neural networks with relu activation function. The Annals of Statistics, 48(4):1875-1897 2020.
Marcin Sendera, Jacek Tabor, Aleksandra Nowak, Andrzej Bedychaj, Massimiliano Patacchiola, Tomasz Trzcinski, Przemyslaw Spurek, and Maciej Zieba. Non-gaussian gaussian processes for few-shot regression. Advances in Neural Information Processing Systems, 34, 2021.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
Taiji Suzuki. Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality, In International Conference on Learning Representations, 2018.
Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputing, 312: 135-153, 2018.
10

---

Under review as a conference paper at ICLR 2023
Ruohan Wang, Massimiliano Pontil, and Carlo Ciliberto. The role of global labels in few-shot classification and how to infer them. Advances in Neural Information Processing Systems, 34, 2021.
Yaqing Wang, Quanming Yao. James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020.
Peiyu Yu. Sirui Xie, Xiaojian Ma, Yixin Zhu, Ying Nian Wu, and Song- Chu- Zhu. Unsupervised foreground extraction via deep region competition. Advances in Neural Information Processing Systems, 34, 2021.
Werner Zellinger, Natalia Shepeleva, Carits-Constantin Dinu, Hamid Eghbal-zadeh, Hoan Duc Nguyen. Bernhard Nessler, Sergei Pereverzyev, and Bernhard A Moser. The balancing principle for parameter choice restance-regularte domain adaptation. Advances in Neural Information Processing Systems, 34:20798-200 2021.
A APPENDIX
A.1 DATASET INFORMATION SEARCHING DETAILS
In Eq. (6), the searched variables AR-DD and e influence LR2 two aspects: 1, they directly influence testing dataset 02 in Eq. (6). 2. they indirectly influence training dataset ar,, which influence the parameters ARRA and AR-B in Eq. (6).
The parameters of AR-A and AR-B trained by an in Lm and these parameters influence the loss LR2- When parameters are searched by gradient decent, each step of training the i-th element of ARR-A, denoted by AR.AR- with LR1 can be written
0R.AR-A = OsARRA - R AL (AR-AR)).
00LAR--
where denotes the learning rate, AgAAR-A denotes the parameter after backpropagated by gradient decent in step. ARRBB be trained by the same mechanism.
we approximately consider the influence of a training dataset aj on AARA as a one step gradient decent (which also means Lm and Ln2 are optimized at the same time), the influence of dataset parameter fLARRD on L12 through AAA can then be analyzed. When only considering the parameter ALAR in gradient decent, the partial derivative of Ia with the respect to the parameter 0,ARDD in
the multiplication between partial derivative 00LARRA and ŌLR2
A0 00GAR-A
0d,ARR- 0L -R dL 8 (AR-A,,)))) -Partial(AR-A,,, (10)
00,AR-D 80, a0 A0,
where we let the backpropagation of Oj,AR-A prior to the backpropagation of 0jAARD at each step. 8 (2RA-A an)
can be calculated since ORAR-D influence aj in L2.
AR AAR-A
Then training ORARRD when considering the influence of both 02 and on can be denoted by
(022( + RR² Σ
0R.AR.D= = ..ARDD R
R Σ
Partial(AR-B. k),
009.AR-D
(11)
where 2 2 (0x2(OAR)))) denotes the derivative only considering the direct influence of 0jAR
A0..AR-D
on testing dataset 02 Since the learning rate R is small value such 0.01, the gradient Partial(AR-A. i. and Partial(AR-a j. have small influence on the gradient decent of ARAARD which is omitted in implementation. The gradient decent of other searched dataset variables have the same characteristics. As a result, we only consider the first aspect the influence on L Lgz when searching probable datasets with T,22.
11

---

Under review as a conference paper at ICLR 2023
PDS 1:1
P-5--2 2-2
-e- 52-3
PDS 1-1 --- FDS 2-2 MEGR-1
PDS 1-
1052-2
9H-3
PD5 1-2
57-1
-9. 57-4
PDS 1-2 --- MEGB-1 ---
MEGR-4
PDS 1-2
5161
-e- 5114
POS 2-1
S2-2
PDS 2-1 MEGB-2
POS 2-1
Sth-
T-test
T-test
200
400
300
1600
260
400
800
1600
200
400
800
1600
Training Samples
Training Samples
Training Samples
(a) Comparing 32 and 256 hidden dimension in Air domain, Shen- zhen data with four time intervals.
(b) Comparing 32 and 256 hidden dimension in HandMEG domain, left hand and wrist movement da- ta (MEGR, MEG Right) with four time intervals.
(c) Comparing 4 and 32 hidden di- mension in Air domain, Shanghai data with four time intervals.
Figure 2: Hyper parameter comparisons in real world complete datasets and the probable datasets searched by PDS and 2. The comparisons between hyper parameters are measured by /-test score with positive/negaaive symbol. The black dash-and-dot lines refer to the t-test score that has statistically significant difference.
A.2 SEARCHED PROBABLE DATASETS ILLUSTRATION
In this part, we show the difference between hyper parameters through the f-test score, where a t-test larger 2.262 means two hyper parameters statistically significantly different with 5% significance level. For easier representation, when comparing the error value of two architectures named AR-EA and AR-EB, we apply a positive (or negative) symbol the (-test score, which corresponds to the situation that the error value of AR-EA is larger (or smaller) than AR-EB. In the figure, when comparing n1 and n2 hidden dimension, AR-EA and AR-EB correspond to the architectures with nı and 12 dimension in showing f-test score with symbol. Here we show the hyper parameters difference of real world complete datasets and searched probable datasets in Fig. 2 where the f-test score with symbol are reported with real word datasets and the datasets searched by PDS 1 and 2 methods in each sub-figure. For example, in Fig. [(a) SZ-1, 2, 3, refers the Shenzhen data with forecasting intervals 1, 2, 3, 4. PDS methods are followed with two numbers, where the first number refers to the type of PDS method and the second number corresponds to the architecture whose performance is aimed to be improved in PDS method (second number 1 or 2 corresponds to AR-EA or AR-EB). For example, PDS 2-1 refers to the probable datasets searched in loss Eq. (6) with PDS 2 methods (because of the first number), where the loss aims to search for the probable dataset that AR-EA (corresponds to 32 dimension, because of the second number) performs better, so AR-A and AR-B in loss Eq. (6) corresponds to the LSTT with 32 and 256 hidden dimensions. Similarly, PDS 1-2, PDS 2-2 refer to the probable datasets searched in loss Eq. (6) with PDS I and 2 method, where the loss aims search the probable datasets that 256 dimension performs better. In the figure, if value is larger or smaller than two black dash-and-dot lines, corresponding comparison has statistical significance.
The results show that with more information required (PDS 1 << PDS 2), PDS method could make better predictions. For example, when comparing hyper parameters between 32 and 256 hidden dimensions in SZ. datasets, PDS 1 method could find the probable dataset that 32 or 256 hidden dimension performs significantly better than another, so makes type predictions. PDS could not find the situation that 256 hidden dimension performs significantly better, so it makes type I prediction. Considering the performance of two hyper parameters that 256 hidden dimension does not performs significantly better in any SZ real world dataset, type 1 is better prediction. When comparing hyper parameter between and 32 hidden dimensions in SH datasets, PDS and 2 both make type predictions although they require different levels of dataset information.
The result in Fig. 2 the situation that SDP 1 and make wrong predictions, where for the same X, there real world datasets with different Y that 32 or 256 hidden dimension performs significantly better than another. In these datasets, PDS methods make type 3 predictions, which avoid making wrong predictions.
12


Reviews:
[{"review_id": "sQgE3V_Pa4", "reviewer": "Reviewer_xHNU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tries to adjust hyper parameters in two types of uncertain dataset information: 1) dataset labels are postponed to be obtained so hyper parameters need to be adjusted without complete dataset information. 2) hyper parameters are adjusted with a subset training dataset since training models with complete training dataset is time consuming. More specifically, this paper proposes several loss functions to search for probable dataset when the complete dataset information is not obtained. The authors conduct experiments on 9 real world datasets to demonstrate the performance of the proposed method.", "summary_of_the_review": "Overall, this paper contributes some new ideas but the empirical evaluation is limited. Also, the presentation needs to be improved.", "strength_and_weaknesses": "**Strengths**:\n\n1. This paper proposes an interesting task: to adjust hyper parameters under different uncertain dataset information settings.\n\n2. The paper proposes several losses to search for probable dataset when the complete dataset information is missing.\n\n3. The paper conducts experiments on multiple real world datasets.\n\n**Weaknesses**:\n\n1. It’s not very clear to me how to train the AR-D. The paper mentions that ‘we let the dataset regression function architecture AR-D be the same with AR-A in $L_{R2}$ since we want to search for the dataset that AR-A performs well and a dataset with architecture AR-A should best fit the AR-A model.’ Initializing AR-D with the parameters of pertain AR-A? Otherwise, if only using the same architecture, is there a guarantee that the AR-A will perform better than AR-B? \n\n2. The experimental part is limited. In the current version, only classification tasks on small datasets have been conducted. I would expect more strong experimental results to support the main claim of the paper. For example, how does the proposed probable dataset searching strategy affect more complex tasks with much bigger datasets?\n\n3. The AES evaluation metric (Eq.8) is not well justified/motivated. How to determine the weights of ‘Wrong’ and ‘Training’?\n\nMinors:\n\n1. I would suggest remaking figure.1 to improve the figure quality.\n\n2. Eq. equation 1 -> Eq.1.\n\n3. A few typos: \n\n  * Page 6, paragraph CP, ‘always make type 3 prediction’ -> ‘always makes type 3 prediction’.\n  \n  * Page 6, paragraph SDP 1, ‘will also performs well’ -> ‘will also perform well’.\n  \n  * Page 6, paragraph SDP 1, try to be consistent to use dataset 1 or data 1.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "n5sVlxzsgT", "reviewer": "Reviewer_Au4D", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes approach that enables simple hyperparameter search in scenarios where either label information is delayed or when the training on full dataset is too costly.  The paper proses to use an auxiliary neural network to predict missing labels and a method to search for probable dataset to select a subset of data to use for training. The proposed solutions are tested on three datasets (Air, WormMotion, HandMEG) and compared to two baselines – the results show that the introduced methods outperform baselines.", "summary_of_the_review": "Overall, the paper studies an interesting problem of model hyperparameter prediction on uncertain data. However, the paper have several important weaknesses that makes it hard to recommend the paper for acceptance: (1) the presentation of the paper is very difficult to follow, (2) the evaluation is rather weak, (3) the paper is not properly positioned w.r.t. previous art.", "strength_and_weaknesses": "Strengths:\nThe idea of predicting model hyperparameters is interesting\n\nWeaknesses (for details, see section below):\nThe paper is very difficult to follow\nThe validation is limited\nLack of comparisons to previous works.\nLimitations are not discussed\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "KnFpG-6vTN3", "reviewer": "Reviewer_d4iE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper mainly discusses adjusting hyper parameters in two types of uncertain dataset information, including the postponed dataset labels and only using a subset training dataset. To infer the probable complete dataset information, this paper presents a probable dataset searching method that searches the variables in our dataset representation. Experimental results have been conducted on 9 real-world data with two types of uncertain dataset information. ", "summary_of_the_review": "This paper mainly discusses adjusting hyper parameters in two types of uncertain dataset information, including the postponed dataset labels and only using a subset training dataset. However, the overall technical novelty should be strengthed and more related SOTA comparisons are needed. ", "strength_and_weaknesses": "Strength:\n(1) To deal with the uncertain information in a dataset, this paper proposes a probable dataset searching method to predict architecture comparison.\n(2) Different experimental settings are studied, and the authors validate the performance of the proposed method in 9 real-world data with two types of uncertain dataset information. \n\nWeakness:\n(1) This paper utilizes a neural network to approximate the dataset regression function and apply several loss functions to search for the probable dataset. In this case, the authors adopt several existing technologies, so the technical contribution is weak.\n(2) The authors state that they propose a probable dataset searching method to predict architecture comparison. However, many SOTA methods are missing in the comparison experiments. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject