Proximal Validation Protocol
Authors: MingFeng Ou
Yiming Zhang
Sai Wu
Gang Chen
Junbo Zhao
Venue: Submitted to ICLR 2023 (2023)

Abstract: Modern machine learning algorithms are generally built upon a train/validation/test split protocol. In particular, with the absence of accessible testing set in real-world ML development, how to split out a validation set becomes crucial for reliable model evaluation, selection and etc. Concretely, under a randomized splitting setup, the split ratio of the validation set generally acts as a vital meta-parameter; that is, with more data picked and used for validation, it would cost model performance due to the less training data, and vice versa. Unfortunately, this implies a vexing trade-off between performance enhancement against trustful model evaluation. However, to date, the research conducted on this line remains very few. We reason this could be due to a workflow gap between the academic and ML production which we may attribute to a form of technical debt of ML. In this article, we propose a novel scheme --- dubbed Proximal Validation Protocol (PVP) --- which is targeted to resolve this problem of validation set construction. Core to PVP is to assemble a \emph{proximal set} as a substitution for the traditional validation set while avoiding the valuable data wasted by the training procedure. The construction of the proximal validation set is established with dense data augmentation followed by a novel distributional-consistent sampling algorithm. With extensive empirical findings, we prove that PVP works (much) better than all the other existing validation protocols on three data modalities (images, text, and tabular data), demonstrating its feasibility towards ML production.

Under review as a conference paper at ICLR 2023
PROXIMAL VALIDATION PROTOCOL
Anonymous authors
Paper under double-blind review
ABSTRACT
Modern machine learning algorithms are generally built upon train/validationneest split protocol. In particular, with the absence of acces- sible testing sets in real-world MI. development, how split out a validation set becomes crucial for reliable model evaluation, selection and etc. Concretely, under a randomized splitting setup, the split ratio of the validation set generally acts as vital meta-parameter; that is, with more data picked and used for validation, it would cost model performance due the less training data, and vice versa. Unfortunately, this implies a vexing trade-off between performance enhancement against trustful model evaluation. However, to date, the research conducted on this line remains very few. We reason this could be due to a workflow gap between the academic and ML production which we may attribute form of technical debt of ML. In this article, we propose a novel scheme dubbed Proximal Validation Protocol (PVP) - which is targeted to resolve this problem of validation set construction. Core to PVP is to assemble proximal set as a substitution for the traditional validation set while avoiding the valuable data wasted by the training procedure. The construction of the proximal validation set is established with dense data augmentation followed by a novel sampling algorithm. With extensive empirical findings. we prove that PVP works (much) better than all the other existing validation protocols on three data modalities (images, text and tabular data), demonstrating its feasibility towards MI. production.
Eestriuttional-coniistee
INTRODUCTION
Most, if not all, machine learning production and research are conducted based on a train/test/valiaation set split protocol. A machine learning engineer or scientist often first receives labeled dataset and splits it into a training and validation set, respectively. The role of the validation set is critical when considering robust model evaluation, selection, hyper-parameter tuning, etc. Post to the validation protocol, the best model being picked would be fed to the testing protocol, where the testing set is generally not accessible during real-worle ML development till this phase.
Notably, prior to splitting the labeled dataset, one needs to determine the split ratio of the validation against the training set. This ratio be very tricky: if fewer samples are picked up for the training protocol, the model validation can be less reliable. Contrarily, the larger validation set effectively shortens the training resources, which may lead to performance degradation. In current days, this ratio is often set based on the experience level of human expert. This problem anchored the split ratio can also be exhibited more complex validation schemes like cross-validation.
Indeed, this problem of setting the (sub-) Jottimllt validation set is often, or mostly, ignored by the academic cohorts in the community. To date, as we scrutinize the related literature, very few works have touched down on this line (Li et al.) 20200] Moss et al.) 2018 Joseph & Vakayil 2021). In hindsight, a large portion of the standardized academic benchmarks have had prefixed validation set split, such as ImageNet (Krizhevsky et al.) (012), COCO (Lin et al. (2044), and SST (Socher (et al., 2001).. Also, the testing set often visible for the evaluation of the academic research. On the one hand, this prefixed validation setup has some merit. For instance, this effectively dedicates the ML research to the model towards innovated model architecture, optimization methods, new learning paradigms, etc. On the other hand, argue this could attribute to the technical debt (Scul- ley et al. (015)) of ML. When considering ML production for real-world applications, we make the following statements: (i)-not many application tags along with adequate large-scale data because

---

Under review as a conference paper at ICLR 2023
Traditional Approach
Ours: Proximal Validation Protocol
Saurce Set
Source Set
Augmentation
Auxiliary Set
Train
Val Set
Srain
Distributienal-crraiseee Sampling
Train
Evaluate
Train
Eva lete
Model
Model
Proximal Validation Set
Figure I: Traditional validation scheme (left) vs. Proximal Validation Protocol (right).
data curation and annotation both cost fortune; (ii)-the testing set is often not accessible: consid- ering the CTR prediction or manufactural defect detection where the testing data present and only present post to the deployment; (iii)-the validation set is almost always decided by the ML experts with their expertise. In these scenarios, how to split out a validation set may sit in the center. The benign condition where validation set preset and fixed almost always does not hold in ML. production.
To this regard, we propose the Proximal Validation Protocol (dubbed PVP). With this novel valida- tion protocol, we attempt to (fully) resolve the split problem and its trade-off. The core idea of PVP rather simple. It first synthetically generates a validation set based on the labeled dataset without any splitting. Then, a novel distributionallconsiin lect the most suitable synthetic data point for validation. The resulting set dubbed the proximal validation set. Thanks to the proposition of the proximal validation set, PVP (in theory) does not rely on any real labeled data point for validation, effectively leading to performance improvement. Notable, the comparison of PVP with the conventional validation protocol is graphically depicted in Figure [
sampling algorithm is applied in order to se-
Empirically, we extensively conduct experiments three data modalities - including tabular data, image data, and text data. We actively compare the PVP with standardized methods like the holdout protocol, K-fold cross validation, well as the very limited related work like Poseph & Vakayil 2021). Besides the series of analytical justifications, we choose three major metrics to form tair and comprehensive comparison: the performance, 1-v gap and variance. Notably, performance means the test score (c.g., AUC and Accuracy) of a model, variance refers to the stability of the estimated performance (on validation set) under different random seeds, and f-v gap indicates the closeness of the estimated performance to the test one. We empirically show that PVP achieves better performance, lower bias and competitive variance than the standardized split-relied methods. With three major data modalities being experimented, we hope that the ideology and instantiation of the PVP can pave a way for more effective validation protocol towards ML production on real-world applications.
At last, we may summarize the contribution of this work as follows:
We propose novel validation scheme-worts - PVP stable and reliable validation pro- tocol relying on only the synthetic data while capable of enhancing the model performance.
The decent empirical results of PVP on three major data modalities manifest its "plug-and- play" nature. Its design is very much input data-dependent but independent of the model, architectures, optimizers, and tasks. We hope PVP can shed some light on data-saving, performance-effeccivv,, lightweight and profound validation procedure.
The code PVP will be made public upon publication.
RELATED WORK
As we mentioned, the related literature remains very few. Looking back to the old days, the vali- dation framework was raised to fix the issue of overfitting (Mosteller & Tukey] 1968 Stone 1974 Geisser 1975), which was first noticed by Larson [1931). Due to the universality of the data split- ting heuristics, the split-relied method can be applied to almost any algorithm in almost any frame- work (Arlot & Celisse 2010).
2

---

Under review as a conference paper at ICLR 2023
Recent works aim to reduce the validation estimate's ' instability, imprecision, and time cost. Specif- ically, the instability refers to variance between multiple training results with random seeds on the same model (Moss al. 2018). And the imprecision means the gap between the model'ss evaluation results on the validation dataset and the test dataset (Zeng & Martinez 2000). These works can be divided into two categories. Some works explore the variants of traditional data-split validation frameworks (Moss et al. 2018 Kohavi et al. 1995 Jiang & Wang 2017 Székely & Rizzo, [013] Jung 2018 Li al. 2020 Tuttaaenn
2021; Zeng & Martinez 2000), such as holdout and k-
fold. The others try to propose a better split algorithm for the validation dataset generation. JJosehh
&
(202T) 202T) Budka & Gabrys (2012) propose methods to sample a specific subset from the training set to generate a validation set in the tabular scenario. [Josehh (2022) propose the optimal train/validate splitting ratio theoretically, but only for the linear regression model.
However, most, if not all, of the literature works are limited by "split-relied" framework. Based on the logic chains from our previous section, they mostly would suffer from the splitting tradeoff problem. Among them, we deem PVP the pioneer attempt to be fully split-free. The split-relied framework and suffer the cain/validation split tradcoff.
3 METHOD
3.1 PROBLEM SETUP
To case the discussion of validation methods, we start with problem setup. First, we define a source set Dere as compilation of the training and potential validation set, i.e., unsplit (labeled) training set. Similarly, the testing set is defined as IM Further, we follow the well-known assumption that the instances of both datasets are I.I.D and drawn from unknown underlying density distribution (X)
D = [[xx""""""" DE D ====(((;" {xx) where (Xi, 21:
F(X,)))
(1)
where N. M denote the number of labeled instances in the source set and test set, respectively, and E Rdd y E R are the input and label space. From the source set Drre,, we further define the validation and train set so as to formulate the traditional validation process (e.g., the holdout scheme).
Definition 3.1 (Validation Process). We define a validation set, D C(Dero), where a (stochastic) collecting function facilitated by different seeds. This definition, in turn, yields defini- tion of a training set, DD = are \ Deal, by eliminating the validation samples from the source set. From here, we farther define mapping function f. (X) + y. Finally, we define the validation process as &f Dr where 3 is a scoring function compiling the inference pass together with an evaluation metric, e.g., accuracy, fl-score.
Now, we describe the evaluation metrics of the validation method, which is essential for judging the quality of the validation method. We formally declare the T-V Gap and Variance from the perspective of validation, following previous works (Arlot & Celisse, 2010 Zeng & Martinez 2000).
Definition 3.2 (T-V Gap). We define the test-validation gap (T-V Gap) as the difference between the test score(i.e., performance on the data space (X)))) and the estimated score of practically yielded function:
ISCS. (X) S(S, )T
Definition 3.3 (Variance). We define C as set of collection functions under different random initialization, and the variance is defined as:
E (D $(f.
S(S, C(Doar)))
(3)
n
Intuitively, T-V Gap signifies the inaccuracy of evaluation. A small T-V Gap corresponds to a precise estimate of the true score (usually refers to performance in the real-world test environment). The Variance measures the stability the evaluation. And low variance means the estimated score is robust under collection function C with different random seed. Ideally, with near-zero T-V Gap
3

---

Under review as a conference paper at ICLR 2023
and Variance, the evaluation result is reliable, stable and able to obtain the best function f with the highest score (on the test set).
Usually, perhaps not practical on many occasions, when more labeled data is set for validation, i.c., gathered by or C, the greater intersection extent between the chosen set becomes larger. On the one hand, this would effectively reduce the Variance. Besides, this will make Dewl closer (X ))), which would reduce the 7-V Gap. On the other hand, this further causes a reduction of the labeled samples for training which may degrade the test score of the algorithm. Following this setup, in the pursuit of decent validation set, we may term it a "split tradeoff"
Essentially, in the conventional randomized cross-validation a plague to set the ratio between samples used for training and samples entertained for validation drawn from the source set. We find that this line of research is very much absent. Most research has set this split ratio as a preset meta-parameter and stuck with it throughout the development. However, as important as it is, we attempt scrutinize this problem and propose a systematic "split-free" solution.
vanilla holdout schemes, it is always
3.2 PROXIMAL VALIDATION PROTOCOL
3.2.1 OVERVIEW OF FRAMEWORK
As introduced in definition 3.1 the validation set is obtained by collecting function C, i.e., Deal C(T while the train set is formed via D Teaee maal However, to achieve a split-free solution, all data are left for training (i.e., D = Da Thus, we design new collecting function, termed C'. The sample set yielded by C is expected to have comparable evaluation quality to its traditional validation set counterpart - comparable T-V Gap and Variance (in definition 3.2 and 3.3) and comparable or superior performance.
As result, manage to build framework PVP for C' and prove that it requires the following two critical components (colored with red in Figure
A data generator G to produce candidate samples from source set Derc, which we define as auxiliary set Dawx = G (Meree. To tackle the obstacle of no available data for validation set construction when all labeled samples of D are saved for training, we generate a set of synthetic data. To prove this concept, we resort to the simplest method - the data augmen- tation family- - to implement the data synthesis process. The reason is two fold: (i)-the augmented data can be arbitrarily large in theory, which may contribute to low variance by providing sufficient samples for validation; (ii)-the methods from the data augmentation family are generalized and can adapt most of the data modalities and tasks.
A cetpaattiiiaaacccssteeeete sampling algorithm a selects distributionally representative samples from the auxiliary set to form a validation set. The chief challenge lies in how to design the strategy to locate suitable samples among the auxiliary set, which can be chaotic and biased due to the randomization nature of data augmentation. Again, we propose a frustratingly simple method by relying on an angle-based distribution approximation (see Figure 2 to chase for small T-V Gap evaluation.
In general, C' can be formulated as C' s (-).. And the output of C (Deree. i.e., proximal validation set Drro, can replace the ID nal in the validation process (in definition 3.1b.
By proposing PVP, entertain the possibility of relying (purely) the generated samples to construct the validation process. Practically, this may pose a great number of advantages over the traditional split-relied validation process because it saves the samples for training. which likely leads to performance gains.
In the following, in correspondence the aforementioned bullets, detail the instantiation of the PVP framework.
3.2.2 DATA GENERATOR
The first step to facilitating a proximal validation process is to form a candidate pool for the proximal validation set, which uses original data points in the source set. We include an external data
4

---

Under review as a conference paper at ICLR 2023
Intra-class angle - Class Center)
Sample distribution Selected Points?
Source
Feature Extractor
Auailiary Set
Praxy Set
Fit distribution P A= p=
Figure 2: Illustration of ceterruutionalcconissttee sampling algorithm. The steps of the algorithm are displayed from left to right. First, the distribution of D is estimated via intra-class angular distribution on feature space. The estimated distribution is then used to select samples from the auxiliary form proximal validation set.
generator G to conduct the construction. The function 9 can be implemented by many existing methods. Specifically, profoundly choose the data augmentation approaches and carefully take them to adapt to the requirements of high validation quality in Section 3.1 Generally, we use this module to generate large number of synthetic candidate data points, thanks to its continuous nature. By forming a large pool, we aim to bring down the variance metric during the final proximal validation. In addition, is pointed out by Xu et al. (2022), the data augmentation scheme when designed and implemented properly - is capable of covering the data space for the most part yet providing denser coverage. This may correspond to the metric of T-V Gap evaluation (definition 3.2 that relates to the following module.
Formally, we can define the data generator 9 and the candidate pool Dour (named the auxiliary ser) more concretely as:
Dave G (DDT (Do (D=
(4)
where G consists of set of Q augmentation functions, i.e., G {go. UI.....9(" In particular. conduct PVP on three fields, and the detailed augmentation methods listed Appendix A There is admittedly rich literature around data augmentation; this is mostly embedded into the train- ing stage for feeding more samples train the model instead of for validation purposes.
Data generation is another feasible way to generate the auxiliary set. However, given the extendibil- ity and computational advantages of the data augmentation methods, we stick PVP with them. We intend to leave the exploration of dataset generation methods, such as generative modeling (Good- fellow al. 2014) and dataset distillation (Wang et al. 2018), to work further.
3.2.3 DISTRIBUTIONAL-CONIISI
SAMPLING ALGORRITH
The prior data augmentation module can, in theory, produce arbitrarily large number of samples for validation. This followed module devised to select the most suitable samples to form the proximal validation
in place of the validation of the conventional counterpart workflow. As mentioned in Section 3.1 high evaluation quality expects low T-V Gap and variance. It can be achieved if the distribution of Ioal differs slightly from F(X, y) and Maal has large volume. However, we do not have the precise form of F. Instead, we only have Dare, which is a set of realizations from F. Thus, we can use the empirical distribution of the source set as a substitute for F. And we abide by the expectation to propose the cnaributioonllconnisttee sampling algorithm aed to select sufficient samples that are distributional representations.
while the augmented data may be able to wrap the data space of the source set, we must be careful with it due to the inductive bias from the data augmentation methods. Revealed from the literature (Zhang al. 2015 He et al., 2201),, the distribution of auxiliary set, produced by a set augmentations, may drift from that of the source set. This drift, on the downside, may produce large T-V Gap in the estimated score in the validation process if directly employ the auxiliary set as a proximal validation set. Therefore, a sampling algorithm is indispensable in proximal validation construction for less biased evaluation.
As mentioned in Section 3.2.2
Briefly, we propose the simplest solution. This algorithm first characterizes the empirical distribu- tion of source set There by an explicit angular distribution (Liu et al. 2020 2017) and then samples
5

---

Under review as a conference paper at ICLR 2023
the angles via an explicit function to locate the corresponding points in the auxiliary set (as illus- trated in Figure 2
Distribution Estimation with Angles. The first step of our algorithm is capture the empirical distribution of the source set. We adopt the simplest but most efficient method intra-class angular
distribution feature space (Liu et al. 2020, Kobayashi 2021 Liu et al., 2017) for proof-of- concept of PVP. Unlike common collecting functions that only consider inter-class distribution (e.g., stratified random sampling), our C' further measures intra-class distributions so as to control the distribution of validation set more finely, and ultimately keep the T-V Gap of the evaluation at a low level.
To be specific, we modeled the intra-class angular distribution on the angles between samples from Døre and their corresponding class centers. Given a sample x with category label y, let z = (xx;) be the features extracted by an extractor Q, where we utilize BERT (Devlin et al.) 2018) for text data and ResNet- 18 He et al. (216)) for images. We define the calculation process of angle as A and define the angle of x relative to its class center as follow:
a.b
a. =x(((x))) arccos(z? (z,,c)), c²), (a. ||a|| ||b||
(5)
c² Σ N2 exp
Ny Σ (2), 2
(6)
プز exp expwy
where c², Ny denotes the class center and the number of labeled instances of class y, respectively. Note that c is calculated by the weighted average of the features within the class y. Assuming the angles obey a Gaussian distribution, we can define the angular distribution for class y (ay)(a";() and its parameters can be obtained via maximum likelihood estimation (MLE):
Nv log p (a (0)
i arg min
Notably, P(yi is an explicit density function. To this end, we can obtain the empirical distribution of each class Merre
Sample through Distribution. With the estimated distribution for the source set, our goal is to find representative points for the distribution as proximal validation samples. In particular, we conduct
the process in a class-wise fashion. We generate angles via the explicit density function of the distribution to locate the corresponding samples in the auxiliary set for each class. Specifically, for a class y, we generate a set of angles by sampling from the distribution P(w)
Ay fa a ~ P(w)
(8)
For each angle in Ay we search for samples in Dawe with the smallest angular gap to construct proximal validation set Iarro
D {(x, arg min [x(((x)) al. Va E Ay (x.g)eDeus
where L. is the number of categories. At this point, we obtain distributionally representative Ippr with sufficient data. It can be employed directly for validation while remaining all the data in Thero for training, as shown in Figure And the entire process of PVP is shown in Algorithm [
4 EXPERIMENTAL EVALUATION
4.1 DATASET AND BASELINE
To validate our approaches, we experiment with three modalities of data: tabular, images, and natural language. All datasets statistics of all datasets are listed in Appendix A.3 And these datasets contain different levels of data volume from 400-20k.
classification tasks; notably, two are put in long-tailed setup. The
Tabular Data. We adopt 4 publicly available datasets from UCI (details in Appendix A.3 which also data sources applied in other works (Wang al. 2020 Bi & Zhang 2018 Moss et al. 2018

---

Under review as a conference paper at ICLR 2023
It is relatively easy to achieve stable and low-biased evaluations with balanced and sufficient data via pross-validttions which makes comparisons between methods less meaningful and persuasive. Therefore, series of datasets with various imbalanced ratios are picked.
Computer Vision. The common and challenging long-tailed distribution is chosen as our evalua- tion environment. And we follow previous works (Park al. 2021: Cui al.) 2019) to construct long-tailed versions of CIFARIO/ 10/100 with imbalance factor = 100 and 10 respectively, named as
CIFARIO/100--T; The details about the construction are illustrated in section A.3
Natural Language Process. We use the Reuters-21578 dataset (Dua & Graft 2017 according to the exact setting in JKFold (Moss al. 2018), where only corn and wheat categories are used.
Model. For tabular data, we use a decision-tree-basee model xghoost (Chen & Guestrin 2016) to perform classification tasks. While for CIFAR-LT and REUTERS, we train ResNet-44 (He et al.) 2016) and DistilBert (Sanh et al. 2019) as classifiers, respectively. Notice that we suppress ran- domness in the model by fixing random seeds and maintaining a consistent batch data feeding order to enable the dataset (train/val set) to be the unique variable. And more implementation details are listed Appendix A.4 including all training syper-parameters.
Method. Generally, three baselines are considered to compare against. First, holdout with train- val split ratio of 8:2. Despite it being widely used, especially in deep learning, owing to high efficiency, instability and deviation are its weak points due to the size of the validation set being still relatively small (e.g., near few-shot setup under hundreds of total samples). Second, k-fold CV where a choice 10 is taken for k. The instability can be improved versus holdout. Third, repeated k-fold CV (named as J-K-Fold (Moss et al., )) the repeat times and k are set to 4 and 5, respectively. Theoretically, it is further enhancement of stability at the cost of time. All these three methods are based on stratified sampling, which makes the percentages of different classes in both validation and train sets essentially the same. Additionally, we also add customized method on the single tabular scenario as an extra baseline, i.c., SPlit (Joseph & Vakayil 202I). SPlit is the latest splitting work, which utilizes support points (Mak & Joseph| 2018) to sample a specific subset from the train set as the validation set. And we set the sample ratio of SPlit the same as holdout (i.e., 8:2). Notably, the saper-parameters mentioned above (i.e., 8:2 for holdout, 10 for k-fold CV, and 4-5 for (-K-Fold) are found by selecting the one with the best validation quality on the three metrics. And we run baselines and PVP 5 times for the image and text datasets considering time complexity issues, while 100 times for each tabular dataset.
Metrics. To comprehensively evaluate validation methods, we propose to compare three metrics simultaneously, i.c., variance, T-V Gap, and scores the test set. (i) For variance, we calculate the standard deviation of estimated scores (scores on the validation set) over all runs. (ii) For T-V Gap, the mean of the absolute gap between scores the validation set and the test set overall runs, which is the same usage in previous work (Zeng & Martinez) 2000) [udka & Gabrys 2012). (iii) For test score, combo - high test score, small variance, and T-V Gap corresponds to good validation.
the mean of scores the test set over all rounds. Likewise, the following
4.2 COMPARISON TO OTHER METIIOSS
We report our results in Table [ We may conclude from the scores that: (i) The performances of models obtained by PVP are superior to all the rivals, and the improvement is significant on datasets, i.c. BankMarket, PageBlocks, Diabetes and CIFARIO-LT. (ii) The T-V Gap are consid- erably reduced compared to all competitors, which is reduced by 32.3% on average baselines. (iii) The variance is maintained between holdout and 10-Fold CV, and is closer to the latter, with higher evaluation stability.In nutshell, PVP ameliorates both the test score and the T-V Gap at the same time while keeping the variance within a competitive level. These main results effectively justify the feasibility of PVP towards a proximal, split-free validation setup.
the best
4.3 WHY DOES PVP IMPROVE MODEL PERFORMANCE?
Table T shows that the Fl score on all datasets is consistently improved compared all baselines. We would like to attribute the performance improvement to the extra training our framework, all the given data (i.e., source set) can be used for training, without the necessity to
data. Specifically, in
7

---

Under review as a conference paper at ICLR 2023
Table 1: Comparisons between PVP and other validation methods seven classification datasets covering all three major data modalities, including tabular data(top), image(bottom left) and text(bottom right). The results are reported in FI, T-V Gap and variance, i.c. Fl VTvv Gapt/Vari / indicates that the higher/lower the metric is, the better and vice versa. The numbers are scaled by 1e2 for straightforward comparison.). And bold indicates superior results.
METHOD
BankMarket
PageBlocks
Diabetes
MushRoom
Holdout
59.19.6/2.9
85.1 3.3/4.0
72.5 3.7/3.5
98.8 0.8/1.2
10-Fold CV
59.4 4.200.7
85.5 4.7/11
72.7 5.5/1.0
98.9 1.4/0.2
4-5-Fold CV
59.1 3.3/0.4
85.9 3.4/0.6
72.8 4.0/0.6
98.9 0.8/0.1
SPlit
59.6 3.3/2.9
86.6 2.6/2.9
73.5
98.9 0.7/0.8
PVP(ours)
61.3 2.6/0.8
89.0 1.2/133
77.4 3.//1.
99.0 0.5/0.6
METHOD
CIFARIO-LT
CIFARI00-LT
METHOD
REUTERS
Holdout
61.2 8.4/233
47.0 1./1122
Holdout
88.7 7.2/2.1
10-Fold CV
63.3 7.7/0.6
49.9 188/0.4
10-Fold CV
89.5 7.1/0.5
4-5-Fold CV
61.1 7.4/0.4
47.4 1,4002
4-5-Fold CV
88.9 6.9/03
PVP(ours)
67.3 18/0.9
50.5 10/0.8
PVP(ours)
90.0 68/0.4
Table 2: Comparison of our PVP and other methods on distribution gap. Reported in Wasserstein distance over 10 times. (Tabular data(top), image(bottom left) and text(bottom right).)
METHOD
BankMarket
PageBlocks
Diabetes
MushRoom
Randomt Holdout)
1.82 0.27
9.54 + 1.32
6.91 0.62
2.46 + 0.37 1.91 = 0.13
SPlit
1.75 + 0.20
7.82 = 0.73
5.87 + 0.63
PVP(ours)
1.56 0.16
5.41 + 0.69
3.90 0.72
1.85 = 0.21
METHOD
CIFARIO-LT
CIFARIO0-LT
METHOD
REUTERS
Random/HHldoun
2.20 0.14
1.85 + 0.03 1,57 + 0.03
Random Holdout) PVP(ours)
3.02 - 0.37 2.96 1 0.12
PVP(ours)
1,51 = 0.10
partition some them as the validation set. Owing to the split-free nature, PVP can save data for training to provide performance gains compared to the split-relied methods.
4.4 PROXIMAL VALIDATION SET VS TRADITIONAL VALIDATION SET
We compare proximal validation set with other validation sets in both quantitative and qualitative aspects: distribution gap and visualization difference. These two aspects demonstrate why the prox- imal validation
work better than the traditional one.
Proximal validation set has smaller distribution gap. We compared the gap between the distri- bution of the validation set and the global distribution (approximated on source set) under different methods. The distribution is quantified via the intra-class angular distribution (as mentioned in Sec- tion 3.2.3) and the distance two distribution is calculated by Wasserstein distance (Takatsu) 2008). As shown in Table the distribution gaps of PVP are consistently smaller than the other methods, empirically verifies that our validation set is better representation of the global distribution, which can also lead to smaller 7-V Gap in evaluation since Deal is more approximated to F in Eq
Proximal validation set are well spread out. We visualize the data representation produced by t-SNE in Figure 3 We use the MushRoom dataset, and different colors represent different ground- truth class labels. We contrast the TSNE embeddings of the validation set produced by two ap- proaches: (a) SPlit (with split ratio 8:2), current best split-based methods, and (b) our method PVP. Compared with SPlit, we can observe that our validation points cover a wider region containing both central and marginal areas. Since PVP are not constrained by the split scheme, the number of validation samples in the proximal validation set can be larger and the coverage can broader, which is presumed to lead to better evaluation quality.
8

---

Under review as a conference paper at ICLR 2023
dimersion
class C (source)
class 2 (source)
L-SNE dineensiion
(sourcel (source!
rlans tow0 Iva0
boel tvall
L-SNE dimeessoo
USE dimension
(a) Validation Set of SPlit
(b) Proximal Validation Set of PVP (ours)
TSNE visualization of the validation set representation on MushRoom. Different colors represent the different classes. The translucent points represent the source set, while the solid- colored ones represent the validation set. The details are referred to in Section 4.4
Figure 3:
Table 3: Using proximal validation set with- out extra training samples. The improvement of results obtained by replacing the origi- nal split validation set with ours. The implies deterioration, while the others the default for improvement.
Table Ablation Study. The deteriora- tion of results obtained by using random sampling (from auxiliary set) instead of our distributional-conisttt sampling. The im- plics improvement while the others are default for deterioation.
DATASET
T-V Gap(+)
Var(+)
FI(+)
DATASET
T-V Gap(-)
Var(-)
FI(-)
BankMarket
1.39
1.40
0.06
BonkMarket
0.30
2.20
0.21
PageBlocks
0.67
1.76
1.01
PageBlocks
3.50
+0.12
0.05
Diabetes
1.54
2.06
1.65
Diabetes
3.35
0.22
1.23
MushRoom
0.06
0.52
0.01
MushRoom
0.06
+0.09
0.00
CIFARIO-LT
3.73
0.28
0.21
CIFARIO-LT
22.87
+0.05
1.23
CIFARIOO-LT
1.00
0.17
0.00
CIFARIO0-LT
21.01
0.06
0.44
REUTERS
-0.70
1.35
0.20
REUTERS
5.16
0.33
2.30
4.5
USING PROXIMAL VALIDATION SET WITHOUT EXTRA TRAINING SAMPLES
We conduct experiments using the same training set as the holdout's rather than the whole training set while replacing the original split validation set of the holdout with our proximal validation set. Table [] shows that even without the extra training data, the performance some degree. This implies that the improvement in performance comes not only from additional training data but also from the high-quality validation set itself. Besides, we can also see that the T-V Gap of our method is still smaller in most cases when the validation set is the only different element. This comparison further confirms the superiority of our method in evaluation precision.
still be enhanced to
4.6 ABLATION STUDY
Sampling. In this section, we attempt to reveal the effec- tiveness of the core algorithm PVP. We compare PVP with a variant: PVP-random, which uses (stratified) random sampling from auxiliary set instead of distributional-consittee sampling. From Table we can observe that the T-V Gap of PVP-random are aggravated, especially for the text and image datasets using deep models. Moreover, the test scores also degrade most datasets. It signifies the indispensability of our sampling algorithm in the entire framework.
Effects of Distributional-Conistee
LIMITATION AND CONCLUSION
In this article, we proposed Proximal Validation Protocol (PVP) fully resolve the train/validation split trade-off. Through extensive experiments datasets in different fields, including seven pub- licly datasets covering tabular, image and text data, we justify the comprehensive validity of PVP from both performance, robustness and reliability perspectives.
Indeed, we have chosen to leave the theoretical understanding of this framework out of the scope of this paper. Mostly, it is due to general lack of theoretical study in this specific line of research around validation set construction. Rather, we intend to empirically prove this concept of split-fiee free proximal validation with the simplest instantiation in this work. We hope to motivate the cohorts in the community to pay more attention to this research line because we believe vital and widely existing for MI production.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sylvain Arlot and Alain Celisse. A survey of cross-validation procedures for model selection. Statis- tics surveys, 4:40-79, 2010.
Jingjun and Chongsheng Zhang. empirical comparison on state-of-the-art multi-class im- balance learning algorithms and new diversified ensemble learning scheme. Showledge-Basedd Systems, 158:81-93, 2018.
Marcin Budka and Bogdan Gabrys. Density-presseving sampling: robust and efficient alternative to cross-validation for error estimation. IEEE transactions on neural networks and learning systems, 24(1):22-34, 2012.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113, 2019.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Yin Cui, Menglin Jia, Tsung-Yi Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattem recognition, pp. 9268-9277, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL htth:::://archhi ics. srcii.ddu/mg
Seymour Geisser. The predictive sample reuse method with applications. Journal of the American statistical Association, 70(350):320-328 1975.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Zhuoxun He, Lingxi Xie, Xin Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. Data augmentation revisited: Rethinking the distribution gap between clean and augmented data. arXiv preprint 1909.09 2019.
Gaoxia Jiang and Wenjian Wang. Eror estimation based on variance analysis of k-fold cross- validation. Pattern Recognition, 69:94-106, 2017.
Roshan Joseph. Optimal ratio for data splitting. Statistical Analysis and Data Mining: The ASA Data Science Journal, 2022.
Roshan Joseph and Akhil Vakayil. Split: An optimal method for data splitting. Technometrics, pp. 1-11, 2021.
Yoonsuh Jung. Multiple predicting k-fold cross-validation for model selection. Journal of Nonpara- metric Statistics, 30(1):197-215, 2018.
Takumi Kobayashi. T-vmf similarity for regularizing intra-class feature distribution. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6616 6-666655, 2021.
10

---

Under review as a conference paper at ICLR 2023
Ron Kohavi al. study of cross-validation and bootstrap for accuracy estimation and model selection. In Ijcai, volume 14, pp. 1137-1145. Montreal, Canada, 1995.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Selmer C Larson. The shrinkage of the coefficient of multiple correlation. Journal of Educational Psychology, 22(1):45, 1931.
Weikai Li, Chuanxing Geng, and Songean Chen. Leave zero out: Towards a no-cross-validation approach for model selection. arXiv preprint arXiv:2012.13309 2020.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. Deep representation learning on long-tailed data: learnable embedding augmentation perspective. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 1970-2979, 2020.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 212-220, 2017.
Simon Mak and V Roshan Joseph. Support points. The Annals of Statistics, 46(6A))2562-2533) 2018.
Henry Moss, David Leslie, and Paul Rayson. Using jk fold cross validation to reduce variance when tuning nlp models. arXiv preprint arXiv: 1806.00666000 07139, 2018.
F. Mosteller and J. W. Tukey. Data analysis, including statistics. In G. Lindzey and E. Aronson (eds.), Handbook of Social Psychology, Vol. 2. Addison- Wesley, 1968.
Seulki Park, Jongin Lim, Younghan Jeon, and Jin Young Choi. Influence-balanced loss for im- balanced visual classification. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 735-744, 2021.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 2019.
David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, -rannorascois Crespo, and Dan Dennison. Hidden technical debt in machine learning systems. Advances in neural information processing systems, 28, 2015.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Mervyn Stone. Cross- validataly choice and assessment of statistical predictions. Journal of the royal statistical society: Series B (Methodological), 36(2):1 11-133, 1974.
Gábor J Székely and Maria L Rizzo. Energy statistics: A class of statisties based on distances. Journal of statistical planning and inference, 1249 2013.
Asuka Takatsu. On wasserstein geometry the space gaussian measures. arXiv preprint orXiv:0801.22500 2008
Henri Tiittanen, Liisa Holm, and Petri Törönen. A new approach to multilabel stratified cross valida- tion with application to large and sparse gene ontology datasets. arXiv preprint arXiv:2109.01425, 2021.
Chen Wang, Chengyuan Deng. and Suzhen Wang. imbalance-xgboos: leveraging weighted and focal losses for binary label-imbalanced classification with xgboost. Pattern Recognition Letters, 136: 197- 2020.
11

---

Under review as a conference paper at ICLR 2023
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.1090 2018.
Mingle Xu, Sook Yoon, Alvaro Fuentes, and Dong Sun Park. A comprehensive survey of image augmentation techniques for deep learning. arXiv preprint arXiv:2205.01441, 2022.
Xinehuan Zeng and Tony R Martinez. Ietrribution-bllanee stratified cross-validation for accuracy estimation. Journal of Experimental & Theoretical Artificial Intelligence, 12(1):1-12, 2000.
Xi Zhang, Yanwei Fu, Andi Zang, Leonid Sigal, and Gady Agam. Learning classifiers from synthetic data using a multichannel autoencoder. arXiv preprint arXiv:1503.03166, 2015.
Jing Zhou, Yanan Zheng, Jie Tang, Jian Li, and Zhilin Yang. Flipda: Effective and robust data augmentation for fho-shot learning. arXiv preprint arXiv:2108.06332 2021.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 METHODS FOR GENERATING AUXILIARY SET
We list the detailed augmentation methods for each dataset in Table 5
augmentation methods used for each dataset. B( b) denotes a beta distribution with parameters and b. Times is the number of augmentation method performed per sample. (For Reuters, we utilize nlpaug to implement these operations.)
Table 5: The list
DATASET
Method
Parameters
Times
BankMarker
MixUp
~ B00.5, the mixing ratio
30
Diabetes
MixUp
À B(0.5, 0.5):the mixing ratio
30
Paoe
Smote
N 5:the number of neighbors
30
MushRoom
Smote
N 5:the number of neighbors
30
GridMask
d1=24, d2=33, rotate=1, ratio=0.4
5
Invert
CIFAR-LT
Solarize
E (256, 192, 128, 0) :olarizaatin degree
Brightness
E (1.9, 1.35, 0.95, 0.1 0.1]:an enhancing factor
ContextualWoxdEEbbuuubiiiiie
E (0.3, 0.5) caugmentatin proportion
ContextualWordEmmmae
PE [0.3, 0.5) :augmentation proportion
SynonymAug
P [0.3, :augmentttin proportion
RandomtordAuggellett
p {0.3, :augmenation proportion
RandomWordAugCCnn
p P {0.3, 0.5) :augmentation proportion
REUTERS
RandomWordAugSwe
P E {0.3, 0.5) caugmentation proportion
Back TractrransationnAuu
AbstSummAAu
WordEmbssAggSussiiun
p (0.3, 0.5) :augmentation proportion
WordEnbsAuulnnert
p (0.3, 0.5) :augmentation proportion
A.2 PSEUDO CODE FOR PROXIMAL VALIDATION PROTOCOL
Algorithm Proximal Validation Protocol
Input: Source set Dare, Number of category L, Classification model f. Scoring function
5. Feature extractor O, Number of samples m. Data augmentation methods set (0: 1
G {91. :...9.....900 .Q}
Mour = G (I) Initiate Darr 0
for = to L do
Assume p is gaussian distribution
0 arg ming
E Exxx,) log (2((())) ()
A. {a= az; ~ p((x 0))m
for E A do
Dero pero U arg migmmn [a(((x)) a.l
end for end for
Train of on D to obtain fr with highest 8(f, Dero)
Output: f*, $(f*, Dero)
https://nlpuggrraatthhhee
https://llpagg..ratttteeess
13

---

Under review as a conference paper at ICLR 2023
A.3 DATASET STATISTICS
Construction details of the long-tailed dataset. As mentioned in Section 4.1 we construct long- tailed versions of CIFARI0/100 with imbalance factor p 100 and 10 following Park et al. (2021); Cui et al. (2019). Specifically, for a k-class dataset, we create a long-tailed dataset by reducing the number of examples per class according to the exponential function n' n. ( E (0, 1), 0, 0, k), where is the original number of examples for class i, while ni is the new number.
Table 6: Dataset statistics. P denotes the imbalance factor, which is the ratio of the number of samples in the largest class to the smallest class. #C' denotes the number of classes. #F denotes the number of feature dimensions. #Source, #Test, #Auxiliary, and #Proximal(Ours) denote the number of samples in source set, test set, auxiliary set, and our proximal validation set, respectively.
DATASET
#Data
#C
#F
#Source
#Test
#Auxiliary
#Proximal
BankMarket
3047
2
7
13
2437
610
73110
2367
PageBlocks
5473
5
170
10
4378
1095
131340
3539
Diabetes
718
2
2
8
574
144
17220
1720
MushRoom
8124
22
406
7718
12180
207
CIPARIO-LT
22406
100
1024
12406
10000
210902
2980
CIFARIOO-LT
29573
100
10
1024
19573
10000
332741
3000
REUTERS
520
2
393
127
5537
944
A.4 IMPLEMENTATION DETAILS
Tabular Data. We used xghoost (Chen & Guestrin 2016) as classifier for tabular datasets. Specifically, we used the XCBClassifie with parameters: objective of 'multi:sofprrb'' booster of 'gbtree', early stopping rounds of 15, max depth of 4, learning rate of 0. estimators 50. While for the features of tabular data, conduct some transformations features. The transformations contains two types: (i) for categorical feature, we use convert it to one- hot feature; (ii) for other types of features, we convert them be normalized. And the dimension of feature varies among datasets.
the original sample generate
CIFARIO-LT and CIFAR100-LL1 We used ResNet44 from He al. (2016) as the classification model for both CIFAR10-L3 and CIFAR 100-IT. We followed the same training procedure (train from scratch), initialization, and hyperparameters as He al. (2016)(i.e., weight decay of 0.0001, momentum of 0.9, minibatch size of 128, total epochs of 200, initial learning rate of 0.1, learning rate decay stage of 100 and 150 epochs). While for the feature extractor, we used ResNet20 from He et al. 2016)). And we train the extractor on source set for each dataset with same training hyperparameters as above. And the output of the last layer before classification is used as features, whose dimension is 72.
Reuters. We used DistilBert (Sanh et al. 2019) as classification model. We finetuned the pre-train model with training parameters: total epochs of 15, batch size of 8, weight decay of 0.01, learning rate 2c-5, While for the feature extractor, we used Bert from Devlin et al. 2018). And we train the extractor source set with same training hyperparameters as above. And the output in the "cls' position of the last layer before classification is used as features, whose dimension is 144.
A.5 ANALYSIS OF TIME COST
As shown in Table we provided some quantitative measurements toward the running time of the proximal validation set construction process. The running time can be devided into two parts: (i) the pre-processing part, which only needs to executed once, regardless of the number of training and validation process. This part contains the generation of auxiliary set (Section 3.2.2 and the estimation(Section (32.3). (ii) the sampling part, i.e., the second step in distributional- consistent sampling algorithm (Section 3.2.3), the execution number of which is consistent with training process. Indeed, the time cost of part one is high majorly due to the generation process.
14

---

Under review as a conference paper at ICLR 2023
However, the operations in part only need to be conduct only matter how many times we perform train and validation on the same dataset. While the operation included in every training process is the sampling part, which can be finished in few seconds and therefore does not incur much additional time cost for the training process.
On the other hand, position that the time cost of construction process be attributed minor point and the reasons are two folds: (i)-this process can be faster via some engineering efforts like multithreading technology. (ii)-it can actually be viewed (better model performance and evaluation quality) at the expense of time, and such a strategy is common in many works, such as AutoAugment (Cubuk et al., 2019), RandAugment (Cubuk al. 2020), FlipDA (Zhou et al. 2021).
strategy achieve certain goal
Table 7: Time cost of PVP.(s denotes seconds and m denotes minute.)
ONE-TIME
EVERY-TIME
DATASET
Generation(s/m)
(stimation(s//m)
Sampling(s)
Training(s/m)
BankMarket
65s
13s
3.0s
4.0s
PageBlocks
91s
18s
8.0s
9.5x
Diabetes
15s
10s
0.5s
0.6x
MushRoom
7s
11s
0.1s
0.2s
CIFARIO-LT
13m
2m
2.5s
30.2m
CIFARI0O-LT
1 11m
3m
2.2s
44. 1m
REUTERS
120m
Im
4.8s
3.0m
A.6 A TEST CASE VALIDATION METHODS SYNTHETIC DATASETS.
Here, we test two standard validation methods on five synthetic datasets (created by difeaarndatasets..ne
classificatior train size of 1000, test size of 1000, class and features of 2) with increasing class imbalance in Figure 4 In this case, we can observe that these methods can work well in ideal scenarios with balanced data. contrast, there is a risk of an imprecise estimation failure under unbalanced and small-scale scenarios, where the discrepancy between the test performance and the estimated one (i.e. the height of the yellow region) is undesirable large. This case illustrates that standard validation schemes do not always work perfectly in simple synthesis scenario, thus we can not easily rely on these methods in complex scenarios in the real world.
433
007
6.5
Figure Holdout and 10 fold cross validation on synthetic dataset.
aee.eeaeiieiiiieememmmmmmmnnnonn Mlssification.hma
15


Reviews:
[{"review_id": "tOXn1j_fH0", "reviewer": "Reviewer_sKMx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is a new validation protocol for classification tasks to address the fundamental problem of conventional training/valiation approaches by separating validation data from training data, called proximal validation protocol (PVP). For achieving this, the authors present a method to construct proximal valiation set by employing data augmentation and a sampling strategy basd on angular distribution. \nTo show the robustness of the PVP, they introduce two evaluation metrics: variance and T-V gap. \nThe authors validate their protocol on three data domains such as text, image, and tabular data, comparing with the standard valiation protocols such as hold-out and k-fold CV. The results seem meaningful. \n ", "summary_of_the_review": "Overall, I like this paper and agree the importance of the topics this paper addresses. However, some limitations exist in experiments. \nSo I decided my score as borderline . If the authors address my major concerns, I am willing to raise my score.", "strength_and_weaknesses": "### Strength\n- The problem to be addressed is both practical and fundamental for ML community. When ML model is developed for real-world application in industry, it is quite challenging to make a validation set proper, fair, and effective. \n- The proposed idea looks simple but effective using data augmentation and angular-based sampling. \n- The experimental results present the efficacy of the proposed PVP against the conventional validation protocols in terms of three metrics.\n- The paper is easy to read and understand.\n\n### Weakness\nI agree that this paper addresses a significant, practical and fundamental problem, so I think this paper can contribute to ML community. But, there are some room for improvements, in particular, of experiments. \n- [Major] Considering that the method for proximal validation set leverages augmentation methods, classification tasks can benefit from this protocol. that is, multiple tasks such object detection, semantic segmentation in computer vision and QA in NLP might be diffult to be applied. I think the authors need to discuss or clarify this point. \n- [Major] I think more experiments on industrial or real-world classification setups are required because the authors emphasize their motivation is from the debt of ML in industry. For example,\n  - Most image recognition tasks leverage fine-tuning from pretrained models. Even if it is meaningful to the results on CIFAR-LT that the authors presented, more results on small-scale data such as Oxford-car/flower using ImageNet-pretrained will improve the contribution. Also, noisy label scenario experiments will be helpful.\n  - Why did the authors use Distill-BERT instead of BERT for a classification model?\n- [Major] More and intensive abation studies are requried for important hyperparameters to support their hypothesis because the authors do not present theoretical justification.\n  - The size or sampling ratio of proximal valiation set from auxiliary set. It is not clear how to decide the size. \n  - Because the authors argue that PVP is independent on model and optimizers, they need to present the evidence such as more experiments on one or two different models and an addtional optimizer. \n- [Major] The authors should present statistical significance results in Table 1 and 2 at least, considering that this paper address evaluation protocol. \n- [Minor] How is the performance under the below setup? This is also a alternative to address the lack of training data caused by validation set. Of coures, this cannot address the promblem of how to make a good validation set. \n   - make a training (T) and valiation (V) set by spliting validation data  from source training set\n   - make a new training set (T') merging training set (excluding valiation set) and the proximal validation set\n   - Train T' and validate (V)\n- [Minor] Tables 3 and 4 can be improved for better readability. For example, the authors use colored arrows (up-red, down-blue).\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qbmKjVa3vTy", "reviewer": "Reviewer_iGVu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper identifies the problem of 'split tradeoff', where common train/validation splits for data result in either less data to train on or poorer model evaluation. Instead, the paper introduces Proximal Validation Protocol (PVP), which, instead of splitting the validation set from a train set, creates a validation set through a combination of: 1) data augmentation on the train data and 2) a devised sampling scheme to preserve the distribution of the augmented validation set to be close to the train set. PVP allows one to use the whole training data in order to train models while still having a useful validation set for model evaluation, thereby avoiding the split tradeoff problem. \n\nExperiments on three data modalities: tabular, image and natural language seem to demonstrate that PVP improves both the test performance of trained models, as well as reducing the gap between test and validation performance (i.e. validation is a better evaluation of model performance), whilst maintaining a competitive variance across different runs of their method (with different validation sets).", "summary_of_the_review": "I am recommending weak reject. The problem identified is important and the proposed solution appears novel, but the experiments are both unclear and (as far as I understand) lacking in thoroughness, as detailed in my weakness section. I'm happy to be corrected though.", "strength_and_weaknesses": "Strengths:\n1. The paper tackles a clear issue (how best to choose a validation set) which has received little attention in the ML literature.\n2. The proposed scheme, using data augmentation to generate a validation set, appears to be novel and goes some way to solving the issue of 'split tradeoff' problem.\n\nWeaknesses:\n1. The paper seems to only use compare validation sets in terms of how well they evaluate the performance of a *fixed model* (the hyperparameters/architectures in section A.4 all appear to be fixed and pre-defined), whereas the purpose of validation sets for model evaluation in ML, to me, is to be able to select between different models/hyperparameters. I think it is very important for the contribution of this paper to demonstrate having better model evaluation via PVP's validation set does actually lead to better model selection e.g. better hyperparameter tuning. If this is already shown in the experiments then it should be made clearer/explicit.\n\n2. More generally, the paper is not presented clearly, at least to me. This is particularly the case in the experiments section, which is particularly problematic given that the paper focuses on showing the empirical strengths of PVP, rather than theoretical justification. This makes it hard to give a proper evaluation of the paper's contributions.\n\nTo start with, the 'F1' score is not properly defined, there is some alluding to it being the (top-1?) test accuracy in definition 3.1, which is what I take the definition to be (correct me if wrong). In any case, it is not clear what the values in table 3 and 4 represent: the word 'deterioration' alludes (but not in a clear way, and should be properly defined in any case) to the *difference* in each metric between using a proximal validation set vs. standard validation set (in table 3), and using random sampling vs. distributional-consistent sampling (in table 4). The reason why this lack of clarity is confusing is that if I understand correctly, there should be no difference between the F1 scores (test accuracies) in the settings presented in tables 3/4. This is because the choice of validation set shouldn't affect the test performance (for a fixed model trained on the same training data, see point 1 above), and hence that column should read as 0.\n\n3. I'd also like to see more thorough empirical evaluation, e.g. to what extent does PVP depend on the choice of data augmentation. What happens to performance if we do just say additive Gaussian noise as our data augmentation? There are many choices that one can make for the data augmentation: is the empirical performance of PVP robust to this choice? Likewise, there seems to be a non-standard softmax-like weighting using for the class center in equation 6, is there a reason for this? Can the authors provide an ablation to this choice of weighting (say to just a standard uniform weighting)?\n\n4. PVP seems to be designed for classification settings (particularly the distribution-preserving sampling), does it also extend to other settings say regression, or non-supervised learning regimes like self-supervised or unsupervised?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dqJYb7DJRdm", "reviewer": "Reviewer_wMBi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In order to maximize the usage of the training set, this paper provides an alternative validation set construction strategy. Specifically, they construct an auxiliary dataset by augmenting the original training set. Then, they pick examples from the constructed auxiliary dataset. The new strategy is evaluated on several datasets.", "summary_of_the_review": "This paper proposes and interesting validation set construction strategy, but the current version lacks enough empirical and theoretical supports.", "strength_and_weaknesses": "In general, this paper looks interesting to me. However, I have concerns about the empirical evaluation and theoretical guarantees. \n\nEmpirically, the proposed strategy is evaluated on several tabular datasets, image datasets, and one text dataset. However, most of these datasets are very small, which can not fully support the claims. For example, the authors use CIFAR10-LT and CIFAR-100-LT, I am wondering why the authors don't use the original CIFAR dataset or ImageNet. A similar issue also happens in the text data. \n\nTheoretically, though the authors mention the theoretical analysis in the limitation. I still would like to see some theoretical results to support the claims in this paper, esp. when they only evaluate the proposed strategy on tiny datasets.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Yj173rFvOE", "reviewer": "Reviewer_cmsD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors propose a new construction of validation set from training set without splitting the training data.  Authors construct the validation set in two steps:\n\n- Data augmentation: Apply different data augmentation techniques, e.g. for image data, authors use GridMask, Invert, Solarize and Brightness, to create data generating procedure from the source dataset. Use this augmentation as a new data generator to consturct an auxiliary data source.\n- Data selection from the auxiliary data: Authors select the data by choosing points that are close to the source data. Closeness is measured by using feature extractors, e.g. for image data authors use a pre-trained ResNet18 and text data authors use BERT.\n\nAuthors then empirically show that this validation procedure leads to higher accuracy and lower test-validation gap and smaller variance on different tasks.", "summary_of_the_review": "I recommend rejecting the paper. I think there’s no reason to believe that the proximal set constructed is actually representative of the data generating distribution and thus should not be used as a validation set. ", "strength_and_weaknesses": "Strengths:\n\n- Motivation for constructing new dataset is clearly presented.\n- Method to construct a validation set is easy to understand\n\nWeaknesses:\n\n- Approach lacks theoretical basis. There is no reason to believe why the validation set is representative of the data generating distribution\n- The metrics used for evaluation are not convincing to show the validation set is representative of test distribution", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject