Question: Is the regularization coefficient set to 10^{-4} for all the methods? If so, do authors have an argument why this parameter shouldn't be tuned for every method separately?
Evidence: We evaluate Kernel SGD on three types of data: tabular, image, and text. Table shows the details of these data sets. The tabular data sets were downloaded from LIBSVM website 1 . The dimensions of mnist and usps are 780 and 256, respectively. We used cifar10 and SARS-CoV-2 (S-CoV) as the image data. S-CoV contains RGB CT scans and is used to identify whether the patient is infected by the virus. Each image in the two data sets was first reshaped into 32 × 32 image and then normalized according to the mean and variance sampled, respectively. We used IMDb and COVID-19-tweets (COV-tw) as the text data. COV-tw collects the English Tweets about COVID-19 and is labeled as informative or not. We took 20% of the training data to serve as the validation set except for COV-tw. In COV-tw, 1000 validation instances are provided. On tabular data, our method was evaluated using a simple two-linear-layer neural network which has 100 neurons in each hidden layer. For image classification, our method was tested on the ResNet-18 network. To solve text classification problems, we used an LSTM network with pre-trained word vectors. Baselines: Our method was compared with the widely used L-BFGS and Equilibrated SGD (ESGD) optimization methods which both use the second-order information in training. ESGD adopts an equilibration preconditioner to reduce the condition number and escape saddle points. We used the implementation of ESGD published by.
The experiments were conducted on a machine with an Intel(R) Xeon(R) Silver 4210 CPU of 126 GB memory and two GeForce RTX 3090 GPUs running on a Linux OS. All the methods including the baselines and our method were implemented using PyTorch. The size of history used in L-BFGS was set as 3 for limited memory. The mini-batch size was set to 64. L 2 regularization was used with its coefficient set to 10 −4 following the setting in the studies. The same regularization coefficient for all the methods leads to a fair comparison. In Kernel SGD, to compute the kernel matrix, we used the radial basis function kernel (i.e., k(x i , x j ) = exp(−γ||x i − x j || 2 2 )). Through the whole experiment, the learning rate and the hyper-parameter γ for the kernel were selected from {0.1, 0.01, 0.001, 0.0001}. We terminated the training when the convergence conditions were satisfied, i.e., the change of loss is less than 1 × 10 −4 in 3 consecutive epochs or the training reaches 500 epochs. We repeated each experiment five times to acquire average performance.
L2 regularization was used with its coefficient set to 10−4 following the setting in the studies (He et al., 2016; Huang et al., 2017). The same regularization coefficient for all the methods leads to a fair comparison.
Answer: The regularization coefficient was set to 10^{-4} for all the methods following the setting in the popular studies [1,2]. The same coefficient for all the methods leads to a fair comparison, because different regularization coefficients lead to different objectives, which makes different methods incomparable. We have added this explanation in the last paragraph of Section 4.1.