Lottery Ticket Structured Node Pruning for Tabular Datasets
Authors: Ryan Bluteau
Robin Gras
Mitchel Paulin
Zachary Innes
Venue: ICLR 2022 Submitted (2022)
Keywords: Lottery Ticket Hypothesis
Tabular
Pruning

Abstract: In this paper we presented two pruning approaches on tabular neural networks based on the lottery ticket hypothesis that went beyond masking nodes by resizing the models accordingly. We showed top performing models in 6 of 8 datasets tested in terms of F1/RMSE. We also showed in 6 of 8 datasets a total reduction of over 85% of nodes and many over 98% reduced with minimal affect to accuracy. In one dataset the model reached a total size of one node per layer while still improving RMSE compared to the larger model used for pruning. We presented results for two approaches, iterative pruning using two styles, and oneshot pruning. Iterative pruning gradually reduces nodes in each layers based on norm pruning until we reach the smallest state, while oneshot will prune the model directly to the smallest state. We showed that the iterative approach will obtain the best result more consistently than oneshot.

Under review as a conference paper at ICLR 2022
LOTTERY TICKET STRUCTURED NODE PRUNING FOR TABULAR DATASETS
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper we presented two pruning approaches on tabular neural networks based on the lottery ticket hypothesis that went beyond masking nodes by resiz- ing the models accordingly. We showed top performing models in 6 of 8 datasets tested in terms of FI/RMSE. We also showed in of 8 datasets a total reduction of over 85% of nodes and many over 98% reduced with minimal affect accu- racy. In one dataset the model reached total size one node per layer while still improving RMSE compared to the larger model used for pruning. We pre- sented results for two approaches, iterative pruning using two styles, and oneshot pruning. Iterative pruning gradually reduces nodes in each layers based norm pruning until we reach the smallest state, while oneshot will prune the model di- rectly the smallest state. We showed that the iterative approach will obtain the best result more consistently than oneshot.
INTRODUCTION
has been understood for while that pruning a network after training can lead much smaller networks that still maintains similar accuracy to the original network. However the question arises, these smaller networks perform just as well as the larger ones, could one not train the smaller network architecture to begin with?
The lottery ticket hypothesis states, "a randomly-initializz network that is initialized such that when trained in isolation it can match the test accuracy of the original network after training for at most the same number of iterations" Frankle & Carbin (2019). These smaller networks are referred to "winning tickets".. The process of finding winning tickets was proposed to start with an initial set of weights, train the network, prune the network, and finally reset the remaining weights back to their initial state.
dense neural network contains a sub-
It has been shown that these winning tickets perform as well, or sometimes better and require less time train when compared the original network. In this paper we focus applying this hy- pothesis to tabular neural networks for several datasets to generate smaller networks with similar performance.
We use tabular neural networks from FastAl Howard et al. (2018) which work on the principle of converting categorical features into embeddings. The models come with strong features such as normalization of continuous features, filling in missing data, and converting dates into categorical features. The models can be trained with their cycle training approach which raises and lowers the learning rate to find the best local minimum while training.
There are many small tabular datasets which often don't 't get optimized research. These datasets filled with categorical features and missing entries while often having few samples to test. In this paper we use the lottery ticket hypothesis to improve FastAl's tabular neural network to easily prune models for any tabular dataset. We test a wide range of datasets from large small including different ranges of categorical and continuous features.
This paper is organized as follows. Section describes related previous works. Section 3 presents the datasets used and our pruming methodology. Experiments, results and comparisons are presented in Section 4. Finally, the paper is concluded in Section

---

Under review as a conference paper at ICLR 2022
2 PREVIOUS WORKS
The lottery ticket hypothesis was first tested by Frankle & Carbin (2019) on dense neural networks and convolutional neural networks for MNIST and CIFARIO. The authors achieved networks 80- 90% smaller while both creating faster networks and increasing the accuracy of the models in com- parison to the original sized models. The authors show that while pruning has been around for a long time, pruning networks often lead to more difficult training with less accuracy. By finding the lottery tickets of a network, difficulty in training of smaller networks is reduced while accuracy can be maintained and even exceed the original model.
Since its introduction, there have been many papers testing the lottery ticket hypothesis on different architectures. Morcos et al. (2019) look to find winning tickets in networks by searching for com- mon winning tickets across many datasets. They test on a range of image datasets such as Fashion MNIST, SVHN, CIFAR- 10/100, ImageNet, and Places365. Their findings were that winning tickets generalized over many datasets, but interestingly the larger datasets produced better winning tickets that were more generalizable than those from smaller datasets. Girish et al. (2021) use the lottery ticket hypotheses for object detection achieving a sparsity of up to 80%.
Chen al. (2021) test the lottery ticket hypothesis ever growing graph neural networks (GNN). For node classification tasks, they show a decrease in multiply-accumuble operations (MACs) of up to 98% while having sparsity of nearly 98% with minimal affect to performance. For linking predictions, they show similar sparsity with a large reduction in MACs.
The field of natural language processing (NLP) has recently exploded with new models since the in- troduction of the transformer architecture Vaswani et al. (2017). The models grow increasingly large, from sizes like the popular BERT Devlin et al. (2018) architecture of up 350 million parameters to sizes like GPT-3 Brown et al. (2020) reaching up to 175 billion parameters. Chen et al. (2020) prune the BERT architecture, but they begin from the pretrained state and test the lottery ticket hypothesis using downstream tasks. The paper only applies unstructured pruning stating "since we perform a scientific study of the lottery ticket hypothesis rather than an applied effort to gain speedups on specific platform, we use general-purpose unstructured pruning" Chen et al. (2020). With un- structured pruning, they are able to find networks with sparsity of 40-90% while testing a range of downstream tasks.
Tabular models are heading towards larger networks as well with the recent transformer architecture making its way to tabular data. Padhi et al. (2021) proposed time series BERT architecture for tabular data called TabBERT and a tabular data generator TabGPT based on the GPT architecture. Gu & Budhkar (2021) use transformers convert text in tabular data into features and released package to process the data. TabTransformer by Huang et al. (2020) uses the transformer architecture to achieve new SOTAS and tested on 15 datasets, competing well with ensemble approaches.
In this paper we focus on tabular dense neural networks with the goal to prune them as much as possible while maintaining accuracy utilizing the lottery ticket hypothesis. We test our models on datasets of different varieties and show competing results for datasets with available comparisons.
3 DATASETS
In this section we describe the tabular datasets used in our experiments. We selected a wide range of datasets from few hundred samples to millions. Each dataset has a variety of tabular features, some containing just continuous features, others containing only categorical features, and many mixed with both. The datasets are described in more detail in this section and we present the num- ber of samples, number of features, evaluation metrics, and vain-validationntestt splits used in our experiments in Table
The first dataset is the Alcohol dataset Cortez & Silva (2008). It is our smallest dataset and con- tains the most features with mix of continuous and categorical features. The dataset contains information on students such as school, gender, age, information like hobbies and goals, their fam- ily and related information such as work, education, size, etc. As quoted from this recent dataset survey Mihaescu & Popescu (2021), "This dataset has also been uploaded Kaggle where 305 publicly available kernels perform exploratory data analysis. Unfortunately, there is not defined any task with specific validation metric such that there is no leaderboard publicly available", so we used
2

---

Under review as a conference paper at ICLR 2022
Table 1: Dataset Information. We provide the dataset names (Video Games abbreviated Games
and Wine Quality abbreviated to Wine), the evaluation metric used, the number of features (categor- ical and continuous), the number of samples and the percentage of halin//ll///estt split.
Dataset
# Samples
Split
Name
Metric
Cat
Cont
Train
Valid
Test
Train
Valid
Test
Alcohol
RMSE
17 10
253
63
79
.64
.16
.20
Games
RMSE
4
10623
2655
3320
.64
.20
Wine
RMSE
0 11
1024
255
320
.64
.16
.20
Chocolate
RMSE
6
1149
287
359
.64
.16
.20
Poker
RMSE
10 |
20008
5002
1000000
.02
.005
.975
Titanic
F1
2
463
154
155
.60
.20
.20
Health
F1
5
59724
14932
18664
.64
.16
.20
Susy
F1
0 18
4000000
500000
500000
.80
.10
.10
the workday alcohol consumption of the student the final goal. The model aims predict the students' workday alcohol consumption which a target range to 5 where is very low and very high consumption.
The next dataset is the Video Games Sales dataset'. The dataset also doesn't ' have a clear final goal or leaderboard information. There features of videos games such rank, publisher, year and Genre and sales information. We have information sales for North America, Europe, Japan, other countries and global sales. Predicting global sales means we would have omit information about sales in other countries as it would be just simple sum of those sales, so we decided to predict North American sales given information on the video game and sales information in other countries not including global sales.
The Wine Quality dataset Cortez et al. (2009) aims to predict quality score between 0 and 10 of the wine given its features. The features are continuous values representing different acidity rates, sugar levels, density, pH and more.
The Chocolate Ratings dataset? was created to generate expert opinions chocolate. We must predict the expert ratings which are values between and 5 where is bad taste and 5 is the best taste. The features include the company, location, type of beans, percentage of cocoa, and origin information.
The Poker Hand dataset Cattral et al. (2002) is very large set of poker hands. Each sample is set of cards indicating the card numbers as features and their suits as 5 more features. The final goal of this dataset is to predict the poker hand such 0 for nothing, 1 for three of a kind, 4 for a straight, 5 for a flush, 6 for a full house, 7 for four of a kind, 8 for straight flush, and for royal flush. We used this as an regression problem where higher (9) the better hand and lower (0) the worse the hand.
pair, 2 for two pairs, 3 for
The Titanic dataset² uses information on passengers of the Titanic predict whether they survived. The features include gender, cabin, location of embarkment, ticket class (Ist, 2nd, 3rd), number of siblings or spouses, number of parents children, age and fare. The goal is predict the survival of the individual (yes or no) using Fl as a metric. There is predefined test set without labels which must be submitted through kaggle to be evaluated, from the labelled train set only. We also take our best available model for this dataset and run it through kaggle to get a test score for their test set in the experiments section. Note that a new dataset Titanic Extended* was introduced with many more features and much less empty features derived from the literature allowing others to achieve 100% accuracy, so we opted to use the more difficult prior version for testing without knowledge of the extended features.
our results reflect train/validation/test split
http:///wwwwwwkgl..ooooo https://ww.kkaagg.. http:///wwwwww.oooooo/////////ee www.:wwwwwwwwe......

---

Under review as a conference paper at ICLR 2022
The Health Insurance dataset' aims to predict vehicle insurance sales to customers of health in- surance. We are given information on the policy holder such as a unique ID (omitted in training), gender, age. has driving license, region, types of vehicle information like age and damage, and information on their premiums. The goal is whether the customer will accept the vehicle insurance which is a binary prediction using F1 metric.
The Susy dataset Baldi et al. (2014) is our largest dataset containing 4 million training samples, 500k validation and 500k test samples. The test set was predefined for this dataset as the last 500k samples in the list. The dataset contains simulation data of a particle collider with the goal to find rare particles. There are eight kinematic features of the collision and 10 functions of those features with the final goal distinguishing signal from background using Fl as metric.
4 METHODOLOGG
To predict on these datasets, we use tabular neural networks repurposed from FastAl Howard et al. (2018) and using their cycle training procedures. The cycle training increases and reduces the learn- ing rate as needed to reach the best local minimum, so we are able to set default learning rates for all models and automatically train our pruning approaches. We kept the number of training epochs the same for all pruned versions of a dataset but used early stopping to avoid overfitting the models based validation scores. The data was preprocessed by converting categorical features to em- beddings, filling missing values in features and normalizing continuous features. The models were altered allow pruning of nodes selected by different pruning approaches we describe below. Py- torch provides functionality to select weights to prune with I.1 norm pruning, a generic LN norm pruning where we selected N=-inf, and random pruning. Of infinite possibilities for N. we chose the interesting case of in- because while N=1 selects nodes based on the sum of all weights, we didn't 't want similar norm such as N=inf which selects the largest weight, so we chose N=-inf which focuses on the smallest weight of each node and prunes based on the largest of these norms (even though technically N=-inf isn't it a norm). We pruned all weights associated to nodes from the linear layers of the model depending on rate P where the pytorch functions provide mask on the nodes setting all of the weights to O.
While the pruning functionality provided by pytorch is useful for testing, we wanted the full benefit of smaller models which provide inference and training time improvements rather than a mask over the weights which still require computation. In this case we designed simple approach to fully prune the weights. First we save the initial untrained weights of the model WO. Then we train the model until reach the optimal validation stopping point and record the weights WI. Next we generate pruning masks for WI depending on the selected norm (or random) and pruning rate P. then apply the masks on the initial untrained weights WO. The remaining subset of weights from WO will be copied over to a smaller blank model sized appropriately for the zero weights. Then we continue to train the pruned model depending on the iterative or oneshot styles described below using the smaller network.
FastAl's tabular models start with a concatenation of categorical embeddings and continuous fea- tures, then linear layers follow with optional batch normalization. We implemented batch normal- ization in the pruning process, but results were better without these layers and removing them left us with smaller and faster models.
4.1 ITERATIVE PRUNING
A simple iterative approach can be thought of starting with a model size and reducing it slowly until we reach an optimal size based on criteria for accuracy and speed performance. The iterative approach uses pruning rate P=0.5 (50%). We describe two iterative pruning styles, one using large model potentially holding many lottery tickets, and the other using a good performing model as starting point with potentially better lottery tickets selected (but possibly smaller in size).
The first approach uses a starting point of a large model [1600, 800] where the parameters are the sizes of the linear layers respectively. We call this starting point the original model. We train the
4

---

Under review as a conference paper at ICLR 2022
original model, prune it at the rate of P, then repeat the pruned model until reach the smallest state possible (ideally [1, IJ).
In the second approach, instead train at various starting points of different model sizes starting with [1600, 800], then continuing with [800, 400] and so on reducing the size by 50% at each step. We train all of these original models in search of the best starting point for the best performing model. Then we run iterative pruning like in the first approach on this potentially smaller model, but ideally with better lottery ticket weights available to be selected.
4.2 ONESHOT PRUNING
The oneshot pruning approach works like iterative, but uses a prune rate of P=1.0. This means we prune the entire model in shot reducing down entirely size of 1J. We oneshot all of the potential starting points we described in iterative (second approach), then we select the best performing oneshot model.
EXPERIMENTATIONS. RESULTS AND DISCUSSION
There are three components our results. The first is best performing accuracy, the second is comparison train and inference times, and the last is a comparison model sizes with minimal affect to accuracy. To evaluate the overall performance of our models, also provide table of comparisons using several different tabular models. In all tests, all random seeds were set to the same value allow fair comparisons and reproducibility of any result shown. We paid careful attention to have the model weights set to the same initial states for all styles of pruning.
Table 2: Train times averaged over all epochs. The size column shows the possible prune states representing the number of nodes in each layer of the neural network, each state 50% smaller than the last. Then for each dataset provide the average epoch train time in seconds followed by the percentage improvement from [1600, 800] in parentheses.
Size
Alcohol
Games
Wine
Chocol.
Poker
Titanic
Health
Susy
0.21s
35.47s
0.90s
1.04s
19.33s
0.38s
139.50s
1.11.88s
1600, 800
(.00)
(.00)
(.00)
(.00)
(.00)
(.00)
(.00)
(.00)
0.15s
12.23s
0.49s
0.65s
12.49s
0.25s
52.40s
1168.57s
800, 400
(.247)
(.655)
(.454)
(.373)
(.354)
(.328)
(.624)
(.676)
0.14s
6.53s
0.39s
0.55s
10.
0.23s
32.45s
529.79s
400, 200
(.302)
(.816)
(.561)
(.468)
(.477)
(.4)
(.767)
(.853)
0.14s
5.24s
0.36s
0.54s
8.52s
0.21s
30.28s
425.54s
200, 100
(.33)
(.852)
(.595)
(.483)
(.559)
(.437)
(.783)
(.882)
0.
4.57s
0.35s
0.54s
7.77s
0.21s
27.68s
337.41s
100, 50
(.322)
(.871)
(.606)
(.478)
(.598)
(.448)
(.802)
(.907)
0.14s
4.54s
0.34s
0.52s
7.2s
0.21s
27.18s
314. 114.14s
50, 25
(.302)
(.872)
(.616)
(.498)
(.627)
(.443)
(.805)
(.913)
0.14s
5.47s
0.35s
0.51s
6.36s
0.21s
27.37s
280.77s
25, 13
(.335)
(.846)
(.615)
(.505)
(.671)
(.439)
(.804)
(.922)
0.14s
4.30s
0.34s
0.51s
5.07s
0.21s
27.2s
289.77s
13, 7
(.324)
(.879)
(.615)
(.507)
(.738)
(.45)
(.805)
(.92)
0.14s
4.31s
0.34s
0.52s
4.95s
0.21s
26.91s
287.14s
7,3
(.321)
(.879)
(.615)
(.496)
(.744)
(.45)
(.807)
(.92)
0.14s
4.63s
0.34s
0.52s
5.24s
0.21s
27.2s
292.28s
3,
(.302)
(.869)
(.622)
(.505)
(.729)
(.449)
(.805)
(.919)
0.14s
4.47s
0.33s
0.50s
5,56s
0.21s
24.28s
302.94s
1,
(.339)
(.874)
(.626)
(.52)
(.712)
(.452)
(.826)
(.916)
5

---

Under review as a conference paper at ICLR 2022
Table 3: Inference times set. The size column shows the possible prune states representing the number of nodes in each layer of the neural network, each state 50% smaller than the last. Then for each dataset provide the inference time on the test in seconds followed by the percentage improvement from [1600, 800] in parentheses.
Size
Alcohol
Games
Wine
Chocol
Poker
Titanic
Health
Susy
0.06s
0.82s
0.08s
0.11s
213.77s
0.06s
4.22s
33.83s
1600, 800
(.00)
(.00)
(.00)
(.00)
(.00)
(.00)
(.00)
(.00)
0.06s
0.7s
0.08s
0.1s
220.94s
0.06s
3.79s
24.12s
800, 400
(.04)
(.145)
(.087)
(.077)
(-.034)
(.048)
(.101)
(.287)
0.06s
0.7s
0.07s
0.1s
188.36s
0.05s
3.71s
21.54s
400, 200
(.038)
(.153)
(.101)
(.09)
(.119)
(.17)
(.12)
(.363)
0.05s
0.68s
0.09s
0.1s
166.78s
0.05s
4.03s
22.75s
200, 100
(.105)
(.175)
(-.083)
(.093)
(.22)
(.159)
(.044)
(.328)
0.05s
0.68s
0.07s
0.1s
154.82s
0.05s
4.04s
21.94s
100, 50
(.11)
(.169)
(.118)
(.098)
(.276)
(.177)
(.041)
(.352)
0.05s
0.7s
0.07s
0.1s
153,34s
0.05s
4.03s
20.55s
50, 25
(.11)
(.143)
(.136)
(.109)
(.283)
(.169)
(.045)
(.393)
0.05s
0.7s
0.08s
0.1s
132.77s
0.06s
4.09s
19.3s
25, 13
(.111)
(.146)
(.1)
(.124)
(.379)
(.139)
(.03)
(.43)
0.05s
0.68s
0.07s
0.1s
105. 15s
0.06s
4.06s
19.7s
13, 7
(.118)
(.169)
(.143)
(.11)
(.508)
(.109)
(.037)
(418)
0.05s
0.68s
0.07s
0.1s
105.67s
0.05s
4.03s
20.57s
7,3
(.122)
(.174)
(.123)
(.12)
(.506)
(.184)
(.044)
(.392)
0.06s
0.68s
0.08s
0.1s
106.744 106.74s
0.05s
3.65s
22.04s
3,
(.044)
(.175)
(.085)
(.1)
(.501)
(.184)
(.135)
(.349)
0.05s
0.68s
0.07s
0.1s
103.05s
0.05s
3,59s
20.54s
1,
(.083)
(.175)
(.156)
(.145)
(.518)
(.156)
(.148)
(.393)
Table and Table show the time taken run the tabular models. The first table contains the training times averaged over all epochs while the second contains the inference time the test set. The tables present the number of seconds taken and the reduction time in percentage compared to the largest model. The trend shows that smaller model can lead to faster training and faster inference until we reach extremely small models few nodes which show some noise of a few milliseconds. The small datasets of few hundred samples don't 't provide much insight, but the larger datasets show a better downtrend. The computations were made on a 2 CPU systems for all tests, no of GPU.
Table 4 highlights our best performing models accuracy-wise. In bold are the models with the best accuracy. The table highlights the RMSE or Fl, the size of the model, the difference in percentage compared to the original model, and the pruning mode that generated the result. Iterative on the largest model (approach improving. Iterative on the best performing model (approach 2) has one best case tied with approach for the poker dataset, but happens that the largest model was the best performing model, both approaches generated the same result. Oneshot had one case of generating best performing model for the smallest dataset in our experiments. There were two datasets, Wine and Chocolate, which could not be pruned further without some loss in accuracy, but they can be pruned extremely small with some degradation in accuracy.
has the best likelihood of generating better model with 5/8 datasets
We compare the tabular models to other models in Table 5 and found that even the original model has the best performing accuracy for all but one result. In this case the Health dataset using SVR resulted in better accuracy than the original model, but not better than pruned models. All comparisons shown are to the original tabular model. The table shows the following models: K Nearest Neigh- bors (KNN), Linear Regression (LR), Pesilln-SSuport Vector Regression (SVR), Gradient Boosting

---

Under review as a conference paper at ICLR 2022
Table 4: Comparison of accuracy (RMSE/F1) for each dataset between the original tabular models, and iterative/oneshot modes. First result is the best performing original models, second result is best performing iterative when pruning the best original model, third result is best performing iterative when pruning the largest model [1600, 800], and the final result is the best performing oneshot model. Results are shown RMSE/F1 depending on the dataset along with the model size, then we include the difference in percentage along with the pruning mode for the model. Bold is marked as best performing accuracy.
Best
Iterative
Iterative
Dataset
Oneshot
Original
(from best original)
(from [1600,
800])
Acc
Acc
Diff%
Ace
Diff%
Acc Size
Diff%
Size
Size
Mode
Size
Mode
Mode
0,90356
0.89509
+0.937%
0.89941
+0.459%
0.89357
+1.106%
Alcohol
7,3
1,
L1
1,
LN
1, 1
LN/Rand
0.30398
0.30136
+0.862%
0.26266
+13.596 %
0.45927
-510066%
Games
25,
13,
100, 50
LN
Rand
0.59206
0.59935
-1.231% %
0.59863
-1.11%
0.61947
-4.63%
Wine
100, 50
13,
LI
25, 13
LN
1,
Rand
0.46411
0.48331
-4.137%
0,48291
-4.051%
0.48117 17
-3.676%
Chocol.
200,100
1,
All
200,100
LN
1,
L1/LN
0.56241
0.53551
+4.783 %
0.53551
+4.783%
0.76128
-35366%
Poker
1600,800
400,200
L1
400,200
L1
1,
LI
0.78571
0.78571
+0.000%
0.7931
+0.941 %
0.78333
-0.303%
Titanic
13,7
7,3
LN
13,
L1
1,
LI/LN
0.82263
0.82322
+0.072%
0.82428
+0.201 %
0.82242
-0.026%
Health
800,400
7,3
LN
200,100
Rand
1,
LN
0.77080
0.77071
-0.012%
0.77109
-0.038%
0.75316 1,
-2.289%
Susy
200,100
100,50
Rand
200,100
L1
LI/LN
Regressor (GBR), Decision Tree (DT) and Random Forest (RF). We also used the equivalent clas- sifier version of these models for the classification tasks. All models were tested on the exact same train, validation and test sets and we also preprocessed the data using in the same way converting text/categorical features to values, normalizing continuous features and filling in missing values. KNN was optimized for K using the validation set, GBR and RF used 100 estimators and SVR used the RBF kernel defaults selected by seikit-learn. Note that Susy has KNN and SVR missing, this is because even with 32 CPU machine and weeks of computation we couldn't produce result in time, so we omitted them from the table.
In comparison with other approaches, some datasets do not have reported results such as Video Games, Alcohol and Chocolate. The Titanic dataset is shown on kaggle achieve 100% on the test set using random forest and other approaches due to an extended features dataset derived from the literature. Without using extended features, running the same model as in our best result (L1 iterative [13, 7I) on kaggle's test set gives us score of 0.77511. Because of missing available comparisons, we provide comparisons using exactly the same data split and preprocessing for all datasets using several models in Table 5 mentioned previously.
For the Red Wine dataset, Dahal et al. (2021) reports the following RMSE results: 0.63245 using a layer neural network, 0.61 163 using GBR, 0.62145 using SVM, and 0.62201 using ridge regression. In this case we hold the best result with 0.59206 RMSE (+3.200%) using an original tabular model sized at [100, 50].
Şekeroğlu (2021) reports several Fl scores on the Health dataset: 0.80 using KNN, 0.77 using naive bayes, 0.76 using I.R, 0.81 using RF, 0.80 using MI.,, 0.72 using SVM. Our best model using

---

Under review as a conference paper at ICLR 2022
Table 5: Comparison of different models the original tabular model in Table 4, top is RMSH/FI depending on the dataset and bottom is the difference in percentage compared to the original tabular model. all but one case (Health SVR) the original tabular model outperforms all of these models. Note that the pruned tabular models are not reflected in these results and the difference is only
computed for original.
Dataset
KNN
LR
SVR
GBR
DT
RF
0,94405
0.96838
0.94680
0.92735
1.08500
0.94940
Alcohol
44.81%% %
-7.174%
4.786%
-2.633%
-20.081 %
-5.073%
1.0253
0.61135
0.98243
0.69862
0.70800
0.68026
Games
237.2922%
-101.115%
223.189%
-29.824%
-132.91% %
123.784%
0.65610
0.62515
0.59983
0.62044
0.81586
0.60776
Wine
-10.816%
-5.589%
-1.312%
44779%
-37.800%
-2.65%%
0.51295
0.51541
0.50163
0.50100
0.63895
0.48868
Chocolate
-00.233%
-110553%
-8.084%
-7.949%
-37.672%
-5.994%
0.73195
0.77344
0.74685
0.71312
1.06652
0.67302
Poker
-30.145%
-37.522%
-32.795%
-26.797%
-896344%
-1.6677%
0.68852
0.73504
0,50000
0.73214
0.68421
0.74380
Titanic
-1.370%%
-6.449%
-36.363%
-6.818%
-12.918%
-5.334%
0.79414
0.82106
0.82367
0.82137
0.71185
0.80276
Health
-3.463%
-0.191%
+0.126%
-0.153%
-13.467%
-2.415%
0.68483
0.76511
0.69232
0.76648
Susy
-11.153%
-0.738%
-10.182%
-0.560%
random iterative pruning (approach 1) and size of [200, 100] achieves an Fl score of 0.82428 (+1.763%)
Best models selected by size with less than 2% divergence in accuracy of the original model. For each dataset (first column), noted the size of the original model in the second col- umn. Then we show the smallest possible model with <2% divergence in accuracy for each of the original, iterative on best original, iterative on [1600,800], and finally oneshot. The table shows the percentage difference in RMSE or F1, and P the prune rate compared to original. If the difference in accuracy 22% then that was the best performing model accuracy-wise and we could not produce a valid smaller model using that approach. Bold is the smallest model which doesn's 't exceed the 2% divergence in accuracy rule.
Table 6:
Iterative
Iterative
Original
Dataset
Orig.
<2%
<2%
Oneshot
<2%
(best orig.)
([1600,800)
Size
Diff (P)
Diff (P)
Diff (P)
Diff (P)
Alcohol
7,
-0.923% (.600)
+0.937% (.800)
+0.459% (.800)
+1.106% (.800)
Games
25, 13
-0.158% (.737)
+0.862% (.474)
+4. 14.% (-.974)
-51 -510086% (.947)
Wine
100, 50
0.000% (:000)
-1.231 % (.867)
-1.110% (.747)
-4.630% (.987)
Chocol.
200,100
0.000% (.000)
-4. 4.137% (.993)
-4.441% (.993)
-3.676% (.993)
Poker
1600,800
-0.768% (.969)
-0.349 % (.984)
% (.984)
-3.3600% (.999)
Titanic
13, 7
0.000% (.000)
-1.754% (.800)
-0.303% (.900)
-0.303% % (.900)
Health
800,400
-0.085% (.997)
-0.188% (.998)
-0.287% (.998)
-0.026 % (.998)
Susy
200,100
-1.558% (.987)
-0.381 % (.987)
-0.397% (.987)
-.2889% (.993)
Museba et al. (2021) reports accuracy as their metric and achieve the following results on the Poker dataset: 0.7687 using the Heterogeneous Dynamic Ensemble Selection based Accuracy and
Diversity (HDES-AD ADD approach, and 0.9068 using their HDES-ADD variant. They also report a comparison Diversity for Dealing with Drifts (DDD) Minku & Yao (2011) with 0.7867, Online
8

---

Under review as a conference paper at ICLR 2022
Accuracy Updated Ensemble (QAUE) Brzezinski & Stefanowski (2014) with 0.7325, and Active Fuzzy Weighting Ensemble (AFWE) Dong et al. (2018) with 0.7126. These approaches all use techniques train ensembles of models and in particular Museba et al. (2021) generates models based on diversity and accuracy. We used RMSE instead of accuracy, but reran our models using F1 to optimize the models for this comparison. We achieved 0.87915 F1, 0.88360 accuracy (-2.558%) using LI iterative [800, 400] (approach 1), and a similar score with a smaller model 0,87958 FI, 0.88462 accuracy (-2.446%) using Random iterative [400, 200] (approach 1).
Azhari et al. (2021) uses a 70/30 train-test split on the Susy dataset at random. We used the recom- mended test set provided for our results of 10% in size using the final 500k samples in the list and a validation set of 10%. They report accuracy as their metric with the following scores: 0.7884 using LR, 0.774 using RF, 0.7546 using DT, 0.793 using Gradient Boosted Tree (GBT). We achieved a better result of 0.77109 F1 using L.1 iterative [200, 100] (approach 1), and an accuracy of 0.80285 (+1.242%)
Finally as our last result, we show comparison of model size to the original tabular model by selecting the smallest model we can with less than 2% divergence in RMSE or FI. We do the same with the original models, so we select the best size for original with less than 2% divergence in accuracy from the best performing RMSE/FI original model. Then we apply the same rule to iterative and finally oneshot. The results are shown in Table 6 highlighting in bold the smallest model following the described conditions. We show the original model size, then we present the difference in accuracy of other models and the prune rate from the same original model. In 7 out of 8 datasets we can generate a smaller model with minimal affect to RMSE/FI including original with selection. In 6 of 8 datasets we outperform original with selection for 2% divergence generating models over 85% smaller and many over 98% smaller.
6 CONCLUSION AND FUTURE WORK
In conclusion, we presented two approaches to pruning tabular neural networks, iterative and oneshot. based on the lottery ticket hypothesis using structured node pruning. The results are pre- sented for 8 tabular datasets of different sizes and feature sets. We improved accuracy in 6 of the 8 datasets when considering accuracy alone. We also show up to 85% reduction in nodes in of datasets considering model size with limited affect to RMSE/F1 and over 98% reduction for many of them. We show that the tabular models outperform several other tabular models where compar- isons weren't i available such KNN, RF, SVR, DT, LR and GBR. Finally, when comparing to other papers, we show an improvement of +3.2 in RMSE for the Wine dataset, +1.763% in Fl for the Health dataset and +1. +1.242% in accuracy for the largest dataset of 5 million samples, Susy.
We found that the iterative approach while pruning a large model obtains the majority of top results compared to pruning better but smaller models, or in comparison oneshot pruming. We go beyond just masking weights and implement a structured pruning approach reducing the model architecture to layer sizes low as [1, I] while improving accuracy in comparison to large model sizes like [1600, 800] for the alcohol dataset. Finally, for each dataset, we show the advantage in training and inference time of structured pruning at each pruning state. Future work will be focused on pruning layers once the size of layer reaches one node, and other focuses will be on feature selection using the lottery ticket hypothesis where pruning nodes responsible for certain features will result in the removal of that feature.
We asked the question "if these smaller networks perform just as well as the larger ones, could one not train the smaller network architecture to begin with?", and we believe it is clear that given the right set of initial weights for the network, a smaller, faster and better network can be trained to begin with. We show the existence of these weights through iterations of pruning, so perhaps there exists an approach to initializing the network as a function of the data using lottery weights.
REFERENCES
Mourad Azhari, Abdallah Abarda, Badia Ettaki, Jamal Zerouaoui, and Mohamed Dakkon. Using machine learning with pyspark and mlib for solving binary classification problem: Case of searching for exotic particles. In Recent Advances in Intuitionistic Fuzzy Logic Systems and Mathematics, pp. (18. Springer, 2021.

---

Under review as a conference paper at ICLR 2022
Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature communications, 5(1):1-9, 2014.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint 20X5 /4165, 2020.
Dariusz Brzezinski and Jerzy Stefanowski. Combining alock-based and online methods in learning ensembles from concept drifting data streams. Information Sciences, 265:50-61 67, 2014.
Robert Cattral, Franz Oppacher, and Dwight Deugo. Evolutionary data mining with automatic rule generalization. Recent Advances in Computers, Computing and Communications, 1(1):296-300, 2002.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. arXiv preprint arXiv:2007.12223 2020.
Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks. In International Conference on Machine Learning, pp. 1695-170000 PMLR, 2021.
Paulo Cortez and Alice Maria Gonçalves Silva. Using data mining to predict secondary school student performance. 2008.
Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Modeling wine preferences by data mining from physicochemical properties. Decision support systems, 47(4): 547-553, 2009.
KR Dahal, JN Dahal, H Banjade, and S Gaire. Prediction of wine quality using machine learning algorithms. Open Journal of Statisties, 11(2):278-2832 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint wrXiv:1810004805, 2018.
Bing-Wei
Fan Dong, Jie Lu, Guangquan Zhang, and Kan Li. Active fuzzy weighting ensemble for dealing with concept drift. International Journal of Computational Intelligence Systems, 2018.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks (2018). arXiv preprint arXiv: /803. 03635, 2019.
Sharath Girish, Shishira R Maiya, Kamal Gupta, Hao Chen, Larry Davis, and Abhinav Shrivastava. The lottery ticket hypothesis for object recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 762-771, 2021.
Ken Gu and Akshay Budhkar. A package for learning on tabular and text data with transformers. In Proceedings of the Third Workshop on Multimodal Artificial Intelligence, pp. 69-73, 2021.
Jeremy Howard et al. fastai. https:: https////gthhb..../ com/ wtss///aaaaaaaa/aaa fastaiaaaaaaaa..... (atcc///sstaa//stta........ 2018.
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06674 2020.
Marian Cristian Mihaescu and Paul Stefan Popescu. Review on publicly available datasets for edu- cational data mining. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 11(3):e1403, 2021.
Leandro L Minku and Xin Yao. Ddd: A ensemble approach for dealing with concept drift. IEEE transactions on knowledge and data engineering, 24(4):619633,, 2011.
Ari S Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. arXiv preprint 1 027793 2019.
10

---

Under review as a conference paper at ICLR 2022
Tinofirei Museba, Fulufhelo Nelwamondo, and Khmaies Ouahada. An adaptive heterogeneous line learning ensemble classifier for nonstationary environments. Computational Intelligence and Neuroscience, 2021, 2021.
Inkit Padhi, Yair Schiff, Igor Melnyk, Mattia Rigotti, Youssef Mroueh, Pierre Dognin, Jerret Ross, Ravi Nair, and Erik Altman. Tabular transformers for modeling multivariate time series. In ICASSP 2021-2021 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp.
IEEE, 2021.
Ali Galip Şekeroğlu. Impacts of feature selection techniques in machine learning algorithms for cross selling: comprehensive study for insurance industry. 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.


Reviews:
[{"review_id": "r0LUhIRbENf", "reviewer": "Reviewer_EuY8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While I find interesting the topic addressed, below are some suggestions for improvement:\n- In the Introduction section try to present in a concise manner which are the paper novel contributions in comparison with state-of-the-art\n- Improve Related Work on sparse neural networks for tabular data. There are some papers addressing this topic. You can start by reading some recent survey papers (e.g., [1]) and some concrete approaches which use sparse neural networks for tabular data (e.g., [2,3]).\n- From Table 1, I can see that the datasets studied have quite a low number of features. To properly validate the proposed method, I suggest also trying datasets with a much larger number of features. \n- Add some algorithms to describe the proposed methods in the Methodology section. Also, please add equations with the pruning criteria. \n- Provide the code developed as open-source for easy reproducibility.\n- Please improve the readability of the Results section, while trying to have a more structured presentation and better qualitative discussions. Currently, it is hard to follow all the details and to understand which is the actual improvement of the proposed method with respect to the state-of-the-art. Preparing some summary plots for the whole experimental evaluation may help. Also, please extend the baseline methods used for comparison to cover well the state-of-the-art.\n- The current writing style of the paper does not follow the academic writing style and it is more like a report. Please try to improve it.\n- There have been previous attempts to address the question from the last paragraph of the Conclusion section. Non-exhaustively, please see for instance [1,2,3]. I suggest trying to rephrase it to make it more accurate.\n\nReferences:\n- [1] Hoefler et al., Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks, JMLR 2021, https://www.jmlr.org/papers/v22/21-0366.html \n- [2] Liu et al., Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware, NCAA 2020, https://link.springer.com/article/10.1007/s00521-020-05136-7 \n- [3] Peterson et al., Cognitive model priors for predicting human decisions, ICML 2019, https://arxiv.org/abs/1905.09397 \n", "summary_of_the_paper": "This paper proposes two approaches to use sparse neural networks on tabular data. The topic is interesting and sometimes overlooked in the literature. The proposed approaches are based on node (likely referring here to neuron) pruning. Overall, I believe that the proposed method description, experimental evaluation, and paper writing quality can be improved. ", "summary_of_the_review": "Overall, even if it offers some interesting perspectives, I believe that this paper is immature and not ready yet for publication. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "m8yWndCrHma", "reviewer": "Reviewer_EcHe", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Comprehensive set of experiments on a tabular network and corresponding datasets using a single dataset\n\nWeaknesses:\n- Lack of novelty; see below\n- Lack of clarity and experimental details.\n", "summary_of_the_paper": "This paper presents an empirical study of pruning algorithms on tabular neural networks using the lottery ticket hypothesis. A comprehensive set of results show that iterative pruning seems to work better compared to one-shot pruning. ", "summary_of_the_review": "My major concern for this paper is the lack of novelty and lack of experimental details. In the end, this is an empirical work providing no novelty. Those details should be extremely clear.\n\nNovelty:\n- The introduction claims the paper proposes 2 pruning methods; However, in the method section, there are only references to two pruning strategies: iterative and one shot. These two strategies are well-known and utilized in the pruning literature; Iterative pruning was initially presented at Han et al. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization, and Huffman Coding. \nFor one-shot, also referred as single-shot pruning, Lee et al. SNIP: SINGLE-SHOTNETWORKPRUNING BASED ONCONNECTIONSENSITIVITY, ICLR 2019\nThe paper does not refer to any approach in the literature. More importantly, pruning is not only about strategy but also the saliency metric used to remove parameters. \n\n\n\n\nOn the experimental side:\n\n- For instance, the paper describes two modalities of pruning (iterative vs one-shot) but does not describe the type of metric used to measure the saliency of each neuron / parameter. Algorithmic details here would be beneficial. There is a reference to the type of norm but seems like those are not details. Would be also good to show numbers for importance based methods (see for instance Molchanov et al. Importance Estimation for Neural Network Pruning, CVPR2019)\n- Network architectures: The algorithm uses the results from FastAi; What are the details of these architectures? Would be good to add some additional architectures to have more conclusive results.\n- Tables seem to be too large (for instance the training time). Adding better tables / graphs would leave space to have these details in place. \n\n- The paper focuses on the lottery hypothesis; However, the ultimate goal is to prune a network as much as possible while maintaining accuracy. Therefore, I would expect comparisons to other approaches not based on the lottery ticket: importance-based pruning; magnitude, prune while training, zero shot pruning.....\n\n\n\n- How are models with less than 2% accuracy selected? I guess that is an empirical selection, right?\n\n\nClarity:\n- Would be more readable if the paper explains the lottery hypothesis and related pruning works and how those approaches can not be applied to tabular data; In the current form, seems like it is just plug and play: The paper states: We focus on applying this hypothesis to tabular neural networks. Why is this challenging? Why is this different to applying to other data / networks? What has been done in that space in terms of pruning / compression?", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "CgRF3IWmRWM", "reviewer": "Reviewer_k3Jq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In terms of significance the problem-to-solve, I think compressing a tabular neural network is of at least modest importance. This is mainly because most existing works on pruning algorithms do not even consider networks designed for tabular datasets. In a sense, this is not too surprising---neural networks are usually not state-of-the-art methods on such datasets, the benefits of overparameterization are obscure in such domains, and the need for processing tabular data on edge devices is worse-motivated than vision/language/speech domains. Still, having a work that formally studies pruning models for tabular datasets will be a nice starting point for future works in the direction. But at this stage, I do not think this paper is ready to be published, due to the following reasons:\n\nOne of the biggest limitations of this paper is that the proposed method is only evaluated on a specific tabular neural network (namely the one provided by FastAI). Up to my knowledge, there are many different types of neural network architectures in the literature designed to process tabular data, including TabNet, TabTransformer and SAINT. Yet, authors evaluate the proposed pruning scheme only on the tabular neural networks that is included in the FastAI framework (which I am not sure about what the framework is implementing). Thus I am so sure if the methods will also perform well on other model architectures, or these methods are very specialized to a specific model. If the latter is the case, I expect the impact of this work to be very small, and is more appropriate to be included as a manual section of the framework rather than an academic publication.\n\nAnother thing that worries me is the experimental validation of the method (and many design choices). Although the structured pruning pipeline with weight rewinding looked quite standard, there were some unusual choices, such as using the $L_{-\\infty}$ criteria for pruning nodes. I am quite curious how much this choice contributes to the performance of the method (when compared to $L_1$, $L_2$, or $L_{\\infty}$). If this is indeed a critical factor for the pruning performance, I think this novel finding could be of interest to a much wider audience. But as of now, I do not think the paper is well-validating the method or this specific component; the paper lacks comparisons to the baseline pruning methods, with/without rewinding and/or using different norms. Also, using $P=0.5$ looked quite extreme to me, as many previous iterative pruning works tend to prune much smaller fraction of weights in each iteration (e.g., the lottery ticket paper). I wonder if this has any critical impact on the final performance.\n\nAs a minor remark: The paper was not 100% clear to me, especially in section 4. Could the authors give a more formal statement of the algorithm (say, as a pseudo-code in \"algorithm\" environment)?", "summary_of_the_paper": "This paper proposes two structured pruning strategies for compressing a (very specific) tabular neural network: iterative pruning, and one-shot pruning. Both methods work by removing the node with smallest $L_{-\\infty}$ norm, i.e., the one with smallest minimum weight of each node. After removing a designated amount of nodes, the initial weights of the models are restored (as in Frankle et al. (2019)), and optionally proceeds for the next pruning iteration. The methods achieve noticeable reduction in the total training (wall-clock) time and inference time, while introducing only small degradation in performance.", "summary_of_the_review": "The paper studies a fairly important problem, but I do not think that the proposed methods are validated well experimentally. Much more additional experiments will be needed to clarify the benefit of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "8BeTQSLwAVi", "reviewer": "Reviewer_Y7kX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The questions are outlined below: \n- While I thank the authors for giving introductions to tabular datasets, the length of them seems too long - taking almost one page. Authors may consider putting some description into the supplementary materials. \n- More experiment settings would be better. For example,  the authors use a very large pruning rate for one-shot pruning while for iterative pruning the rate is small. Maybe consider comparing one-shot pruning with P=0.5. \n- The way where authors present their results seems a little confusing - especially putting RMSE and F1 together as accuracy. We prefer lower RMSE but higher F1, so the calculation of diff% is inconsistent in Table 4 for example. \n- Would the authors provide a detailed explanation on why we need sparse models for tabular datasets? Increasing the performance a little bit seems weak for me - is there any challenge like over-parameterization? We need many rounds of IMP to reduce the model size from [1600,800] to [1,1] while the inference time seems to only be reduced by 90%. More importantly, it seems that we can pick a small network from the beginning (if I have understood correctly), which means we don't need pruning to get compressed networks. I find this is an essential difference between tabular and other (CV, NLP, speech) models, i.e., the latter one doesn't permit such a big modification to the models' structure otherwise the performance is greatly harmed. However, maybe it is not the case on tabular models. \n", "summary_of_the_paper": "This paper investigates model pruning and the Lottery Ticket Hypothesis in the context of tabular datasets and model training. The authors apply a set of pruning techniques to the tabular neural networks from FastAI and examine whether the LTH still holds on the tabular datasets and their corresponding models. Different pruning methods are used, namely different norms of selecting the parameters to be pruned, to see if the winning tickets exist. The results suggest that on some datasets the winning ticket exist, and outperform classical methods like random forest. ", "summary_of_the_review": "The experiments and the methods are clear in this paper, occasionally with some problems which I have mentioned above. From my perspective of view, the main problem here is motivation. I would give a score of 5 currently, but I am also willing to change my score if the authors provide useful responses. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: ### Summary

The paper demonstrates the applicability of pruning to tabular datasets, which aren't typically explored in the literature on pruning. The work identifies that yes, pruning can indeed be applied to this domain with some success.


### Discussion

#### Strengths
  
An unconventional domain that, nonetheless, should be studied. 

#### Weaknesses

The empirical setup does not include comparisons to baselines or ablations (e.g., different importance metrics).

### Decision

I recommend Reject. Reviewer k3Jq provides a precise and constructive set of criticisms that if solved would make for an interesting and significant piece of work.
Decision: Reject