Lottery Ticket Structured Node Pruning for Tabular Datasets

In this paper we presented two pruning approaches on tabular neural networks based on the lottery ticket hypothesis that went beyond masking nodes by resizing the models accordingly . We showed top performing models in 6 of 8 datasets tested in terms of F1/RMSE . We also showed in 6 of 8 datasets a total reduction of over 85 % of nodes and many over 98 % reduced with minimal affect to accuracy . In one dataset the model reached a total size of one node per layer while still improving RMSE compared to the larger model used for pruning . We presented results for two approaches , iterative pruning using two styles , and oneshot pruning . Iterative pruning gradually reduces nodes in each layers based on norm pruning until we reach the smallest state , while oneshot will prune the model directly to the smallest state . We showed that the iterative approach will obtain the best result more consistently than oneshot . 1 INTRODUCTION . It has been understood for a while that pruning a network after training can lead to much smaller networks that still maintains similar accuracy to the original network . However the question arises , if these smaller networks perform just as well as the larger ones , could one not train the smaller network architecture to begin with ? The lottery ticket hypothesis states , ” a randomly-initialized , dense neural network contains a subnetwork that is initialized such that - when trained in isolation - it can match the test accuracy of the original network after training for at most the same number of iterations ” Frankle & Carbin ( 2019 ) . These smaller networks are referred to as ” winning tickets ” . The process of finding winning tickets was proposed to start with an initial set of weights , train the network , prune the network , and finally reset the remaining weights back to their initial state . It has been shown that these winning tickets perform as well , or sometimes better and require less time to train when compared to the original network . In this paper we focus on applying this hypothesis to tabular neural networks for several datasets to generate smaller networks with similar performance . We use tabular neural networks from FastAI Howard et al . ( 2018 ) which work on the principle of converting categorical features into embeddings . The models come with strong features such as normalization of continuous features , filling in missing data , and converting dates into categorical features . The models can be trained with their cycle training approach which raises and lowers the learning rate to find the best local minimum while training . There are many small tabular datasets which often don ’ t get optimized in research . These datasets are filled with categorical features and missing entries while often having few samples to test . In this paper we use the lottery ticket hypothesis to improve FastAI ’ s tabular neural network to easily prune models for any tabular dataset . We test a wide range of datasets from large to small including different ranges of categorical and continuous features . This paper is organized as follows . Section 2 describes related previous works . Section 3 presents the datasets used and our pruning methodology . Experiments , results and comparisons are presented in Section 4 . Finally , the paper is concluded in Section 5 . 2 PREVIOUS WORKS . The lottery ticket hypothesis was first tested by Frankle & Carbin ( 2019 ) on dense neural networks and convolutional neural networks for MNIST and CIFAR10 . The authors achieved networks 80- 90 % smaller while both creating faster networks and increasing the accuracy of the models in comparison to the original sized models . The authors show that while pruning has been around for a long time , pruning networks often lead to more difficult training with less accuracy . By finding the lottery tickets of a network , difficulty in training of smaller networks is reduced while accuracy can be maintained and even exceed the original model . Since its introduction , there have been many papers testing the lottery ticket hypothesis on different architectures . Morcos et al . ( 2019 ) look to find winning tickets in networks by searching for common winning tickets across many datasets . They test on a range of image datasets such as Fashion MNIST , SVHN , CIFAR-10/100 , ImageNet , and Places365 . Their findings were that winning tickets generalized over many datasets , but interestingly the larger datasets produced better winning tickets that were more generalizable than those from smaller datasets . Girish et al . ( 2021 ) use the lottery ticket hypotheses for object detection achieving a sparsity of up to 80 % . Chen et al . ( 2021 ) test the lottery ticket hypothesis on ever growing graph neural networks ( GNN ) . For node classification tasks , they show a decrease in multiply-accumulate operations ( MACs ) of up to 98 % while having a sparsity of nearly 98 % with minimal affect to performance . For linking predictions , they show similar sparsity with a large reduction in MACs . The field of natural language processing ( NLP ) has recently exploded with new models since the introduction of the transformer architecture Vaswani et al . ( 2017 ) . The models grow increasingly large , from sizes like the popular BERT Devlin et al . ( 2018 ) architecture of up to 350 million parameters to sizes like GPT-3 Brown et al . ( 2020 ) reaching up to 175 billion parameters . Chen et al . ( 2020 ) prune the BERT architecture , but they begin from the pretrained state and test the lottery ticket hypothesis using downstream tasks . The paper only applies unstructured pruning stating ” since we perform a scientific study of the lottery ticket hypothesis rather than an applied effort to gain speedups on a specific platform , we use general-purpose unstructured pruning ” Chen et al . ( 2020 ) . With unstructured pruning , they are able to find networks with sparsity of 40-90 % while testing a range of downstream tasks . Tabular models are heading towards larger networks as well with the recent transformer architecture making its way to tabular data . Padhi et al . ( 2021 ) proposed a time series BERT architecture for tabular data called TabBERT and a tabular data generator TabGPT based on the GPT architecture . Gu & Budhkar ( 2021 ) use transformers to convert text in tabular data into features and released a package to process the data . TabTransformer by Huang et al . ( 2020 ) uses the transformer architecture to achieve new SOTAs and tested on 15 datasets , competing well with ensemble approaches . In this paper we focus on tabular dense neural networks with the goal to prune them as much as possible while maintaining accuracy utilizing the lottery ticket hypothesis . We test our models on 8 datasets of different varieties and show competing results for datasets with available comparisons . 3 DATASETS . In this section we describe the tabular datasets used in our experiments . We selected a wide range of datasets from as few hundred samples to millions . Each dataset has a variety of tabular features , some containing just continuous features , others containing only categorical features , and many mixed with both . The datasets are described in more detail in this section and we present the number of samples , number of features , evaluation metrics , and train-validation-test splits used in our experiments in Table 1 . The first dataset is the Alcohol dataset Cortez & Silva ( 2008 ) . It is our smallest dataset and contains the most features with a mix of continuous and categorical features . The dataset contains information on students such as school , gender , age , information like hobbies and goals , their family and related information such as work , education , size , etc . As quoted from this recent dataset survey Mihaescu & Popescu ( 2021 ) , ” This dataset has also been uploaded on Kaggle where 305 publicly available kernels perform exploratory data analysis . Unfortunately , there is not defined any task with specific validation metric such that there is no leaderboard publicly available ” , so we used the workday alcohol consumption of the student as the final goal . The model aims to predict the students ’ workday alcohol consumption which is a target range of 1 to 5 where 1 is very low and 5 is very high consumption . The next dataset is the Video Games Sales dataset1 . The dataset also doesn ’ t have a clear final goal or leaderboard information . There are features of videos games such as rank , publisher , year and Genre and sales information . We have information on sales for North America , Europe , Japan , other countries and global sales . Predicting global sales means we would have to omit information about sales in other countries as it would be just a simple sum of those sales , so we decided to predict North American sales given information on the video game and sales information in other countries not including global sales . The Wine Quality dataset Cortez et al . ( 2009 ) aims to predict a quality score between 0 and 10 of the wine given its features . The features are continuous values representing different acidity rates , sugar levels , density , pH and more . The Chocolate Ratings dataset2 was created to generate expert opinions on chocolate . We must predict the expert ratings which are values between 1 and 5 where 1 is bad taste and 5 is the best taste . The features include the company , location , type of beans , percentage of cocoa , and origin information . The Poker Hand dataset Cattral et al . ( 2002 ) is a very large set of poker hands . Each sample is a set of 5 cards indicating the card numbers as 5 features and their suits as 5 more features . The final goal of this dataset is to predict the poker hand such as 0 for nothing , 1 for one pair , 2 for two pairs , 3 for three of a kind , 4 for a straight , 5 for a flush , 6 for a full house , 7 for four of a kind , 8 for straight flush , and 9 for a royal flush . We used this as an regression problem where higher ( 9 ) the better hand and lower ( 0 ) the worse the hand . The Titanic dataset3 uses information on passengers of the Titanic to predict whether they survived . The features include gender , cabin , location of embarkment , ticket class ( 1st , 2nd , 3rd ) , number of siblings or spouses , number of parents or children , age and fare . The goal is to predict the survival of the individual ( yes or no ) using F1 as a metric . There is a predefined test set without labels which must be submitted through kaggle to be evaluated , but our results reflect a train/validation/test split from the labelled train set only . We also take our best available model for this dataset and run it through kaggle to get a test score for their test set in the experiments section . Note that a new dataset Titanic Extended4 was introduced with many more features and much less empty features derived from the literature allowing others to achieve 100 % accuracy , so we opted to use the more difficult prior version for testing without knowledge of the extended features . 1https : //github.com/GregorUT/vgchartzScrape , https : //www.kaggle.com/gregorut/videogamesales 2https : //www.kaggle.com/rtatman/chocolate-bar-ratings 3https : //www.kaggle.com/c/titanic/data 4https : //www.kaggle.com/pavlofesenko/titanic-extended The Health Insurance dataset5 aims to predict vehicle insurance sales to customers of health insurance . We are given information on the policy holder such as a unique ID ( omitted in training ) , gender , age , has a driving license , region , types of vehicle information like age and damage , and information on their premiums . The goal is whether the customer will accept the vehicle insurance which is a binary prediction using F1 as a metric . The Susy dataset Baldi et al . ( 2014 ) is our largest dataset containing 4 million training samples , 500k validation and 500k test samples . The test set was predefined for this dataset as the last 500k samples in the list . The dataset contains simulation data of a particle collider with the goal to find rare particles . There are eight kinematic features of the collision and 10 functions of those features with the final goal distinguishing signal from background using F1 as a metric .

Summary: This paper proposes two structured pruning strategies for compressing a (very specific) tabular neural network: iterative pruning, and one-shot pruning. Both methods work by removing the node with smallest $L_{-\infty}$ norm, i.e., the one with smallest minimum weight of each node. After removing a designated amount of nodes, the initial weights of the models are restored (as in Frankle et al. (2019)), and optionally proceeds for the next pruning iteration. The methods achieve noticeable reduction in the total training (wall-clock) time and inference time, while introducing only small degradation in performance.