Group-Connected Multilayer Perceptron Networks

1 INTRODUCTION . Deep neural networks have been quite successful across various machine learning tasks . However , this advancement has been mostly limited to certain domains . For example in image and voice data , one can leverage domain properties such as location invariance , scale invariance , coherence , etc . via using convolutional layers ( Goodfellow et al. , 2016 ) . Alternatively , for graph data , graph convolutional networks were suggested to leverage adjacency patterns present in datasets structured as a graph ( Kipf & Welling , 2016 ; Xu et al. , 2019 ) . However , there has been little progress in learning deep representations for datasets that do not follow a particular known structure in the feature domain . Take for instance the case of a simple tabular dataset for disease diagnosis . Such a dataset may consist of features from different categories such as demographics ( e.g. , age , gender , income , etc . ) , examinations ( e.g. , blood pressure , lab results , etc . ) , and other clinical conditions . In this scenario , the lack of any known structure between features to be used as a prior would lead to the use of a fully-connected multilayer perceptron network ( MLP ) . Nonetheless , it has been known in the literature that MLP architectures , due to their huge complexity , do not usually admit efficient training and generalization for networks of more than a few layers . In this paper , we propose Group-Connected Multiplayer Perceptron ( GMLP ) networks . The main idea behind GMLP is to learn and leverage expressive feature subsets , henceforth referred to as feature groups . A feature group is defined as a subset of features that provides a meaningful representation or high-level concept that would help the downstream task . For instance , in the disease diagnosis example , the combination of a certain blood factor and age might be the indicator of a higher level clinical condition which would help the final classification task . Furthermore , GMLP leverages feature groups limiting network connections to local group-wise connections and builds a feature hierarchy via merging groups as the network grows in depth . GMLP can be seen as an architecture that learns expressive feature combinations and leverages them via group-wise operations . The main contributions of this paper are as follows : ( i ) proposing a method for end-to-end learning of expressive feature combinations , ( ii ) suggesting a network architecture to utilize feature groups and local connections to build deep representations , ( iii ) conducting extensive experiments demonstrating the effectiveness of GMLP as well as visualizations and ablation studies for better understanding of the suggested architecture . We evaluated the proposed method on five different real-world datasets in various application domains and demonstrated the effectiveness of GMLP compared to state-of-the-art methods in the literature . Furthermore , we conducted ablation studies and comparisons to study different architectural and training factors as well as visualizations on MNIST and synthesized data . To help to reproduce the results and encouraging future studies on group-connected architectures , we made the source code related to this paper available online 1 . 2 RELATED WORK . Fully-connected MLPs are the most widely-used neural models for datasets in which no prior assumption is made on the relationship between features . However , due to the huge complexity of fully-connected layers , MLPs are prone to overfitting resulting in shallow architectures limited to a few layers in depth ( Goodfellow et al. , 2016 ) . Various techniques have been suggested to improve training these models which include regularization techniques such as L-1/L-2 regularization , dropout , etc . and normalization techniques such as layer normalization , weigh normalization , batch normalization , etc . ( Srivastava et al. , 2014 ; Ba et al. , 2016 ; Salimans & Kingma , 2016 ; Ioffe & Szegedy , 2015 ) . For instance , self-normalizing neural networks ( SNNs ) have been recently suggested as state of the art normalization methods that prevent vanishing or exploding gradients which help training feed-forward networks with higher depths ( Klambauer et al. , 2017 ) . From the architectural perspective , there has been great attention toward networks consisting of sparse connections between layers rather than having dense fully-connected layers ( Dey et al. , 2018 ) . Sparse connected neural networks are usually trained based on either a sparse prior structure over the network architecture ( Richter & Wattenhofer , 2018 ) or based on pruning a fully-connected network to a sparse network ( Yun et al. , 2019 ; Tartaglione et al. , 2018 ; Mocanu et al. , 2018 ) . However , it should be noted that the main objective of most sparse neural network literature has been focused on improving the memory and compute requirements while maintaining competitive accuracies compared to MLPs . As a parallel line of research , the idea of using expressive feature combinations or groups has been suggested as a prior over the feature domain . Perhaps , the most successful and widespread use of this idea is in creating random forest models in which different trees are trained based on different feature subsets in order to deal with high-dimensional and high-variance data ( Breiman , 2001 ) . More recently , feature grouping is suggested by Aydore et al . ( 2019 ) as a statistical regularization technique to learn from datasets of large feature size and a small number of training samples . They do the forward network computation by projecting input features using samples taken from a bank of feature grouping matrices , reducing the input layer complexity and regularizing the model . In another recent study , Ke et al . ( 2018 ) used expressive feature combinations to learn from tabular datasets using a recursive encoder with a shared embedding network . They suggest a recursive architecture in which more important feature groups have a more direct impact on the final prediction . While promising results have been reported using these methods , feature grouping has been mostly considered as a preprocessing step . For instance , Aydore et al . ( 2019 ) uses the recursive nearest agglomeration ( ReNA ) ( Hoyos-Idrobo et al. , 2018 ) clustering to determine feature groups prior to the analysis . Alternatively , Ke et al . ( 2018 ) defined feature groups based on a pre-trained gradient boosting decision tree ( GBDT ) ( Friedman , 2001 ) . Feature grouping as a preprocessing step not only increases the complexity and raises practical considerations , but also limits the optimality of the selected features in subsequent analysis . In this study , we propose an end-to-end solution to learn expressive feature groups . Moreover , we introduce a network architecture to exploit interrelations within the feature groups to reduce the network complexity and to train deeper representations . 1We plan to include a link to the source code and GitHub page related to this paper in the camera-ready version . 3 PROPOSED METHOD . 3.1 ARCHITECTURE OVERVIEW . In this paper , we propose GMLP which intuitively can be broken down to three stages : ( i ) selecting expressive feature groups , ( ii ) learning dynamics within each group individually , and ( iii ) merging information between groups as the network grows in depth ( see Figure 1 ) . In this architecture , expressive groups are jointly selected during the training phase . Furthermore , GMLP is leveraging feature groups and using local group-wise weight layers to significantly reduce the number of parameters . While the suggested idea can be materialized as different architectures , in the current study , we suggest organization of the network as architectures resembling a binary tree spanning from leaves ( i.e. , features ) to a certain abstraction depth closer to the root2 . As the network grows deeper , after each local group-wise weight layer , half of the groups are merged using pooling operations , effectively reducing the width of the network while increasing the receptive field . At the last layer , all features within all groups are concatenated into a dense feature vector fed to the output layer . 3.2 NOTATION . We consider the generic problem of supervised classification based on a dataset of feature and target pairs , D : ( x1 : N , y1 : N ) , where xi ∈ < d , yi ∈ { 1 . . . C } , and N is the number of dataset samples . Furthermore , we define group size , m , as the number of neurons or elements within each group , and group count , k , as the number of selected groups which are essentially subsets of input features . Also , L is used to refer to the total depth of a network . We use zli ∈ < m to refer to activation values of group i in layer l. In this paper , we define all vectors as column vectors . 3.3 NETWORK LAYERS . In this section , we present the formal definition of different GMLP network layers . The very first layer of the network , Group-Select , is responsible for organizing features into k groups of size m each . A routing matrix , Ψ , is used for connecting each neuron within each group to exactly one feature in the feature set : z01 : k = Ψx , ( 1 ) 2Please note that , in this paper , tree structures are considered to grow from leaves to the root . In other words , in this context , limiting the depth is synonymous with considering the tree portion spanning from a certain depth to leave nodes . where Ψ ∈ { 0 , 1 } km×d is a sparse matrix determining features that are present in each group . As we are interested in jointly learning Ψ during the training phase , we use the following continuous relaxation : Ψi , j ≈ exp ( ψi , j/τ ) ∑j′=d j′=1 exp ( ψi , j′/τ ) . ( 2 ) In this equation , ψ is a real-valued matrix reparameterizing the routing matrix through a softmax operation with temperature , τ . The lower the temperature , the more ( 2 ) converges to the desired discrete and sparse binary routing matrix . Note that , in the continuous relaxation , the matrix ψ can be optimized via the backpropagation of gradients from classification loss terms . In the next section , we provide further detail on temperature annealing schedules as well as other techniques to enhance the Ψ approximation . Based on selected groups , we suggest local fully-connected weight layers for each group : Group-FC . The goal of Group-FC is to extract higher-level representations using the selected expressive feature subsets . This operation is usually followed by non-linearity functions ( e.g. , ReLU ) , normalization operations ( e.g , Batch Norm ) , and dropout . Formally , Group-FC can be defined as : zl+1i = f ( W l iz l i + b l i ) , ( 3 ) where W li ∈ < m×m and bli ∈ < m are the weight matrix and bias vector , applied on group i at layer l. Here , f represents other subsequent operations such as non-linearity , normalization , and dropout . Lastly , Group-Pool is defined as an operation which merges representations of two groups into a single group , reducing network width by half while increasing the effective receptive field : zl+1i = pool ( z l i , z l i+k/2l+1 ) , ( 4 ) where zli and z l i+k/2 are the ith group from the first and second halves , respectively ; and pool is a pooling function from < 2m to < m . In this study , we explore different variants of pooling functions such as max pooling , average pooling , or using linear weight layers as transformations from < 2m to < m . Please note that while we use a similar terminology as pooling in convolutional networks , the pooling operation explained here is not applied location-wise , but instead it is applied feature-wise , between different groups pairs . The values of m and k are closely related to the number and order of feature interactions for a certain task . Using proper m and k values enables us to reduce the parameter space while maintaining the model complexity required to solve the task . However , finding the ideal m and k directly from a given dataset is a very challenging problem . In this work , we treat m and k as hyperparameters to be found by a hyperparameter search .

Summary: The paper describes an MLP architectures for problems in which the features do not have a known structure (eg, tabular data). A "differentiable routing matrix" partitions the data into K blocks. Then, standard MLPs are applied to each block and the results are recursively aggregated by moving forward in the model.