Triangular Dropout: Variable Network Width without Retraining

One of the most fundamental design choices in neural networks is layer width : it affects the capacity of what a network can learn and determines the complexity of the solution . This latter property is often exploited when introducing information bottlenecks , forcing a network to learn compressed representations . However , such an architecture decision is typically immutable once training begins ; switching to a more compressed architecture requires retraining . In this paper we present a new layer design , called Triangular Dropout , which does not have this limitation . After training , the layer can be arbitrarily reduced in width to exchange performance for narrowness . We demonstrate the construction and potential use cases of such a mechanism in three areas . Firstly , we describe the formulation of Triangular Dropout in autoencoders , creating models with selectable compression after training . Secondly , we add Triangular Dropout to VGG19 on ImageNet , creating a powerful network which , without retraining , can be significantly reduced in parameters . Lastly , we explore the application of Triangular Dropout to reinforcement learning ( RL ) policies on selected control problems . 1 INTRODUCTION . In this paper we describe Triangular Dropout , a novel dropout ( Srivastava et al. , 2014 ) technique that imparts a powerful property to fully-connected layers : the layer can be reduced in width after training to select a desired size of representation . This enables structured pruning in fully-connected layers such that the pruned network is densely connected . This exposes a trade-off between performance and network width that can be varied without retraining the network . Additionally , constructing and applying Triangular Dropout is straightforward . Triangular Dropout amounts to masking minibatches during training with a binary lower triangular matrix with zeros above the diagonal . After training , the relevant weights can be copied into a physically smaller dense network . Architecturally , Triangular Dropout represents a shift towards models that can be densely downsized to fit the needs of a particular application . For example , a wide model could be trained once on a computer cluster , and narrower versions would be immediately available for use on personal machines , mobile devices , IoT devices , or other embedded platforms . An image processing pipeline could have its frame-rate or feature descriptor length altered as needed , perhaps to process more samples in exchange for lower performance per sample . Triangular Dropout is tangent to many areas of research , and includes ideas of compression , structured network pruning , and variable computation . Our objective is not to outperform in any specific domain , but rather to demonstrate a technique that we find has useful properties from several areas and is very easy to implement . We discuss related works to Triangular Dropout in Section 2 , and formally define our mechanism in Section 3 . We then investigate some applications of Triangular Dropout in unsupervised learning of latent encodings ( Section 4 ) , supervised learning in larger models ( Section 5 ) , and reinforcement learning for locomotion control problems ( Section 6 ) . Sections 7 and 8 further discuss the characteristics of Triangular Dropout and suggest potential future research directions . 2 BACKGROUND . The width of a specific network layer is perhaps most critical in the domain of autoencoding ( Rumelhart et al. , 1986 ) , in which the width of the encoding layer ( along with its activation ) describes the amount of information retained in an encoded sample . The smaller the latent space ( narrower the encoding layer ) , the more compressed the learned representation , and therefore , the more lossy the compression performed by the network . There are countless variations of autoencoders which seek to improve on this compression ratio ( Theis et al. , 2017 ) or introduce additional properties to the latent variables ( Vincent et al. , 2008 ; Rifai et al. , 2011 ) . The degree to which the latent variables are independent is the subject of disentanglement research , which introduces further properties into the latent variables . β-VAE ( Higgins et al. , 2017 ) demonstrates that the gaussian distributions in a VAE ( Kingma & Welling , 2014 ) can be made increasingly orthogonal in exchange for reconstruction accuracy . The independence and ordering of learned features can also be enforced manually by methods such as PCAAE ( Pham et al. , 2020 ) , which learns one latent variable at a time , progressively widening the latent layer to learn additional features via additional training . A comparison to PCAAE is discussed in detail in Section 4 . Autoencoding makes use of a well-known tradeoff between network size and capability to encourage compression , a relationship that holds in other domains ( Amodei & Hernandez , 2019 ; Kaplan et al. , 2020 ; McCandlish et al. , 2018 ) . This is perhaps most well known with regards to ImageNet ( Deng et al. , 2009 ) , in which historically a clear correlation is seen between network size and performance ( summarized nicely in Figure 1 of Tan & Le ( 2020 ) ) . However , such large models can also be very inefficient in their use of parameters , often learning unnecessary weights or redundant representations . Network pruning is a technique to reduce an over-parameterized network down to essential parameters , recovering a sub-network that is similar or even superior in performance ( Blalock et al. , 2020 ; Frankle & Carbin , 2018 ) . In unstructured pruning , the network is sparse , whereas in structured pruning , the sub-network is dense . Triangular Dropout falls into the latter category , but examines fully-connected layers instead of the typical pruning of convolutional elements , as in Anwar et al . ( 2015 ) ( channels and kernels ) , Li et al . ( 2016 ) ( filters ) , and He et al . ( 2017 ) ( channels ) . Triangular Dropout additionally does not require finetuning and is not iterative , as in the above works and many others ( Molchanov et al. , 2016 ; Luo et al. , 2019 ) , although fine-tuning could of course be applied if desired . In Section 5 , we explore dense prunings recovered by Triangular Dropout on the classification head of VGG19 , an early ImageNet solution that is well-known to be overparameterized as evidenced by subsequent works with fewer parameters ( Iandola et al. , 2016 ; Tan & Le , 2020 ) . This overparameterization and history of use makes it popular for pruning studies , especially on ImageNet ( Blalock et al. , 2020 ) . There is also a great deal of relevant work from the conditional computation domain , exploring architectures that dynamically find appropriate sub-networks for a given input ( Han et al. , 2021 ) . In this approach networks are trained to enable selection of sub-networks , rather than simply pruning the network after initial training . Few approaches examine variable feed-forward layer width , for example Bengio et al . ( 2015 ) use reinforcement learning to learn which nodes in a fully-connected layer can be masked out on a per-sample basis . Ba & Frey ( 2013 ) achieve a similar effect through deep belief networks creating layer masks . Most methods rely on selective routing through entire sub-branches or sub-modules , i.e . Sabour et al . ( 2017 ) . There is a related area of work in recurrent networks which adapts computation in the time ( rollout ) dimension during inference , mainly by choosing where and when to update a hidden state ( Graves , 2016 ; Campos et al. , 2017 ; Jernite et al. , 2017 ) . In contrast to these areas , Triangular Dropout specifically seeks dense representations that capture the entirety of the input space , and does so in a non-recurrent manner . Network size selection has additional repercussions in the deep reinforcement learning ( DRL ) domain . Works like amplified control ( Paulhamus et al. , 2019 ) and robust predictable control ( Eysenbach et al. , 2021 ) seek compressed policies to perform tasks , using information bottlenecking ( Tishby et al. , 2001 ) to enforce this compression . Robust predictable control demonstrates that compression creates policies that are more risk-averse and potentially more transferable . Such compressability is further related to task complexity measurement , an open problem ( see the implications of task complexity discussed in Srivastava et al . ( 2019 ) ) . We explore the application of Triangular Dropout to DRL in Section 6 . 3 TRIANGULAR DROPOUT . Triangular Dropout is simple to implement in practice . For a fully-connected layer of width n , processing a minibatch x of B datapoints , the output y has size ( B , n ) . During training , standard dropout ( with probability p ) multiplies the output element-wise by a random binary mask M of the same size as the output : y = M ⊙ α ( xTW + b ) where M ∼ Bernoulli ( p ) where W and b are the layer weights and biases , respectively , and α is some activation function . Triangular Dropout has the same form , but the mask M is populated as a binary lower triangular matrix instead of a randomly sampled one : y = MTri ⊙ α ( xTW + b ) where MTri = 1 0 . . . 0 1 1 . . . 0 ... ... . . . ... 1 1 . . . 1 This is visualized in Figure 1 . In the case where B > n , the rows of MTri are simply repeated to reach the desired size . In the case where B < n , a triangular block matrix may be used . For any B , an approximate alternative would be to independently sample rows such that a random number of 1 ’ s are followed by 0 ’ s . The implications of a triangular mask can be viewed in different ways . Considering the rows of the output y , the mask creates a set of datapoints which have been processed with the full range of layer widths from 1 to n. If the batch size is equal to n , the ith row represents a training sample in which the effective layer width is i . Training now amounts to something like a multitask learning problem , in which the “ tasks ” are processing data with varying network widths . We have an even distribution of data from all these tasks , and the loss function reflects training over this set of architectures ( widths ) : y = y1,1 0 0 . . . → effective width 1 y1,2 y2,2 0 . . . → effective width 2 y1,3 y2,3 y3,3 . . . → effective width 3 ... ... ... . . . Taking this multitask view , we do not re-scale the layer outputs as in normal dropout ; we want the downstream network to be able to handle any of the ” tasks ” ( any of the potential widths of this layer ) . However , the “ tasks ” in this view are not independent . Examining the columns of the output , the jth column represents the output of the jth layer node , masked out for the first j rows . Because the mask is triangular , all of the samples which are non-zero at column j are necessarily also non-zero at columns 1 through j − 1 . That is , parameters relevant to the jth node are only updated for samples in which the previous j − 1 nodes ( smaller widths ) are also updated . This is not true for subsequent nodes : nodes j + 1 and onwards may or may not be masked when node j is utilized . This cascading dependence is what masks Triangular Dropout work . The parameters related to node j can be learned with a guarantee that the preceding j− 1 outputs in terms of width are also present . However , they can not rely as heavily on outputs j + 1 and beyond , and are essentially restricted from fully co-optimizing with these outputs . During test time , it is up to the user to decide on the masking strategy . Providing a mask filled with ones makes the layer approximately equivalent to a typical fully-connected layer , or masking out the last N − j nodes causes the layer to behave as though it has width j . To permanently ablate the layer down to some desired width , the relevant model parameters ( portions of W and b from this layer and the next ) simply need to be copied into a smaller model , as shown in Figure 1 .

Summary: This paper proposes Triangular Dropout, a mechanism for training fully-connected layers whose width can be decreased at test time, by using dropout masks with a particular structure at train time. The proposed approach compares favorably to previous methods when training autoencoders of varying bottleneck size on MNIST. Authors show that Triangular Dropout can also be used to tune the parameter count - accuracy trade-off at test time on VGG-19. Finally, the proposed method is used to measure task complexity in RL by using the minimum layer width required to solve the task as a proxy.