Triangular Dropout: Variable Network Width without Retraining
Authors: Edward W Staley
Corban G Rivera
Neil Joshi
Venue: ICLR 2022 Submitted (2022)
Keywords: architecture
compression
variable network
neural network design
deep learning

Abstract: One of the most fundamental design choices in neural networks is layer width: it affects the capacity of what a network can learn and determines the complexity of the solution. This latter property is often exploited when introducing information bottlenecks, forcing a network to learn compressed representations. However, such an architecture decision is typically immutable once training begins; switching to a more compressed architecture requires retraining. In this paper we present a new layer design, called Triangular Dropout, which does not have this limitation. After training, the layer can be arbitrarily reduced in width to exchange performance for narrowness. We demonstrate the construction and potential use cases of such a mechanism in three areas. Firstly, we describe the formulation of Triangular Dropout in autoencoders, creating an MNIST autoencoder with selectable compression after training. Secondly, we add Triangular Dropout to VGG19 on ImageNet, creating a powerful network which, without retraining, can be significantly reduced in parameters with only small changes to classification accuracy. Lastly, we explore the application of Triangular Dropout to reinforcement learning (RL) policies on selected control problems, showing that it can be used to characterize the complexity of RL tasks, a critical measurement in multitask learning and lifelong-learning domains.

Under review as a conference paper at ICLR 2022
TRIANGULAR DROPOUT: VARIABLE NETWORK WIDTH WITHOUT RETRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
One of the most fundamental design choices in neural networks is layer width: it affects the capacity of what network learn and determines the complexity the solution. This latter property is often exploited when introducing informa- tion bottlenecks, forcing network learn compressed representations. However, such an architecture decision is typically immutable once training begins; switch- compressed architecture requires retraining. In this paper we present a new layer design, called Triangular Dropout, which does not have this limita- tion. After training, the layer can be arbitrarily reduced in width to exchange performance for narrowness. We demonstrate the construction and potential use cases of such mechanism in three areas. Firstly, we describe the formulation of Triangular Dropout in autoencoders, creating models with selectable compres- sion after training. Secondly, we add Triangular Dropout VGGI9 on ImageNet, creating powerful network which, without retraining, be significantly duced in parameters. Lastly, we explore the application of Triangular Dropout to reinforeement learning (RL) policies selected control problems.
ing to
INTRODUCTION
In this paper we describe Triangular Dropout, novel dropout (Srivastava et al. (0044) technique that imparts powerful property to fully-connected layers: the layer can be reduced in width after training to select a desired size of representation. This enables structured pruning in fully-connected layers such that the pruned network is densely connected. This exposes trade-off between perfor- mance and network width that can be varied without retraining the network. Additionally, construct- ing and applying Triangular Dropout is straightforward. Triangular Dropout amounts to masking minibatches during training with a binary lower triangular matrix with zeros above the diagonal. After training, the relevant weights can be copied into physically smaller dense network.
Architecturally, Triangular Dropout represents shift towards models that can be densely downsized fit the needs of partieular application. For example, wide model could be trained once on computer cluster, and narrower versions would be immediately available for use on personal ma- chines, mobile devices, IoT devices, or other embedded platforms. image processing pipeline could have its frame-rate or feature descriptor length altered as needed, perhaps to process more samples in exchange for lower performance per sample. Triangular Dropout is tangent to many areas of research, and includes ideas of compression, struc- tured network pruning, and variable computation. Our objective not to outperform in any specific domain, but rather to demonstrate technique that find has useful properties from several areas and is very easy to implement.
We discuss related works to Triangular Dropout in Section 2 and formally define our mechanism in Section 3 We then investigate some applications of Triangular Dropout in unsupervised learning of latent encodings (Section 4. supervised learning in larger models (Section 5). and reinforce- ment learning for locomotion control problems (Section (6). Sections 7 and R further discuss the characteristics of Triangular Dropout and suggest potential future research directions.

---

Under review as a conference paper at ICLR 2022
2 BACKGROUND
The width of specific network layer is perhaps most critical in the domain of autoencoding (Rumel- hart et al. 1986). in which the width of the encoding layer (along with its activation) describes the amount of information retained in an encoded sample. The smaller the latent space (narrower the encoding layer), the more compressed the learned representation, and therefore, the more lossy the compression performed by the network. There are countless variations of autoencoders which seek to improve this compression ratio (Theis al. (0017) introduce additional properties to the latent variables (Vincent et al. 2008 Rifai et 2011).
The degree to which the latent variables independent is the subject of disentanglement research, which introduces further properties into the latent variables. B- VAV (Higgins et al., 2017) demon- strates that the gaussian distributions in VAE (Kingma & Welling 2014) can be made increasingly orthogonal in exchange for reconstruction accuracy. The independence and ordering of learned fea- tures can also be enforced manually by methods such as PCAAE (Pham et al., 2020), which learns one latent variable at a time, progressively widening the latent layer to learn additional features via additional training. A comparison to PCAAE discussed in detail in Section
Autoencoding makes use of a well-known tradeoff between network size and capability to encourage compression, a relationship that holds in other domains (Amodei & Hernandez, 2019 Kaplan al.] 2020 McCandlish et al., 2018). This is perhaps most well known with regards to ImageNet (Deng) ct al. 2009). in which historically a clear correlation is seen between network size and performance (summarized nicely in Figure 1 of Tan & Le 20200))
However, such large models can also very inefficient in their use of parameters, often learning unnecessary weights redundant representations. Network pruning is technique to reduce an over-parameterrized network down to essential parameters, recovering a sub-network that is similar or even superior in performance (Blalock et al. 2020: Frankle & Carbin 2018). In unstructured pruning, the network is sparse, whereas in structured pruning, the sub-network is dense. Triangular Dropout falls into the latter category, but examines fully-connected layers instead of the typical pruning of convolutional elements, as in Anwar et al. (2015) (channels and kernels), (ernels),Liitt al. (2016) (filters), Hed al. (2077) (channels). Triangular Dropout additionally does not require finetuning and is not iterative. as in the above works and many others (Molchanvv et al. 2016: Luo et al. 2019). although fine-tuning could of course be applied if desired. In Section 5 we explore dense prunings recovered by Triangular Dropout on the classification head of VGG19, an early ImageNet solution that well-known to be overparameterized as evidenced by subsequent works with fewer
parameters (landola et al., 2016 Tan & Le 2020). This overparameterizztion and history of use makes it popular for pruning studies, especially on ImageNet (Blalock et al. 2020).
There is also great deal of relevant work from the conditional computation domain, exploring architectures that dynamically find appropriate sub-networks for a given input (Han et al. [00TT. In this approach networks trained to enable selection of sub-networks, rather than simply pruning the network after initial training. Few approaches examine variable feed-forward layer width, for example (engioo al. (2015) reinforcement learning to learn which nodes in fully-connected layer can be masked out on a per-sample basis. Ba & Frey (2013) achieve a similar effect through deep belief networks creating layer masks. Most methods rely on selective routing through entire sub-branched or sub-modules, i.e. (Sabour et al. (2017). There is a related area of work in recurrent networks which adapts computation in the time (rollout) dimension during inference, mainly by choosing where and when to update a hidden state (Graves 2016 Campos et al. 2017 2017). In contrast these areas, Triangular Dropout specifically seeks dense representations that capture the entirety of the input space, and does so in non-recurrent manner.
2017]Jermit Jernite et al.,
Network size selection has additional repercussions in the deep reinforcement learning (DRL) do- main. Works like amplified control (Paulhamus et al.] 2019) and robust predictable control (Ey- senbach et al. 2021) seek compressed policies to perform tasks, using information bottleneck- ing (Tisbyy et al. 2001) to enforce this compression. Robust predictable control demonstrates that compression creates policies that are more risk-averse and potentially more transferable. Such com- pressability is further related to task complexity measurement, open problem (see the implications of task complexity discussed in Srivastava et al. (2019)). We explore the application of Triangular Dropout to DRL in Section 6
2

---

Under review as a conference paper at ICLR 2022
3 TRIANGULAR DROPOUT
Triangular Dropout is simple implement in practice. For fully-connected layer of width n, processing a minibatch x of B datapoints, the output has size (B,n). During training, standard dropout (with probability p) multiplies the output element-wise by random binary mask M of the same size as the output:
= M + bb where M ~ (eBnoulll(ll))
where W and b are the layer weights and biases, respectively, and is some activation function. Triangular Dropout has the same form, but the mask M is populated as binary lower triangular matrix instead of randomly sampled one:
1
0
y MIR O aa(x"W W bb where Mr
:
This is visualized in Figure In the case where B > n, the rows of Mrri simply repeated to reach the desired size. In the case where < n, a triangular block matrix may be used. For any B, an approximate alternative would be to independently sample rows such that random number of I's are followed by O's.
The implications of triangular mask can be viewed in different ways. Considering the rows of the output the mask creates set of datapoints which have been processed with the full range of layer widths from n. the batch size equal n, the ith row represents a training sample in which the effective layer width is i. Training now amounts to something like a multitask learning problem, in which the "tasks" are processing data with varying network widths. We have an even distribution of data from all these tasks, and the loss function reflects training over this set of architectures (widths):
V1,1
0
effective width I effective width 2
/1,2
/2,2
0
y
/1,3
12,3
V3,3
effective width 3
Taking this multitask view, do re-scale the layer outputs as normal dropout; the downstream network to be able to handle any of the "tasks" (any of the potential widths of this layer).
However, the "tasks" in this view are not independent. Examining the columns of the output, the jth column represents the output of the jth layer node, masked out for the first rows. Because the mask is triangular, all of the samples which are non-zero at column j are necessarily also non-zero at columns 1 through j 1. That is, parameters relevant to the jth node are only updated for samples in which the previous
1 nodes (smaller widths) are also updated. This is not true for subsequent nodes: nodes j + and onwards may or may not be masked when node ý is utilized.
This cascading dependence is what masks Triangular Dropout work. The parameters related to node j can learned with a guarantee that the preceding j - outputs in terms of width are also present. However, they cannot rely heavily outputs j + 1 and beyond, and are essentially restricted from fully co-optimizing with these outputs.
During test time, it is up to the user to decide on the masking strategy. Providing mask filled with ones makes the layer approximately equivalent to typical fully-connected layer, or masking out the last N j nodes causes the layer behave though it has width j. To permanently ablate the layer down to some desired width, the relevant model parameters (portions of W and b from this layer and the next) simply need to be copied into smaller model, shown in Figure
3

---

Under review as a conference paper at ICLR 2022
Train:
Test:
Wittth -2 twelghts masitee)
Width =2 (welghts copted)
Layer Width,
Batch Size, B
Ut propped Quiputs
Figure Diagram Triangular Dropout application training (Left) and used during test or deployment (Right). During test, layer width can be simulated by masking the layer output, realized by copying relevant weights into a narrower architecture.
VARIABLE COMPRESSION IN AUTOENCODERS
To further detail the properties of Triangular Dropout and compare it to alternative methods, demonstrate the construction an autoencoder with a selectable latent size. We train an autoen- coder on MNIST (Deng 2012) with latent size and use Triangular Dropout on the latent layer. After training. we can select any level of compression S n and receive both a valid encoding and reconstruction. We also train autoencoders the CelebA dataset (Liu et al. 2015) (image size reduced to 64x64) in order to explore Triangular Dropout on a more difficult autoencoding task.
4.1 BASELINES
We compare MNIST results quantitatively (reconstruction loss) and qualitatively (visualization of reconstruction) to two baselines: autoencoders that trained for specific latent sizes through n, and PCAAE, which trains an autoencoder of size n progressively. PCAAE trains one latent variable time and includes correlation minimization loss term. PCAAE also trains separate encoder for each latent variable, and a new decoder for each width. This means that, like the set of standard autoencoders. PCAAE requires training runs.
Compared to standard autoencoders, our expectation is that Triangular Dropout must sacrifice some compression exchange for the introduction of some one-way dependence among features (the first i features are not very dependent on the i + 1th feature). Conversely, we would not expect a typical autoencoder have this property. The encoded features from typical autoencoder, having no encouragement to be independent or disentangled, should only be meaningful when the full feature descriptor is retained.
We would expect that PCAAE further sacrifices reconstruction loss for even more stringent inde- pendence among learned features. However, the features are independent, reconstructions may still be meaningful when less important features are removed from the final model.
4.2 EXPERIMENTS AND RESULTS
A convolutional autoencoder with latent size 32 was trained with PCAAE and Triangular Dropout to reconstruct MNIST digits. The smaller widths of PCAAE were saved off during training. Addition- ally, standard autoencoders were trained for sizes through 32. We examine the reconstruction loss of the models as the latent width increased during training and decreased via ablation of the final 32-width model. Triangular Dropout falls into the latter category by design, but has performance similar to the checkpoint models trained at specific sizes. These results are shown in Figure 2 Full architecture and training details available in the Appendix.
4

---

Under review as a conference paper at ICLR 2022
Separate
Ablated
PCAAE Checkpts
Ablated
Triangular Drop
Reconstructin
0.30
0.25
0.20
0.15
0.10
10 15
20
25
30
Encoding Size,
Figure 2: Reconstruction loss versus encoding size for a variety of methods. Triangular Dropout (red) most similar to independent autoencoders trained at specific encoding sizes (black). PCAAE models produced during progressive training are shown in blue. Light grey and light blue show the reconstruction loss of the full-size standard autoencoder and PCAAE models, respectively, when the latent space is ablated down given size.
Notably, our hypotheses regarding feature independence reconstruction loss hold true: for a given latent size z, the most performant model (without feature ablation) ) is a standard autoencoder trained that width, the least performant is PCAAE which introduces correlation loss, and somewhere between is Triangular Dropout. The baseline models with ablated encodings do not have comparable reconstruction losses unless the ablation is very minor (most latent features retained). Triangular Dropout, trained only at size 32, can be ablated down to very small sizes while only performing slightly worse than individual autoencoders at those sizes. Triangular Dropout roughly encompasses the superset of these autoencoders, while only requiring single training run.
Figure 3 shows the qualitative reconstruction each of these cases, for selected latent widths. The standard autoencoders trained at increasing sizes (Figure 3 A) show increasing reconstruction fi- delity, with recognizable digits appearing around 4-6 latent features. When removing features from the size 32 autoencoder (Figure B), the encoding are clearly corrupted unless most features present. PCAAE (Figure 3 C and D) has only similar reconstructions to the standard autoencoders sufficient features present, and can also content with significant ablations. Finally, Trian- gular Dropout (Figure 3 E) has similar reconstructions to the individual autoencoders, even though its latent size is determined by ablation. A user of this model could select some latent size approxi- mately between 6 and 32 and expect it to yield recognizable digits.
We also explored the application of Triangular Dropout the CelebA dataset, and find similar effects. This model had a latent size of 128. Full architectural and training details are available in the Appendix. As seen in Figure F. the most compressed features of the autoencoder also capture the most significant features, with additional detail captured by subsequent outputs. We also noted that Triangular Dropout may have unusual behavior with batch normalization (loffe Szegedy) 2015) or variational encodings (Kingma & Welling 2014), which shift or redefine outputs with value of zero. These artifacts are primarily seen when width severely reduced. Reconstructions using these architectures are available in the Appendix.
5 PARAMETER REDUCTION IN LARGE NETWORKS
The results of our MNIST autoencoding experiments demonstrate the primary property of a Trian- gular Dropout layer: it can be reduced in width after training and still provide meaningful output. We now consider the ramifications of such a mechanism in the supervised learning domain.
Many powerful models for image feature extraction and natural language cannot be easily utilized by the average practitioner at their full sizes. To make them more accessible and study the effects of scale, multiple specific sizes are often trained. For example, VGG (Simonyan
large that they
5

---

Under review as a conference paper at ICLR 2022
(A) Individual Autoencoders
(8) Ablated Autoencoder N=32
7 171
7771 7 7 7 7
4
>
2
>
P
A
J 7 4 2
7 2 9 0 4
2
2
2
2
6
6
6
6
6
2. %
2
9
9
9
2
R
R
7
9 6 4
0
0
0
0
O
O
O
O
O
O
O
1
7
7
2
7
7
2
(E)
Triangular Dropout Autoencader
4
a
2
2
2
4
4
7 7 7 7
1=1
12
20
8 3
2
2
2
2
2
2
2
Z
2
9
9 9 9
4
4
(c) PCAAE Checkpoints
(D) Ablated PCAAE N=32
0 0
0
O
6
O
7
7
7
7
7
7
7
2 2 2
7 7
7 7 7
7
7
7
7
7
7
2
8 8
9
4
4
4
4
4
4
4
4
4
4
2
2
2
2
3
0
3
2
5
2
2 2
2 0
2
2
4
Q
Q
C O
9 O
6 O
9 O
9 O
10
12
16
24
0
0
0
0
0
O
O
O
O
O
0
O 4
2
0
0
O
O
O
2
2
8
7
4
4
4
4
4
10
ref
(F)
Triangular Dropout Autoencoder on CelebA 64x64
1
2
6
6
10 12
16
24
32
64 96
112
120
ref
Figure 3: Qualitative analysis of MNIST reconstructions for all methods. Colored borders corre- spond to line color in Figure 2 A: Reconstructions from individual autoencoders at selected sizes, with original inputs on the right. B: Width 32 autoencoder with latent features ablated to desired size. C: Reconstructions from PCAAE checkpoint models produced by progressively training latent channels towards full width of 32. D: Width 32 PCAAE with latent features ablated to desired size. E: Reconstructions from Triangular Dropout trained once at size 32, with smaller size encod- ings created via ablation.
& Zisserman 2015) is often used at one of 4 sizes (VGG11, VGG13, VGG16, and VGG19), and so GPT-2 (Radford et al. (2019) (sizes small, medium, large, and extra large). We envision Triangular Dropout as stepping stone towards single master models that can be reduced in size arbitrarily to fit the needs of any given user. In this manner, Triangular Dropout could be considered a pruning technique in which the model is trained in preparation for structured pruning.
To motivate this view, we retrain the fully connected classification portion of VGG19 on ImageNet using Triangular Dropout for the two hidden layers of size 4096. We then investigate the fidelity of the model when run at reduced widths. We concentrate on this portion of the model because is folly-comnected and contains the majority of VGG's parameters (roughly 123 million out of 143 million in total). Specific training details are available in the appendix.
5.1 MODEL ACCURACY FINDINGS
VGG19 retrained with Triangular Dropout does not achieve quite the accuracy of the original VGG19, achieving 69.7 percent accuracy when not reduced in width, compared to 72.4 percent accuracy of the original VGG19 model. This may or may not significant depending on the appli- cation; this would be considered significant difference in often-used benchmark like ImageNet, but in a unique real world problem may be acceptable in exchange for the properties that Triangular Dropout imparts to the network.
Additionally, the success of the model with two layers of Triangular Dropout indicates that Triangu- lar Dropout layers can potentially be stacked in sequence create multi- layer MLP with adjustable width. In Section 6 we investigate a network with 3 hidden layers, all of which utilize Triangular Dropout.

---

Under review as a conference paper at ICLR 2022
5.2 REDUCED WIDTH
Our primary finding is that Triangular Dropout allows the classification portion of VGG19 to be reduced in width after training, resulting in a dense sub-network that trades accuracy for narrow- ness. This a known tradeoff from prior work in network pruning (see pruning results for VGGI6 on ImageNet in Blalock et al. (2020), figure 3), which Triangular Dropout makes available without in- troducing sparsity, requiring fine-tuning, iterative training. Our results using full-batch Triangular Dropout (batch size 4096) are summarized in Table T and graphically in Figure with the curve in bold. As seen in this plot, accuracy of around 50 percent can be achieved with only 1 percent of the original classifier parameters. This corresponds to hidden layers that are roughly 10 percent wide in the full model. The hidden layers can be reduced in size by half and only incur 0.1 percent loss in classification accuracy. This corresponds to ablating more than half of the classifier parameters, and roughly 46 percent of all the parameters in VGG19.
5.3 EFFECTS OF BATCH SIZE
Since training with such a large batch size is not always practical for large image models, we also explore training with batch sizes that are fractions of the full-size batches. In this setting, the tri- angular mask takes the form of a block matrix with blocks of size row by d columns, where d = 4096/B. The width ablation curves are shown in Figure 4 The primary result is that the full-size batches appear the most stable and perhaps best-performing, but all batch sizes have similar trends of accuracy vs width. Notably, this tradeoff does seem begin breaking down with very small batches, as seen by batch sizes 128 and 256 having irregular curve profiles.
Ablated Width
Accuracy
Classif. Params
Param, Reduction
VGG19 Reference
72.4%
123,642,856
N/A
4096 (full)
69.7%
123,642.856
0.0%
2048
69.6%
57.627.624
53.4%
1024
69.2%
27.765,736
77.3%
512
67.8%
13,621,224
89.0%
256
65.3%
6,745.376
94.3%
128
61.7%
3,356,904
97.3%
54.9%
1664,8556
98.6%
32
38.8%
836,904
99.3%
Table I: Accuracy and parameter numbers for VGG19 trained with Triangular Dropout, with various layer widths after training. The first row shows reference values for the original VGG19 architecture.
COMPRESSED RL POLICIES
In this section, we explore the application of Triangular Dropout to sequential decision making. Specifically, we first train RL policies on four locomotion control tasks from the MuloCo (Todorov eet al., 2012) environment suite within OpenAl Gym (Brockman et al.) (016). We then then distill the learned policies into secondary networks that use Triangular Dropout, either a single hidden layer or for all hidden layers. We examine the performance of these distilled policies for different levels of width reduction, providing measures for both the compressibility of these policies and the complexity of the tasks they solve. Finally, select compressed architecture that still performs well for each task and evaluate its performance when retrained from scratch.
We examine four environments: Hopper-v3, HalfCheetah-v3, Walker2d-v3, and Ant-v3, all of which come from OpenAl Gym's MuJoCo environment collection. We selected these tasks because they have similar objectives (locomote forward the highest speed possible), but may differ in complex- ity.
6.1 POLICY DISTILLATION WITH TRIANGULAR DROPOUT
We first determine if Triangular Dropout can be used to create variable-width policy networks, and compare the performance of the ablated policies to the original experts. We also wish to explore the compressibility of the policy (how much it can be reduced in width without severely affecting
7

---

Under review as a conference paper at ICLR 2022
0.7
0.6
Validation Top-1 Accuracy
0.3
VGG19 Original
0.2
B-256
B=512
0.1
B=1024
B-2048
0.0
409 (full)
0.001
0.010
0.100
1.000
Fraction Retained Parameters
Figure 4: Validation accuracy of VGG19 with Triangular Dropout the two hidden layers in the classification MLP are modified in width after training. Several batch sizes are tested, reduced from the full-sized square batch (bold). For comparison, the reported validation accuracy of the original VGG19 also shown (teal).
performance) is potential metric of task complexity, an important and open problem reinforce- ment learning, multitask learning, and meta-lcarning. We do not use Triangular Dropout in RL directly because the masking changes the policy's s behavior during data collection.
For each environment, we first train an expert policy (three fully-connected layers of size 48) us- ing proximal policy optimization (PPO) (Schulman et al.] 2017), and then imitate them through supervised learning. To imitate these expert policies, first gather (state, action) pairs from sev- eral thousand evaluation runs with occasional random actions taken to prevent over-sampling of the most likely trajectories. We randomly down-select to 100,000 data points and train imitate this policy in a supervised fashion. The new model consists of 3 fully-connected layers of size 48, with either the middle layer using Triangular Dropout, or all layers. After training, we can reduce the width of the Triangular Dropout layers to enforce a compression in the mapping of states to actions. Training details of both the experts and imitation networks are available in the Appendix.
model to
We examine the performance of the ablated policies by finding the smallest layer widths that main- tain least 90% performance (speed) compared to the original experts. We measure performance in terms of average top speed over 100 episodes, which is smoother metric than raw reward (and the primary goal of the environments). We find that in the case with single middle Triangu- lar Dropout layer, the policies for Hopper, HalfCheetah, Walker2d, and Ant can be compressed to middle bottlenecks of 8, 2, 2, and 7 respectively. For imitation networks consisting only hidden Triangular Dropout layers, we find widths of 8, 6, 6, and 9 recover at least 90% of the expert per- formance, respectively. This implies that Hopper and Ant may be more complex than HalfCheetah and Walker, since they seem to require more parameters in their solutions. It is also interesting that HalfCheetah and Walker2d can be performant with a bottleneck of size 2, which is far less than the action space of these environments (size 6). As described in Paulhamus al. (2019), this may imply subspace of possible actions.
that the solutions reside in
6.2 COMPARISON TO RETRAINED ARCHITECTURES
Finally, we take the 90% performance ablated architectures and retrain them from scratch in the same manner the original experts. Here the reduced Triangular Dropout layers have been re- placed with fully-comnectdd layers of the reduced size. Comparing the retrained experts with the original expert policies, the retrained experts had comparable (and in some cases better) top speeds to the originals, despite their vastly reduced network size. Compared to the reduced networks with Triangular Dropout (before retraining), there cases in which retraining yields performance gains,
8

---

Under review as a conference paper at ICLR 2022
but this not always true. For some problems simply reducing the policy width after imitating with Triangular Dropout will maintain performance.
A detailed summary of the ablation experiments in this section shown in Figure 5 RL training curves and hyperparameters are available in the appendix, along with numerical values for top speed and average reward of all policies.
Architecture
Hopper-v3
HalfCheetah-v3
Walker2d-v3
Ant-v3
action
48x48
Top Speed w
T. Drop 48
48x48
state
18
12
18
12
18
12
18
24
action
Drop 48
T. Drop
T. Drop 48
state
18
18
12
10
12
Ablated Width
Legend:
Speed vs. Abbated Width
Expert Speed
Speed
Retrained
Figure 5: Curves describe imitator performance on each environment as the Triangular Dropout layers are reduced in width after training, using architectures shown the far left. We only show widths 1 through 24 for readability. Horizontal lines indicate reference performance of original experts (black), ablated policies with least 90% expert performance (teal), and retrained policies using the compressed architectures (magenta).
7 DISCUSSION
Throughout our experimentation, we found several characteristics of Triangular Dropout that were surprising. Firstly, the results indicate that features learned with Triangular Dropout indeed one- way dependent. That is, the n + 1th feature relies on the nth to be meaningful, but not the other immediately clear that this property should form inherently, since for given
way around. training batch there are many examples for which adjacent features are both optimized.
significant finding was that Triangular Dropout seems encourage compression in leamed features, enabling structured pruning after training. In many cases the network used far less of its available width than we had allocated, and our surprise did not spread representation over all available resources. We believe Triangular Dropout to be an interesting mechanism that could lead to networks which are self-organizing and variable size. In addition to the useful properties discussed above, network that can be pruned to dense sub-networks without retraining has many practical benefits, especially as Al models get larger and mobile and embedded platforms become increasingly widespread.
Perhaps
FUTURE DIRECTIONS
There are several promising directions in which to continue this work. Triangular Dropout has only been designed and tested for fully-connected layers, and immediate future work could exam- ine reforming Triangular Dropout for convolutions transformers. Most prior work in structured pruning has focused on convolutions. Additionally. Triangular Dropout allows a network layer to be resized in width, but not depth. A corresponding mechanism which would allow for variable-depth networks post-training would be similarly useful, and very powerful when combined with variable width. Investigating solutions for applying Triangular Dropout directly to on-policy RL would also be worthwhile, as we believe our technique holds promise for developing compressed policies.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Dario Amodei and Danny Hernandez. Ai and compute. 2019. URL htwp:httttt/ppoeeeoo /ppena comm
blog/a ali and- comput e/
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of convolutioal neural networks. CoRR,
2015.
Jimmy Ba and Brendan Frey. Adaptive dropout for training deep neural networks. In
C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 26. Curran Asso- ciates, Inc., 2013. URL atttps://proceedaagg....
pdf
Emmanuel Bengio, Pierre-Lue Bacon, Joelle Pineau, and Doina Precup. Conditional computaaion in neural networks for faster models. CoRR, abs/1511.002977, 2015. [RLLattpp//axxivoor///
abs/1511 06297
Davis W. Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John V. Guttag. What is the state of neural network pruning? CoRR, abs/2003.03033. 2020. /ttpsstts://arxiv.
abs/2003 03033
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Victor Campos, Brendan Jou, Xavier Giró-i-Nieto, Jordi Torres, and Shih-Fu Chang. Skip RNN: learning to skip state updates in recurrett neural networks. CoRR, 02//1708.06344 2017. URL
http: /ttp:////xxvva wwwpwrwwwwww..../eooeee..roers/e 708 06834
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference compuler vision and pattern recognition, pp. 228-2482255 255. leee, 2009.
Li Deng. The mnist database handwritten digit images for machine learning research. JEEE Signal Processing Magazine, 29(6):111-14,, 2012.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Robust predictable control, 2021.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Training pruned neural net-
works. CoRR, abs/1 00365,, 2018. Uttp://:://axxi org/abs//////////// /ttabb/18333 03635
Alex Graves. Adaptive computation time for recurrent neural networks. CoRR, abs/1603.08883, 2016. URL/ttpp://axx....rr//ssss http://aabb//13.0883.33
Yizeng Han, Gao Huang, Shiji Song. Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey, CoRR, abs2212.000009 2021. URU/ttpp::/axxvvvoor///aas 04906
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. CoRR, abs/1707.06168, 2017. htllittt://axxvvoooog htapg//bb//1000 06168
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
Forrest N. landola, Song Han, Matthew W. Moskewicz. Khalid Ashraf, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and j0.5mb model size, 2016.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03167, 2015. URL htlt:://aa//iii .org/ abs/ 1502. 03167
Yacine Jernite, Edouard Grave, Armand Joulin, and Tomas Mikolov. Variable computation in recur- rent neural networks, 2017.
10

---

Under review as a conference paper at ICLR 2022
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Diederik Kingma and Max Welling. Auto-encoding variational bayes, 2014.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. CoRR, 0aas/1608.08710 2016. URL http:://axxiv.rrg/ wtwpg////////////////////////
08710
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Jian-Hao Luo, Hao Zhang, Hong- Yu Zhou, Chen-Wei Xie, Jianxin Wu, and Weiyao Lin. Thinet: Pruning enn filters for thinner net. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 41(10):5525-2533 2019, doi: 10.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training, 2018.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient transfer learning. CoRR, M2//1611.0644/ 2016. URL
http:////gaabaa..a.o../ L 06440
B. L. Paulhamus, Edward W. Staley, Corban G. Rivera, Kapil D. Katyal, and Chien-Ming Huang. Amplified control for robotic teleoperation. 2019.
Chi-Hieu Pham, Saïd Ladjal, and Alasdair Newson. Pcaae: Principal component analysis autoen- coder for organising the latent space generative networks, 2020.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dor- mann. Stable baselines3, httpp::/gittub.ccmm/o////// Co. http:/taabebbaseeeeeeee 2019.
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto- encoders:
Explicit invariance during feature extraction. In Proceedings of the 28th International Conference International Conference Machine Learning, ICML pp. 833 840, Madison,
WI, USA, 2011. Omnipress. ISBN 9781450306195.
D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning Internal Representutions by Error Propagation, pp. 318-362. MIT Press, Cambridge, MA, USA, 1986. ISBN 026268053X.
Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton. Dynamic routing between capsules. CoRR, abs/1710.09829 2017. uttptt::://axiv.co http://wwxwvv.oooab////i/o absp//ls..11...... 09829
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):129-1198, 2014. URL
9rivastaval4a.htmm
Shivangi Srivastava, Maxim Berman, Matthew B. Blaschko, and Devis Tuia. based lifelong learning. CoRR, abs/1907.0966 2019. URL. http:://rxxiv.org////0
0907.0966
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks, 2020.
11

---

Under review as a conference paper at ICLR 2022
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autoencoders, 2017.
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. Pro- ceedings of the 37th Allerton Conference on Communication, Control and Computation, 49, 07 2001.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5000 5533,, 2012. doi: 10. ww.11099/ROS.2012668660000
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. ICML '08, pp. (096-103,, New York, NY. USA, 2008. Association for Computing Machinery. ISBN 9781605582044 doi: 10.1 1455/ 5390156. 1390294. URL Utt.tttp..ebbb.cco.....///////11
12

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
A.1
AUTOENCODER EXPERIMENTS
In this section we detail architectures, training routines, and additional experiments concerning Tri- angular Dropout applied to MNIST autoencoders.
A.1.1 MNIST ARCHITECTURE AND TRAINING DETAILS
For all MNIST cases the network architectures were as follows, given some encoder output size z:
Model
Type
Activation
Parameters
Encoder
Convolution
Relu
32 Filters, 9x9 Kernel, Stride
Convolution
Relu
32 Filters, 7x7 Kernel, Stride 1
Convolution
Relu
32 Filters, 3x3 Kernel, Stride
Fully-Connected
linear
4608 Inputs, 2 Outputs
Decoder
Fully-Connected
Relu
z Inputs, 4608 Outputs 32 Filters, 3x3 Kernel, Stride I
Transp. Conv.
Relu
Transp. Conv.
Relu
32 Filters, 7x7 Kernel, Stride I
Transp. Conv.
Relu
32 Filters, 9x9 Kernel, Stride
Convolution
sigmoid
Filter, 3x3 Kernel, Stride 1, Padding 1
Models are trained with batch size 1024 using the Adam optimizer, with a learning rate 0.005 that decreases by factor of 10 if the average epoch loss does not decrease by least 2 percent over the last 15 epochs. This is repeated for 5 decreases of the learning rate. MNIST images were normalized values between O and and binary cross entropy used the loss for reconstruction.
A.1.2 AVAILABLE WIDTH VERSUS COMPRESSION
We additionally performed experiments to test if the available encoding size affected the learned representations in Triangular Dropout. If extra width is available, does the model utilize that width or is it ignored? In Figure 7 we show a plot of autoencoders trained using Triangular Dropout with encoding widths up to 32, however they were trained with available widths of 4, 8, 16, 32, 64, 128, 256, and 512.
Drop
Drop N-8
0.225
Drop Nm32
Test Reconstructon Loss
0.200
Drop N=128
0.175
Drop N=512
0.150
Encoding Size,
Figure 6: MNIST reconstruction loss for the widths through 32 of autoencoders trained with Triangular Dropout and variety of available latent widths.
We find that there is a clear pattern models with more available width being less compressed, but the overall difference in performance diminishes as available width increases.
13

---

Under review as a conference paper at ICLR 2022
A.1.3 CELEBA ARCHITECTURE AND TRAINING DETAILS
We use the following architecture for CelebA experiments. The VAE results described below simply use two parallel encoding layers for the mean and variation, respectively. Models were trained for 100 epochs with batch size 128. We used the Adam optimizer with learning rate 0.001, decreasing by factor of 10 every 30 epochs.
Model
Type
Activation
Paramelers
Encoder
Convolation
Filters, Kernel, Stride Optional BN
Convolution
Relu
128 Filters, 5x5 Kemel. Stride Optional BN
Convolution
Relu
64 Filters, 3x3 Kernel. Stride
Fully-Commected
Relu
7744 Inputs, 2048 Outputs
Fully-Connecter
linear
2048 Inputs. 3 Outputs
Decoder
Culle-Coneetted
Inpuls, 2048 Outpuls
Colly-Cooneeted
Relu
2048 Inputs, 7744 Outputs
Transp. Conv.
Relu
128 Filters, 3x3 Kemel, Stride 1, Optional BN
Transp. Conv.
Relu
128 Filters, 5x5 Kemel, Stride 2, Outpul Padding 1, Optional BN
Transp. Conv.
Relu
128 Filters, 5x5 Kernel, Stride 2, Output Padding
Convolution
Relu
128 Filters, 3x3 Kernel, Stride 1, Padding 1, Optional BN
Convolution
Relu
128 Fillers, 3x3 Kernel, Stride 1, Padding 1, Optional BN
Convolution
Relu
128 Filters, 3x3 Kernel, Stride 1, Padding 1
Convolution
sigmoid
Filter, 3x3 Kernel, Stride 1, Padding
A.1.4 CELEBA RECONSTRUCTIONS WITH ALTERNATIVE ARCHITECTURES
Please see reconstruction examples on the following page.
14

---

Under review as a conference paper at ICLR 2022
(A) Triangular Dropout Autoencoder
CAI
6AT
2
4
12
16
20
24
32
48
64
96
128
ref
(B) Triangular Dropout Autoencoder with Batch Normalization
PA
2
1
2
4
12
16
20
24
32
48
64
96
128
ref
(C) Triangular Dropout Variational Autoencoder (VAE)
==
1
2
4
8
12
16
20
24
32
48
64
96
128
ref
Figure 7: Reconstructions of previously unseen CelebA images various architectures using Tri- angular Dropout at the encoding layer.
15

---

Under review as a conference paper at ICLR 2022
A.2 VGG19 TRAINING DETAILS
Our version of VGG19 with Triangular' Dropout was created by retraining the classification portion of VGG19. The convolutional layers were frozen and not retrained. The convolutional layers output 25088 features for given input, which then pass through two hidden layers of size 4096, and finally output as 1000 features to be interpreted classification scores for the image. Our model simply replaced the two hidden layers with Triangular Dropout layers of the same size.
We trained our model for 100 epochs using a batch size of 4096 order to apply full triangular mask to the batch. We used SGD with an initial learning rate of 0.01, reducing the learning rate by factor of 10 every 30 epochs.
It is unclear if our model would have behaved differently if the entire architecture were retrained, which may have enabled Triangular Dropout be applied the feature descriptors of length 25088. This is of interested because VGG19's 25088 features are often used as a starting point for other image processing tasks.
A.3 RL TRAINING SCHEMES
In this section we detail hyperparameters and training routines needed to reproduce our experiments from Section
A.3.1 PPO HYPERPARAMETERS AND TRAINING DETAILS
The PPO expert discussed in Section consisted of separate policy and value networks. The net- works were MLPs with three hidden layers of size 48 (or less for the retrained policies), with input and output sizes determined by the environment. Value networks consisted of two layers of width 128. All layers (besides value output) used hyperbolic tangent the activation function. We used the implementation of PPO from Stable Baselines 3 (Raffin et al. 2019).
Hyperparameter Training Steps
Value million
Hyperparameter Entropy Coefficient
Value
0.0
Parallel Actors
1
Clip Range Adam Learning Rate
0.2
Rollout Length
2048
0.0003
Training Epochs
10
Gamma, Y
0.99
Value Coefficient
0.5
Lambda, À
0.95
Table PPO Hyperparameters
A.3.2 PPO LEARNING CURVES AND STATISTICS
For reference, we provide the learning curves and final performance of our PPO agents. Please reference the figures on the subsequent pages.
A.3.3 IMITATION NETWORK TRAINING DETAILS
The imitation networks consisted of MLPs with 3 hidden layers of width 128, the middle layer using Triangular Dropout. Internal layers used the Relu activation function while the final layer used hyperbolic tangent. The networks action) pairs collected from running the PPO-trained policies on their respective environments. The networks were trained for 240 epochs with a batch size of 1024, using the Adam optimizer with leamiing rate of 0.01. Every 80 epochs, the learning rate was decreased by a factor of 10.
trained in supervised fashion
100,000 (state,
16

---

Under review as a conference paper at ICLR 2022
4000
Hopper-v3
HallChestahwva
Walker2d-va
3000
Episade Reward
2000
1000
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Training Stops
lee
Figure PPO training curves for expert policies
Hopper-v3 HallCheetah-v3
Walker2d v3 Ant-v3
3000
Episode Reward
2000
-1000
1.25
1.50
1.75
Training Steps
Figure 9: PPO training curves for retrained policies having compressed middle layer.
Hopper-v3
HallCheetah-v3
3000
Walker2div3
Episode Reward
2000
1000
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Training Steps
Figure 10: PPO training curves for retrained policies having all layers compressed.
17

---

Under review as a conference paper at ICLR 2022
Environment:
Hopper-v3
HalfCheetah-v3
Walker2d-v3
Ant-v3
Top Speed
0.76
3.95 0.05
4.47 0.50
3.95 0.42
Original Experts
Reward
2812 9444
47
1861 1189
1570 +
Architecture
48/48/48
48/48/4
48/48/
48/48/ 148//88
Top Speed
5.05
0.04
4.26
3.97 0.30
90% Speed Compressed (Middle)
Reward
1353 297
57
+ 255
1867 :
Architecture
48/8/48
48/2/48
48/2/48
48/7/48
Top Speed
4.42 0.32
3.00
4.40 0.48
3.80 0.27
90% Speed Compressed (A
Reward
1051 261
1365 89
1990 8
2028 467
Architecture
8/8/8
6/6/6
6/6/6
9/9/9
Top Speed
4.60 0.66
7.49 7.292
4.37 + 0.71
4.37 0.56
Retrained (Mi)
Reward
2356 +984
3318
1826 720
2099 + 787
Architecture
48/8/48
48/2/48
48/2/48
48/7/48
Top Speed
0.661
4.95
4.25 0.49
3.53 0.18
Retrained (All)
Reward
2422 1031
2150
1460 8
2208 479
Architecture
8/8/8
6/6/6
6/6/6
9/9/9
Figure 11: Average top speed, average reward, and architecture details for all agents the RL. experiments.
18


Reviews:
[{"review_id": "YKRDTAlFcCf", "reviewer": "Reviewer_Te1U", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Update after discussion: I increased my overall evaluation after the discussion and related literature has been updated and the conclusions adjusted. I also added a mention to the ethical considerations associated with the use of face data in my review, after the introduction of new results on CelebA data set.\n\nThis paper is written in a comprehensive way it is easy to follow, and the description of the experiments and presentation of the results is clear. The main appeal of the proposal, in my opinion, is the simplicity of the technique and I do see potential uses, some of which are described in the paper, such as flexible selection of the amount of resources during inference time. However, the weaknesses I see and that I describe below outweigh these strengths.\n\nIn my opinion, one of the main weaknesses of the paper is the lack of an in-depth discussion of the position of this paper within the existing literature. There exists abundant literature on model compression, which is the main use of the proposal, given that this is a topic of great interest for the machine learning industry which currently dominates the field. However, the paper falls short at explaining what gap it fills or what bottleneck it addresses that is missing in the existing literature. For example, there has been a vast amount of work on network pruning, but all that is mentioned in the paper in relation with this section of the literature is in the last paragraph of Section 2 Background, with a couple of sentences arguing how Triangular Dropout is different from pruning. I believe that beyond justifying the distinctive aspects of this paper, the paper would be stronger with a more comprehensive comparison with existing works, explaining which aspects are similar, which are different, which are the common and unique use cases, etc. \n\nRelated to this, several of the conclusions from the paper are stated as though they were completely novel and not previously presented in the, for instance, pruning literature. For example, in Section 5.2, the authors discuss that their results show \"that there are significant diminishing returns on classifier accuracy with respect to number of parameters\". Or later, in Section 6: \"The results from Section 5 indicate that VGG19 could use a more compressed architecture and still perform well\". This is well-known from the existing literature, but there is no discussion that places these results into a broader context.\n\nThe limited novelty could have been compensated by an in-depth investigation of the properties of Triangular Dropout. However, I will argue that the paper also falls short at shedding sufficient light about the proposed idea. I do value that the authors investigated Triangular Dropout in three distinct domains, that is image autoencoders, image classification and reinforcement learning. Nonetheless, the depth of the experimental setup does not seem sufficient to make a strong submission, in my opinion. For example, the autoencoder section (4) is limited to one architecture trained on MNIST, which is currently considered a toy data set by the community; the supervised learning section (5) is limited to the use of Triangular Dropout on the fully connected layers of VGG only. Many relevant questions remain open, which the authors leave for future work: does Triangular Dropout behave similarly on other architectures and data sets? Why is Triangular Dropout not analysed in convolutional layers? Even in Section 5.3 we read that \"[i]t is unclear\" whether the proposed technique \"would have behaved differently if the entire architecture were retrained\". In my opinion, this is an important question that a strong paper should give an answer to.\n\nFinally, I believe that some of the conclusions drawn by the authors in the paper go beyond what the experimental setup and results can support. For example, in Section 5.1, the last paragraph states that \"the success of the model with two layers of Triangular Dropout indicates that Triangular Dropout layers can be stacked in sequence to create a multi-layer MLP with adjustable width\". In order to confidently conclude that Triangular Dropout can be stacked in multiple layers, I believe that the authors should provide more experiments with several architectures (and ideally data sets) where more than two layers are stacked.\n\nI end the review with a series of shorter comments and questions:\n\n* I think a discussion of the _limitations_ of the proposal would make a stronger paper.\n* The authors qualify as \"small\" a drop in performance from 71.3 % to 69.7 % on top-1 accuracy on ImageNet. I believe many in the community would disagree.\n* Perhaps a more comprehensive naming of the sections would explicitly mention _autoencoders_, _image classification_ and _reinforcement learning_ in sections 4, 5 and 6, respectively.\n* The chosen name for the technique, Triangular Dropout, could be misleading since _dropout_ is widely associated with both _regularisation_ and _random_ dropout of neurons. Triangular Dropout is not aimed at regularisation and is not random, which makes it hard to interpret without further reading.\n\n### Typos\n\nBelow are some potential typos or mistakes I have identified during my review.\n\n* Page 4: with a guarantee that the proceeding -> with a guarantee that the preceding\n* Page 5: unless the ablation ois -> unless the ablation is\n* Page 2: incorrect opening double quotes in \"ablation\" and \"pruning\". Check rest of the paper too.", "summary_of_the_paper": "This paper proposes Triangular Dropout, a technique that multiplies the output of a fully connected layer by lower triangular matrix, with the aim of providing flexible selection of the dimensionality of the layer post-training. The paper analyses the use of this technique on an autoencoder trained to reconstruct MNIST images, on VGG trained on ImageNet and on reinforcement learning tasks.", "summary_of_the_review": "The paper is well written and presents an interest idea, but the weaknesses outweigh the strengths, to my assessment of the paper. The main concerns are related to the novelty of the proposal and the discussion with existing works, the depth of the experimental setup. I believe that a more in-depth analysis and stronger connections and comparisons with related techniques could make a stronger paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KemOSag5DSh", "reviewer": "Reviewer_CzhB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n- The proposed method is simple yet effective, allowing the models to retain most of their performance when the width is gradually reduced.\n- Experiments were extensive, carried out in three different areas, autoencoder, image classification, and reinforcement learning, and well illustrate the potential of the proposed method.\n\n[Weaknesses]\n- The experiments successfully illustrate the potential of the proposed method but its usefulness is still in doubt.\n1. The autoencoder experiment was carried out on MNIST, which is kind of a toy example; and\n2. The image classification experiment was based on VGG19, which is very outdated.\n\n(I do not know much about RL and won't judge the third experiment.)\n\n[Details]\n- The mask $M$ seems to be applied elementwise to the initial output $x^TW+b$. In this way, for the equation at page 2 bottom, matrix inner product should be used instead of matrix multiplication. The current formulation might cause confusion.\n\n[Questions]\n- Is it true that, after applying the lower-triangular mask, the elements of the output not only depend on different network widths but also utilize different portions of the batch of samples? If yes, is it the case that removing the last column retains most of the performance because parameters in the last column were trained on the fewest samples? I would like to see more discussions on this aspect.", "summary_of_the_paper": "This paper proposes a method, called Triangular Dropout, to allow a fully connected layer in a network to have variable width at deployment, which is achieved by applying a lower-triangular mask to the output of the fully-connected layer.\nExperiments were conducted in three different scenarios, autoencoder, image classification, and reinforcement learning. Results show that models trained with Triangular Dropout retain most of their performance when being ablated.", "summary_of_the_review": "The research topic is of value and the proposed method is simple and very effective on toy examples.\nLeaving Triangular Dropout for convolutions or transformers to future work makes this paper less informative than expected.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5et2egdQeKh", "reviewer": "Reviewer_yaJC", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents a clever idea that is very relevant to practical applications of neural networks, and could potentially impact industrial applications that require tight or variable control over the speed / performance tradeoffs of the system.\nUnfortunately, as it stands it fails to provide the level of experimental validation that would be required to make the case for the usefulness of the approach. It's unfortunate because the fundamentals of the method are strong and well-motivated.\nThere are a number of schemes one could imagine using to generate a progressively larger model: PCAAE is a good baseline, Progressive Networks could be another (https://arxiv.org/abs/1606.04671), but in the end the ceiling is to retrain a different model each time. It's important to characterize the cost of using this scheme against that ceiling, and the paper attempts to do so. However the choice of benchmarks is problematic:\n- MNIST is an ok place to start to ascertain that the method isn't broken, no issues here.\n- VGG is not a good benchmark to evaluate complexity vs performance. It is notoriously very far from any complexity-vs-performance optimality curve. You can do pretty much *anything* to a VGG network, and get a 10x improvement in complexity. The vast majority of network connections in a VGG net are useless, and can be pruned in any number of ways. So this benchmark is completely uninformative in ascertaining whether this scheme incurs a significant penalty compared to retraining in any practical application. I would urge the authors to pick any family of models that are known to constitute a strong complexity vs. accuracy tradeoff (MobileNets, EfficientNets, ...), and look at how this dropout scheme fares against full retraining. It doesn't have to beat them to be worthwhile, but anyone wanting to use this scheme will want to know what the gap looks like.\n- MuJoCo baselines are also notoriously insensitive to network capacity above a threshold, and collapse rapidly below it. Any practical user of those baselines would hence either never use a model that's too small to solve the task, and be completely insensitive to parametric complexity above that threshold. Since the proposed scheme doesn't speed up training over using a large network, there is no practical interest in this scheme here. Because of the overall insensitivity (shown by the step function in Fg 5), we can't extract from this experiment how well this scheme works compared to naive retraining.\nLearning whether the effective gap in performance between this scheme and full retraining of a SOTA model is small would be a big deal, and characterizing that gap well would make the difference between a clever, not-well-qualified idea and a great paper.", "summary_of_the_paper": "This paper proposes a modification to the popular dropout scheme which enables networks to be resized without having to retrain them. The key idea is to dropout units conditionally on others in a sequential fashion, so that a natural importance ordering is induced over each uint of a MLP layer, and no co-dependencies are created between less important units and more important ones, allowing the less important ones to be pruned.", "summary_of_the_review": "Sound idea, with good potential, but experimental validation lacking.\nEDIT POST REBUTTAL: Downgraded to strong reject, high confidence. Merely acknowledging the gaps in validation in the paper doesn't address concerns about experimental validation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "2sipMjo0M8", "reviewer": "Reviewer_8MmW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Positives**\n\n- The paper is very well written. Ideas are conveyed clearly, with useful diagrams and math. Experimental choices are properly explained and justified.\n- The method is simple and should be easy to adopt by practitioners.\n- Experiments are well designed and prove the authors' hypotheses.\n\n\n**Concerns**\n\n- Although I flagged the experimental design as one of the strengths of the paper because I believe they clearly show the authors' points, it is also true that most experiments are quite toyish (MNIST) or not entirely relevant given the current trends in deep learning (most modern architectures, such as ResNets, do not have fully-connected layers). The paper would be much stronger if it showed the benefits of Triangular Dropout on more realistic settings, and I believe that GAN or VAE experiments similar to the ones in the PCAAE paper would be a good way to show this.\n- The background / related work draws quite interesting connections with existing approaches, but in my opinion it should also cover the relationship to the field of neural networks with adaptive computation (which has been mostly developed in the context of RNNs). Some of these works are related to directions mentioned in the future work section such as variable depth (ACT, S-ACT, PonderNet, SkipRNN). There are other works that consider variable width for RNNs, such as Jernite et al. (2017). Please find relevant references below.\n- Authors seem to assume that $B \\geq n$ so that they can create a lower triangle dropout matrix. This can be problematic for many models used in practice, such as the VGG-19 architecture in the ImageNet experiments, as one may not be able to allocate thousands of examples per batch due to hardware or optimization constraints. However, if I understood the method correctly, the only real requirement is for every row in the dropout matrix to be composed of $p$ 1's followed by $n-p$ 0's (so that one can just sample $p \\sim Uniform[0, n-1]$ for every row). I would appreciate if authors could clarify this, and would suggest including this in the paper should it be correct.\n\n**Missing references**\n\nACT\n>Graves. \"Adaptive computation time for recurrent neural networks\". arXiv preprint arXiv:1603.08983, 2016.\n\nS-ACT\n>Figurnov et al. \"Spatially adaptive computation time for residual networks.\" CVPR 2017.\n\nPonderNet\n>Banino et al., Pondernet: Learning to ponder. arXiv preprint arXiv:2107.05407.\n\nSkipRNN\n>Campos et al., \"Skip RNN: Learning to skip state updates in recurrent neural networks.\" ICLR 2018.\n\nVCRNN\n>Jernite et al., \"Variable computation in recurrent neural networks\". In ICLR, 2017.\n\n**Minor comments / questions**\n\n- How are the autoencoder ablations in Figure 2 performed? Do you just use the first $n$ units in the bottleneck? Have you tried optimizing/searching for the binary mask with $n$ elements that provides the best performance?\n- Have you visualized the training curves of the autoencoder with Triangular Dropout? Does it need more iterations to converge than a standard autoencoder? \n- The VGG experiments use a huge batch size, which could result in overfitting. Have you trained the same network without Triangular Dropout to rule this out? You may have done this already, I'm unclear about where the accuracy for the baseline VGG-19 comes from (did you retrain it yourselves?).\n\n**Typos**\n\n- Page 3: wrong quotation marks for *\"tasks\"* (after the equation in the center of the page)\n- Page 4, first line: masks -> makes\n- Page 5: *ois very similar* -> *is very similar*\n- Section 6.1: wrong latex command when citing Kirkpatrick et al.\n- Section 6.1: wrong quotation marks before *smallest* (twice in the same line)\n- Section 6.2: *Any-v3* -> *Ant-v3*", "summary_of_the_paper": "This paper proposes Triangular Dropout, a mechanism for training fully-connected layers whose width can be decreased at test time, by using dropout masks with a particular structure at train time. The proposed approach compares favorably to previous methods when training autoencoders of varying bottleneck size on MNIST. Authors show that Triangular Dropout can also be used to tune the parameter count - accuracy trade-off at test time on VGG-19. Finally, the proposed method is used to measure task complexity in RL by using the minimum layer width required to solve the task as a proxy.", "summary_of_the_review": "This a good paper that proposes a simple yet effective idea and conveys it very clearly. However, I believe that it is currently below the threshold for ICLR due to the scale and relevance of its experiments. I will update my score if authors can provide experiments on more realistic settings (please see my review above for suggestions). I also have some minor concerns regarding the related work section and some technical details that can be solved throughout the discussion period, please see my list of concerns above for more details.\n\n\n===== POST-REBUTTAL EDIT ====\nUpdated score from 5 to 6", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes triangular dropout training to provide adaptive capacity of the network at inference time. The proposed approach is simple and sound. However, the experiments are lacking in terms of complexity of the task and up-to-date architectures (e.g., transformers or convolutional layers) to demonstrate the effectiveness of the method.
Therefore I recommend this paper for rejection.
Decision: Reject