Neural Parameter Allocation Search
Authors: Bryan A. Plummer
Nikoli Dryden
Julius Frost
Torsten Hoefler
Kate Saenko
Venue: ICLR 2022 Poster (2022)
Keywords: efficient training methods
cross-layer parameter sharing

Abstract: Training neural networks requires increasing amounts of memory. Parameter sharing can reduce memory and communication costs, but existing methods assume networks have many identical layers and utilize hand-crafted sharing strategies that fail to generalize. We introduce Neural Parameter Allocation Search (NPAS), a novel task where the goal is to train a neural network given an arbitrary, fixed parameter budget. NPAS covers both low-budget regimes, which produce compact networks, as well as a novel high-budget regime, where additional capacity can be added to boost performance without increasing inference FLOPs.  To address NPAS, we introduce Shapeshifter Networks (SSNs), which automatically learn where and how to share parameters in a network to support any parameter budget without requiring any changes to the architecture or loss function. NPAS and SSNs provide a complete framework for addressing generalized parameter sharing, and can also be combined with prior work for additional performance gains. We demonstrate the effectiveness of our approach using nine network architectures across four diverse tasks, including ImageNet classification and transformers.

Published as a conference paper at ICLR 2022
NEURAL PARAMETER ALLOCATION SEARCH
Julius Bryan A. Plummer*, Nikoli Dryden Jullis Frost! Torsten Hoefler; Kate Saenko Boston University, *ETH Zürich, MIT-IIB Watson Al Lab (bppum, jul jusf, saenko)@bu. edu fnikoli dryden, torsten. horstem.hbeflereeiiff..ttit
ABSTRACT
Training neural networks requires increasing amounts of memory. Parameter sharing can reduce memory and communication costs, but existing methods assume networks have many identical layers and utilize hand-crafted sharing strategies that fail to generalize. We introduce Neural Parameter Allocation Search (NPAS), a novel task where the goal is to train a neural network given an arbitrary, fixed parameter budget. NPAS covers both low-budget regimes, which produce compact novel high- regime, where additional capacity can
networks, as well be added to boost performance without increasing inference FLOPs. To address NPAS, we introduce Shapeshifter Networks (SSNs), which automatically learn where and how to share parameters a network to support any parameter budget without requiring any changes to the architecture loss function. NPAS and SSNs provide a complete framework for addressing generalized parameter sharing, and can also be combined with prior work for additional performance gains. We demonstrate the effectiveness of our approach using nine network architectures across four diverse tasks, including ImageNet classification and transformers.
INTRODUCTION
Training neural networks requires ever more computational resources, with GPU memory being a significant limitation (Rajbhandari et al. (021]. Method such as checkpointing (e.g.. Chen et al. [0166]
(eeg.,RRn Gomez et al., 2017: Jain et al. (2200] and out-of-core algorithms (e.g.. Ren et al. 2021) have been developed to reduce memory from activations and improve training efficiency. Yet even with such techniques, Rajbhandari et al. (00TT) find that model parameters require significantly greater memory bandwidth than activations during training, indicating parameters are key limit on future growth. One solution is cross-layer parameter sharing, which reduces the memory needed to store parameters, which also reduce the cost of communicating model updates in distributed training (Lan et al.
2020 Jaegle et al. 202T) and federated learning (Konečný et al. 2016 McMahan et al. 2017). 7), as the model is smaller, and can help avoid overfitting (Jaegle al. (021). However, prior work in parameter sharing (e.g.. Dehghani et al., 2019 Savarese & Maire, [2019] Lan et al. 2020 Jaegle al. 2222T) has significant limitations. First, they rely suboptimal hand-crafted techniques for deciding where and how sharing occurs. Second, they rely on models having many identical layers. This limits the network architectures they apply to (e.g.. DenseNets (Huang al., 2017) have few
such layers) and their parameter savings is only proportional to the number of identical layers.
To move beyond these limits, we introduce Neural Parameter Allocation Search (NPAS), a novel task which generalizes existing parameter sharing approaches. In NPAS, the goal is to identify where and how to distribute parameters in a neural network to produce a high-performing model using an arbitrary, fixed parameter budget and no architectural assumptions. Searching for good sharing strategies is challenging in many neural networks due to different layers requiring different numbers of parameters weight dimensionalities, multiple layer types (e.g., convolutional, fully-connected, recurrent), and/or multiple modalities (e.g., text and images). Hand-crafted sharing approaches, as in prior work, can be seen for complex networks and have no guarantee that the sharing strategy is good. Trying all possible permutations of sharing across layers is computationally infeasible even for small networks. To our knowledge, we are the first to consider automatically searching for good parameter sharing strategies.
implementation of NPAS, but they can be complicated to create
*indicates equal contribution

---

Published as a conference paper at ICLR 2022
Neural Perameter Allocation Paramater Sharing
Prior work in Gross-
Knowlddge Ditillation
Pruning/Duemitiigg
Netwarks
makes training more efficient
X
X
supports any parameter budget (low and high))
generalizes architecture
Z
focuses improving task performance
Figure 1: Comparison of related tasks. Neural Parameter Allocation Search (NPAS) is novel task where the goal is to train a neural network given a fixed parameter budget. This generalizes prior work, which supports only subset of NPAS's settings. E.g., pruning can decrease network parameters, but starts from full network, and many cross-layer parameter sharing works rely on hand-crafted strategies that work for only limited set of architectures. See Section
2 for discussion.
By supporting arbitrary parameter budgets, NPAS explores two novel regimes. First, while prior work considered using sharing to reduce the number of parameters (which we refer to as low-budget NPAS, LB-NPAS),
also increase the number of parameters beyond what an architecture typically uses (high-budget NPAS, HB-NPAS). HB-NPAS can be thought of as adding capacity to the network in order to improve its performance without changing its architecture (e.g., without increasing the number of channels that would also increase computational time). Second, we consider cases where there fewer parameters available to layer than needed to implement the layer's operations. For such low-budget cases, we investigate parameter upsampling methods to generate the layer's 's weights.
A vast array of other techniques, including pruning (Hoefler et al. 2021), quantization (Gholami iet al. (202I), )) knowledge distillation (Gou et al. 2022), ). and low-rank approximations (e.g.. Wu 2019 Phan al. 2020) used to reduce memory and/or FLOP requirements for model. However, such methods typically only apply at test/inference time, and actually are more expensive to train due to
requiring fully-trained large network, contrast to NPAS. Nevertheless, these are also orthogonal to NPAS and can be applied jointly. Indeed, we show that NPAS can be combined with pruning or distillation to produce improved networks. Figure T compares NPAS to closely related tasks.
To implement NPAS, we propose Shapeshifter Networks (SSNs), which can morph given parameter budget to fit any architecture by learning where and how share parameters. SSNs begin by learning which layers can effectively share parameters using short pretraining step, where all layers are generated from a single shared set parameters. Layers that use parameters good candidates for sharing during the main training step. When training, SSNs generate weights for each layer by down- or upsampling the associated parameters as needed.
similar way are then
We demonstrate SSN's effectiveness in high- and low-budget NPAS on variety of networks, including vision, text, and vision-language tasks. E.g., a LB-NPAS SSN implements a WRN-50-2 (Zagoruyko & Komodakis] 2016) using 19M parameters (69M in the original) and achieves Error @5
ImageNet (Deng et al. 2009) 3% lower than a WRN with the same budget. Similarity, we achieve a
1 1% boost SQuAD v2.0 (Rajpurkar et al. 2016) with 18M parameters (334M in the original) over
ALBERT (Lan et al. 2020), prior work for parameter sharing in Transformers ( (Vaswani et al.) (017)) For HB-NPAS, we achieve 1-1.5% improvement in Error on CIFAR (Krizhevsky) 2009) by adding capacity to traditional network. In summary, our key contributions are:
We introduce Neural Parameter Allocation Search (NPAS), novel task in which the goal is to implement a given network architecture using any parameter budget.
To solve NPAS, we propose Shapeshifter Networks (SSNs), which automate parameter sharing. To our knowledge, SSNs are the first approach to automatically learn where and how to share parameters and to share parameters between layers of different sizes or types.
We benchmark SSNs for LB- and HB -PAS and show they create high high-performing networks when either using few parameters or adding network capacity.
We also show that SSNs can be combined with knowledge distillation and parameter pruning to boost performance over such methods alone.
2 NEURAL PARAMETER ALLOCATION SEARCH (NPAS)
In NPAS, the goal is to implement a neural network given a fixed parameter budget. More formally:
2

---

Published as a conference paper at ICLR 2022
Neural Parameter Allocation Search (NPAS): Given neural network architecture with layers fi,.... L,, which each require weights wi....: WL, and a fixed parameter budget 8, train a high-performing neural network using the given architecture and parameter budget.
Any general solution to NPAS (i.e., that works for arbitrary @ or network) must solve two subtasks:
1. Parameter mapping: Assign to each layer & subset of the available parameters.
2. Weight generation: Generate & 's weights w; from its assigned parameters, which may be any size.
Prior work, such as Savarese & Maire 2019) and Ha et al. (20T6), are examples of weight generation methods, but in limited cases, e.g., Savaresee & Maire (2019) does not support there being fewer parameters than weights. To our knowledge, no prior work has automated parameter mapping, instead relying on hand-crafiee heuristics that do not generalize to many architectures. Note weight generation must be differentiable so gradients can be backpropagated to the underlying parameters.
NPAS naturally decomposes into two different regimes based on the parameter budget relative to what would be required by a traditional neural network (i.e., Σ / |ay versus [0)):
/w:a/ < [0). This regime has traditionally been the goal of cross-layer parameter sharing, and reduces memory at training and test time, and consequentially reduces communication for distributed training.
Low-budget (LB-NPAS), with fewer parameters than standard networks ? [wy
High-budget (HB-NPAS), with more parameters than standard networks ② 5 /w,l > 10). This is, to our knowledge, a novel regime, and can be thought of as adding capacity to a network without changing the underlying architecture by allowing layer access more parameters.
Note, in both cases, the FLOPs required of the network do not significantly increase. Thus, HB-NPAS significantly reduce FLOP overhead compared to larger networks.
The closest work to ours are Shared WideResNees (SWRN) Savarees & Maire 2019), Hypernetworks (HN) (Ha et al.] (2016), and Lookup-based Convolutional Networks (LCNN) (Bagherinezhad et al.) 2017). Each method demonstrated improved low-budget performance, with LCNN and SWRN focused improving sharing across layers and HN learning directly generate parameters. However, all require adaptation for new networks and make architectural assumptions. E.g. LCNN was designed specifically for convolutional networks, while HN and SWRN's benefits are proportional to the number of identical layers (see Figure B). Thus, each method supports limited architectures and parameter budgets, making them unsuited for NPAS. LCNN and HN also both come with significant computational overhead. E.g., the CNN used by Ha et al.] requires 26.7M FLOPs for a forward pass on a 32 x 32 image, but weight generation with HN requires an additional 108. 55M FLOPs (135.2M total). In contrast, our SSNs require 0.8M extra FLOPs (27.5M total, 5x fewer than HN), Across networks we consider, SSN overhead for a single image is typically 0.5-2% of total FLOPs. Note both methods generate weights once per forward pass, amortizing overhead across batch (e.g., SSN overhead is reduced to 0.008 10008-0.034 0.03% for batch size 64). HB-NPAS is also reminiscent of mixture-of-expertt (e.g.. (e.g.|Shazzer (Shazeer al. 2017); both increase capacity without significantly increasing FLOPss, but NPAS allows this overparameterization to be learned without architectural changes required by prior work.
NPAS can be thought of searching for efficient and effective underlying representations for neural network. Methods have been developed for other tasks that focus on directly searching for more effective architectures (as opposed to their underlying representations). These include neural architecture search (e.g.. [Bashivan et al., 2019 Dong & Yang 2019 Tan et al.
2019 Xiong et al. 2019 Zoph & Le 2017) and modular/sel. assembling networks (e.g.. Alet al.
2019 Ferran Alet 2018 Devin et al. 2017). While these tasks create computationally efficient architectures, they do not reduce the number of parameters in a network during training like NPAS (i.e., they cannot be used to train very large networks or for federated or distributed learning applications), and indeed are computationally expensive. NPAS methods can also provide additional flexibility to architecture search by enabling them to train larger and/or deeper architectures while keeping within a fixed parameter budget. In addition, the performance of any architectures these methods create could be improved by leveraging the added capacity from excess parameters when addressing HB-NPAS.
3

---

Published as a conference paper at ICLR 2022
P
Parameter Groups| 0 3M parameter budget
Weight generation ($3.1)
Mapping learned preliminary training ($3.2)
Upsampling (sizes for
F. F.interpolate
Inter Mask Tiles
Colrishows mapping
300k
2
100k
0g 2.6M
upl
2008
1M
IM
Sharing between modalities and/or layer types
Murtsiiedi In windows
masks
Fc]300x1024
f2 1021024X1024
Downsampling (sizes Fsl
Text
[w: 300k
[ = 1M
FC 1024552
orttogonal init lasened
WAvg Emb
oethaggonll
[ws] = 500k Rujlsdowrsappl
Wydt
W. W.b-shored
Conv 3x
FC18192x1024
Image
[Wg]]
[ 8.4M 1NG Palrupsamplc
temolates:
2F
< (ccrommampll
Mg op
+ E
Figure 2: Overview of Shapeshifter Networks (SSNs). To train SSN begin by learning mapping of layers to parameter groups (Section 3.2). Only layers within the same group share parameters. Then, during the forward pass each layer uses its shared parameters to generate the weights it needs for its operation (Section 3.1 Note that SSNs do not change a network's 's architecture or loss function, and automates the creation of parameter sharing strategy for network.
SHAPESHIFTER NETWORKS FOR NPAS
We now present Shapeshifter Networks (SSNs), framework for addressing NPAS using generalized parameter sharing to implement neural network with an arbitrary, fixed parameter budget. Figure 2 provides an overview and example of SSNs, and we detail each aspect below. An SSN consists of provided network architecture with layers fi...... and a fixed budget of parameters 0, which are partitioned into P parameter groups (both hyperparameters) containing parameters P1....P. Each layer is associated with single parameter group, which will provide the parameters used to implement it. This mapping is learned in a preliminary training step by training a specialized SSN and clustering its layer representations (Section 3.2) To implement each layer, SSN morphs the parameters in its associated group to generate the necessary weights; this uses downsampling (Section 33..I] when the group has more parameters than needed, or upsampling (Section 3.1.2) when the group has fewer parameters than needed. SSNs allow any number of parameters to "shapeshift" into a network without necessitating changes to the model's loss, architecture, or hyperparameters, and the process can be applied automatically. Finally, we note that SSNs are simply one approach to NPAS. Appendices D contain ablation studies and discussion of variants we found to be less successful.
3.1 WEIGHT GENERATION
Weight generation implements layer L, which requires weights wy, using the fixed set of parameters in its associated parameter group 0z. (We assume the mapping between layers and parameter groups has already been established; see Section (32) There three cases to handle:
1. [yyl = |0,| (exactly enough parameters): The parameters are used as-is.
2. Wa [0, (excess parameters): We perform parameter downsampling (Section 3.1.1
3. Woa 10,1 (insufficient parameters): We perform parameter upsampling (Section 3.1.2
We emphasize that, depending on how layers are mapped to parameter groups, both down- and upsampling may be required in an LB- or HB-NPAS model.
3.1.1 PARAMETER DOWNSAMPLING
When a parameter group 8, provides more parameters than needed to implement a layer l, we perform bemplate-based downsampling generate w;. To do this, we first split f, into up to K (a hyperparameter) templates T......K where each template 7 is the same dimension as w;. 0, does not evenly divide into templates, we ignore excess parameters. To avoid uneven sharing of parameters between layers, the templates for each layer are constructed from 0, in a round robin fashion. These templates are then combined to produce wy; if only template can be produced we instead use it directly. We present two different methods of learning to combine templates. To simplify presentation, will assume there are exactly K templates used.
WAyg (Savarese& & Maire 2019) This learns a vector a; E I which is used to produce a weighted average of the templates: W; = Σ K
" T... The a; are initialized orthogonally to the of all other
4

---

Published as a conference paper at ICLR 2022
layers in the same parameter group. While efficient, this only implicitly learns similarities between layers. Empirically, we find that different layers often converge to similar as, limiting sharing.
Emb To address this, embedding. We use a learnable vector Φ; E ER where E is the size of the layer representation; we use E = 24 throughout, in the parameter group and parameterized by W E R Kxx and by E IRK, is then used to construct an o; for the layer, which is used as in WAvg. That is, a₂ = W + by and w. = Σ K a We considered more complex methods (e.g.. MIPss nonlinearities), but they did not improve performance.
instead more directly learn representation of the layer using layer
found it to work well. A linear layer, which is shared between all layers
While both methods require additional parameters, this is quite small in practice. WAvg requires K" additional parameters per layer. Emb requires E 24 additional parameters per layer and KE + K = 24K + K parameters per parameter group.
3.1.2 PARAMETER UPSAMPLING
If instead a parameter group 0, provides fewer parameters than needed to implement a layer L, we upsample 0, to be the same size as w, As a layer will use all of the parameters in 0z. we do not use templates. We consider two methods for upsampling below.
Inter As a naive baseline, we use bilinear interpolation to directly upsample f,. However, this could alter the patterns captured by parameters, as it effectively stretches the receptive field. In practice. we found fully-connected and recurrent layers could compensate for this warping, but it degraded convolutional layers compared to simpler approaches such as tiling 0y.
Mask To address this, and avoid redundancies created by directly repeating parameters, we propose instead to use a learned mask to modify repeated parameters. For this, we first use n = [ 0, tiles of 0g to be the same size as w; (discarding excess in the last tile). We then apply a separate learned mask to each tile after the first (i.e., there are n - masks). All masks are a fixed "window" size, which take to be 9 by default (to match the size of commonly-user 3 x 3 kernels in CNNs), and are shared within each parameter group. To apply, masks are multiplied clement-wise over sequential windows of their respective tile. While the number of additional parameters depends on the amount of upsampling required, as the masks are small, this is negligible.
3.2 MAPPING LAYERS TO PARAMETER GROUPS
We now discuss how SSNs can automatically learn to assign layers to parameter groups in such way that parameters can be efficiently shared. This is in contrast to prior work on parameter sharing (e.g.. Ha et al.] 2016 Savarese & Maire| 2019 Jaegle et al. (02T), which required layers to be manually assigned to parameter groups. Finding an optimal mapping of layers to parameter groups challenging, and brute-force approach is computationally infeasible. We rely instead on SSNs learning a representation for each layer as part of the template-basee parameter downsampling process, and then use this representation to identify similar layers which can effectively share parameters.
To do this, we perform a short preliminary training step in which we train a small (i.e., low parameter budget) SSN version of the model using single parameter group and modified means of generating templates for parameter downsampling. Specifically, for a layer L, we split 0 into K" evenly-sized weight generation, each T.K is then
templates T.!....KK' Since we wish to use downsampling-basee resized with bilinear interpolation to be the same size as w,. Next, we train the SSN as usual, using WAvg or Emb downsampling with the modified templates for weight generation (there is no upsampling). By using small parameter budget and remplate-based weight generation where each template comes from the same underlying parameters, we encourage significant sharing between layers we can measure the effectiveness of sharing. We found that using budget equal to the number of weights of the largest single layer in the network to work well. Further, this preliminary training step is short, and requires only 10-15% of the typical network training time.
Finally, we construct the parameter groups by clustering the learned layer representations into P groups. As the layer representation, take the a; or o; learned for each layer by WAvg or Emb downsampling (resp.). We then use k-means clustering to group these representations into groups, which become the parameter groups used by the full SSN.
5

---

Published as a conference paper at ICLR 2022
4 EXPERIMENTS
Our experiments include a wide variety of tasks and networks in order to demonstrate the broad applicability of NPAS and SSNs. We adapt code and data splits made available by the authors and report the average of five runs for all comparisons except ImageNet and ALBERT, which average three runs. A more detailed discussion on SSN hyperparameter settings can be found in Appendices B In our paper we primarily evaluate methods based on task performance, but we demonstrate that SSNs improve reduce training time and memory in distributed learning settings in Appendix G
Compared Tasks. We briefly describe each task, datasets, and evaluation metrics. For each model. we use the authors' implementation and hyperparameters, unless noted (more details in Appendix A
Image Classification. For image classification the goal is to recognize if an object is present in an image. This is evaluated using Error@k, k, i.e., the portion of times that the correct category does not appear in the top & most likely objects. We evaluate SSNs on CIFAR-10 and CIFAR- 100 (Krizhevsky 2009), which are composed of 60K images of 10 and 100 categories, respectively, and ImageNet (Deng et al. 2009), which is composed of 1. 1.2M images containing 1,000 categories. We report Error @ on CIFAR and Error @5 for ImageNet.
Image-Sentence Retrieval. In image-sentence retrieval the goal is match across modalities (sen- tences and images). This task is evaluated using Recall K= 10] for both modal directions (six numbers), which we average for simplicity. We benchmark on Flickr30K (Young et al.) (014)) which contains / images for training/eestin//vlliattooo and COCO (Lin et al., 2014), which contains 1 123K/IK/IK images for paining/festin//alldaaiii about five descriptive captions. We evaluate SSNs using EmbNet (Wang et al.) 2016) and ADAPT- T2I (Wehrmann et al. 2020). Note that ADAPT-T2I has identical parallel layers (i.e.. they need different outputs despite having the same input), which makes sharing parameters challenging.
For both datasets each image has
Phrase Grounding. Given a phrase the task is to find its associated image region. Performance is measured by how often the predicted box for a phrase has at least 0.5 intersection over union with its ground truth box. We evaluate on Flickr30K Entities (Plummer et al., (017)) which augments Flickr30K with 276K bounding boxes for phrases in image captions, and Referlt (Kazemzadeh et al., 2014), which contains 20K images that are evenly split between training/testing and 120K region descriptions. We evaluate our SSNs with SimNet (Wang et al.) (018)) using the implementation from Plummer et al. (2020) that reports state-of-the-art results on this task.
Question Answering. For this task the goal is to answer a question about a textual passage. We use SQuAD v (Rajpurkar et al. 2016), which has 100K+ question/answer pairs on 500+ articles, and SQuAD v2.0 (Rajpurkar et al.) 2018). which adds 50K unanswerable questions. We report F1 and EM scores on the development split. We compare our SSNs with ALBERT (Lan et al.] 2020), a recent transformer architecture that incorporates extensive, manually-designed parameter sharing.
4.1 RESULTS
We begin our evaluation in low-budget (LB-NPAS) settings. Figure 3 reports results on image classification, including WRNs (Zagoruyko & Komodakis 2016), DenseNets (Huang al. (017). and EfficientNets (Tan & Le 2019): Table I contains results on image-sentence retrieval and phrase grounding. For each task and architecture we compare SSNs same parameter-sized networks without sharing. In image classification, we also report results for SWRN (Savarese & Maire 2019) sharing: but note it cannot train a WRN-28-10 10 or WRN-50-2 with fewer than 12M or 40M parameters. resp. We show that SSNs can create hirh-performing models with fewer parameters than SWRN is capable of, and actually outperform it using 25% and 60% fewer parameters on C-100 and ImageNet, resp. Table demmonttrates that these benefits generalize to dision-languag tasks. In Table 2 we also compare SSNs with ALBERT (Lan et al.. 2020). which applics manually-designn to BERT (Devlin et al. 2019), and find that SSN's 's learned parameter sharing outperforms ALBERT. This demonstrates that SSNs can implement large networks with lower memory requirements than is possible with current methods by effectively sharing parameters.
parameter sharing
We discuss the runtime and memory performance implications of SSNs extensively in Appendix G In short, by reducing parameter counts, SSNs reduce communication costs and memory. For example,

---

Published as a conference paper at ICLR 2022
Image Classification on CIFAR-100 100
Image Classification on ImageNet
original archlesture paramoters
WSN lalp dan perinu
Erron@h
25
5.0%
Decedleni orgp it aV peane
SURosaa-t (Javaras kiiis 2019
95%
47%
Erro@@s
10.0
7.5
15%
24% of original arenitacture parsmatatss
15%
Uliceerhiiting 5:le paans.
23%
20
35NDANIN-76-122 15
oeggisa WAN
SWRN 50
10
15
15 40
#Parameters (M)
#Parameters (M)
Figure 3: Comparison of reducing the parameter parameters by adjusting layer sizes vs. our SSNs us- ing the same number of parameters. We also compare to prior work in parameter sharing, specifically SWRN (Savarese & Maire (219). Note that due to architecture assumptions made by SWRN, cannot reduce the number of parameters of WRN-28- 10 by more than than 33% of the original and 58% of the parameters of a WRN-50-2. In contrast, we can not only use far fewer parameters that perform better than SWRN, but having to increase our parameter usage. See text for additional discussion.
also implement even larger networks (WRN-76-12) without
Table 1: SSNs vs. similar size traditional networks on image-sentence retrieval and phrase grounding.
Dataset
Method
Performance
Image-Sentence Retrieval
% orig params
6.9%
13.9%
100%
EmbNet
56.5
70.2
74.1
F30K
SSN-EmbNet
67.9
71.6
74.4
EmbNet
74.2
77.3
81.4
COCO
SSN-EmbNet
76.7
78.2
82.1
% orig params
34.7%
70.9%
100%
ADAPT-T2I
76.6
80.6
83.3
F30K
SSN-ADAPT-T2
81.2
81.7
82,9
ADAPT-T2I
83.8
85.2
86.8
COCO
ISN-ADAPT-
85.3
86.2
87.0
Phrase Grounding
% orig params
7.8%
15.6%
100%
SimNet
70.6
71.0
71.7
F30K Ent
SSN-SimNet
71.9
72.1
72.4
SimNet
58.5
59.2
61.1
Referlt
SSN-SimNet
59.5
59.9
61.0
our SSN-ALBERT-Large trains about 1.4x faster using 128 GPUs than BERT-Large (in line with results for ALBERT), and reduces memory requirements by about 5 GB (1/3 of total).
As mentioned before, knowledge distillation and parameter pruning can help create more efficient models at test time, although they cannot reduce memory requirements during training like SSNs. Tables 3 and show our approach can be used to accomplish similar goal as these tasks. Comparing our LB-NPAS results in Table and the lowest parameter setting of HRank, we report a 1.5% gain over pruning methods even when using less than half the parameters. We note that one can think of our SSNs in the high budget setting (HP-NPAS) as mixing together a set random initializations of network by learning to combine the different templates. This setting's benefit is illustrated in Table 3 and Table where our HB-NPAS models report a 1-1.5% gain over training a traditional network. As reminder, in this setting precompute the weights of each layer once training is complete so they require no additional overhead at test time. That said, best performance on both tasks comes from combining our SSNs with prior work on both tasks.
4.2 ANALYSIS OF SHAPESHIFTER NETWORKS
In this section we present ablations of the primary components of our approach. A complete ablation study, including, but not limited to comparing the number of parameter groups (Section 3.2 and number of templates (K in Section 3.1.I can be found in Appendices B
the strategies generating weights from templates described Section 3.1.] when using a single parameter group. In these experiments set the number of parameters as the amount required to implement the largest layer in a network. For example, the ADAPT-T2I model requires 14M parameters, but its bidirectional GRU accounts for 10M of those parameters, all SSN variants in this experiment allocate 10M parameters. Comparing to the baseline, which involves modifying the original model's number and/or size of filters so they have the same number of parameters as
7

---

Published as a conference paper at ICLR 2022
Table 2: Comparison of ALBERT- and
Table 3: Knowledge distillation experiments CIFAR- 100 using OFD (Heo et al. 2019) and SRR (Yang et al. 202Tb. HB-SSN's budget is 4x the student's size. For all settings using our SSNs improve performance over distillation alone.
large (Lan et al. 2020) vs SSNs using P
4 learned parameter groups. Scores are F1 and EM on dev sets,
Network
SQuAD vI.1 SQuAD v2.0
Base
12M params (11% of orig)
Teacher Error@
Student
Baseline OFD SRR
ALBERT
89.2/82.1
79.8/76.9
WRN-16-2
27.30
24.43 24.03
LB-SSN
90.1/83.0
80.7/78.0
WRN-40-4 20.50
HB-SSN
26.53
23,99 23,42
Large
18M params (5% of orig)
ResNet-10
31.58
31.06 30.09
ALBERT
90.5/83.7
82.1/79.3
ResNet-34 27.95
HB-SSN 30.88
30.41 29.63
LB-SSN
91.1/84.4
83.0/80.1
Table 4:
Parameter pruning experiments with HRank (Lin et al., 2020). HB-SSN's budget is 4x the model's size during pretraining, then we generate the final weights for pruning. In all settings using our SSNs improve performance over pruning alone.
Table Parameter downsampling com- parison (Section 3.1.1) (S..tioo(3...) and WRN-50-2 for C-10/100 and ImageNet, resp. Baseline adjusts the number and/or size of filters rather than share parameters. See Appendix for additional details.
using WRN-28-10
C-10 Error
% original % original
Method
ResNet-55 HB-SSN
test FLOPs test params
% orig Reduced
SSNs (ours)
Dataset
Base
6.74
5.21
100.0
100.0
params
Baseline
WAvg Emb
HRank
6.48
5.86
70.7
83.2
C-10
4.22
4.00
3.84
11.3%
HRank
10.75
9.94
20.3
28.4
C-100
22.34
21.78 21.92
LB-SSN
9.26
100.6
12.2
ImageNet
27.5%
10.08
7.38
6.69
SSNs, we see that the variants of our SSNs perform better, especially on ImageNet where we reduce Erro@e5 by 3%. Generally we see slight boost to performance using Emb over WAvg. Also note that prior work in parameter sharing, e.g., SWRN (Savarese & Maire 2019), can not be applied to the settings in Table since they require parameter sharing between layers of different types and different operations, or have too few parameters. E.g., the WRN-28-IP 10 results use just under 4M parameters, but, as shown in Figure a). SWRN requires a minimum of 12M parameters.
In Table 6 we investigate one of the new challenges in this work: how to upsample parameters so a large layer operation can be implemented with relatively few parameters (Section 3.1.2). For example, our
180 results use about 0.5M parameters, but the largest layer defined in the network requires just under 4M weights. We find using our simple learned Mask upsampling method performs well in most settings, especially when using convolutional networks. For example, on CIFAR- 100 improves Error@ by 2.5% over the baseline, and 1.5% over using bilinear interpolation (inter). While more complex methods of upsampling may seem like they would improve performance (e.g., using an MLP, or learning combinations of basis filters), we found such approaches had two significant drawbacks. First, they slow down training time significantly due to their complexity, so only a limited number of settings are viable. Second, in our experiments we found many had numerical stability issues for some datasets/tasks. We believe this may be due to trying to learn the local parameter patterns and the weights to combine them concurrently. Related work suggests this can be resolved by leveraging prior knowledge about what these local parameter patterns should represent (Denil et al., 2013). i.e., you define and freeze what they represent. However, prior knowledge not available in the general case, and data-driven methods of training these local filter patterns often rely on pretraining steps of the salllyparrmeterieed network (e.g., Denil et al. (003)). Thus, they are not suited for NPAS since they can not training large networks with low memory requirements, but addressing this issue would be good direction for future work.
(enil
Table compares approaches for mapping layers to parameter groups using the same number of parameters as the original model. We see a small, but largely consistent improvement over using a traditional (baseline) network using SSNs. Notably, our automatically learned mappings (auto) perform par with manual groups. This demonstrates that our automated approach can be used without loss in performance, while being applicable to any architecture, making them more flexible than hand-crafted methods. This flexibility does come with computational cost, step that learns to map layers to parameter groups resulted in a 10-15% longer training time for equivalent epochs. That said, parameter sharing methods have demonstrated an ability converge
preliminary
8

---

Published as a conference paper at ICLR 2022
(a) Random subset filters from a WRN-16-2 model trained on C-100
(b) Parameter groups for WRN-28-10
P 14 parameter groups (maximum sharing possible
No Sharing
SSN (1 Param Group)
with the manual assignments Savarese Maire, 2019)
Manual:
Layer
30000
BOOD
CSW
Layer 11
Dano 10030
City AOOOOO
Auto (ours):
SONEO
Con
Layer 15
Corn Zcalo
Parameter group
3
Figure 4: Qualitative comparison the (a) generated weights for a WRN-16-2 and (b) the parameter groups for WRN-28-11 10 model trained on C-100. See Section 4.2 for discussion.
Table 6: Parameter upsampling comparison (Section 3.1.2 "Reduced Baseline" adjusts the number and/or size of filters rather than share parameters. See Appendix B for the
Table 7: Compares methods of creating parameter groups (Section 3.2). These results demonstrate that our automatically learned mappings (auto) pro- vide more flexibility without reducing performance compared to manual mappings.
original models' performance.
% orig Reduced
SSNs (ours)
SSNs (ours)
Dataset
params
Baseline
Inter Mask
Dataset
Baseline
Single Random Manual Auto
C-10
6.35
6.05 5.07
C-10
3.57
3.71
3.63
3.38
3.42
1.4%
C-100
28.35
26.84 25.20
C-100
19.44
19.99
20.36
19.29
19.24
ImageNet 2.2%
15.20
14.83
14.19
ImageNet
5.84
6.18
5.91
5,82
5.86
faster Bagherinezhad et al. 2017 Lan et al. 2020). Thus, exploring more efficient training strategies using NPAS methods like SSNs will be good direction for future work.
Figure a) compares the 3 x kernel filters at the early, middle, and final convolutional layers of WR--16-2 of traditional neural network (no parameter sharing) and our SSNs where all layers belong to the same parameter group. We observe a correspondence between filters the early layers, but this diverges in deeper layers. This suggests that sharing becomes more difficult the final layers, which is consistent with two observations we made about Figure b)), which visualizes parameter groups used for SSN-WRN-28 WRN-28-11 to create 14 parameter group mappings. First, we found the learned parameter group mappings tended to share parameters between layers early in the network, opting for later layers share no parameters. Second, the early layers tended to group layers into 3-4 parameter stores across different runs, with the remaining 10-11 parameter stores each containing single layer. Note that these observations were consistent across different random initializations.
5 CONCLUSION
We propose NPAS, novel task in which the goal is implement a given, arbitrary network architecture given a fixed parameter budget. This involves identifying how to assign parameters to layers and implementing layers with their assigned parameters (which may of any size). To address NPAS, we introduce SSNs, which automatically learn how to share parameters. SSNs benefit from parameter sharing in the low-budget regime-reducinn memory and communication requirements when training-aan and enable novel high-budget regime that can improve model performance. We show that SSNs boost results on ImageNet by 3% improvement in Error@5 over a same-sized network without parameter sharing. Surprisingly, very different layers. Further, we show that SSNs can be combined with knowledge distillation and parameter pruning achieve state-of-the- of--he--rttttt results that also reduce FLOPs at test time. One could think of SSNs as spreading the same number parameters across more layers, increasing effective depth, which benefits generalization (Telgarsky 2016), although this requires further exploration.
also find that parameters can be shared among

---

Published as a conference paper at ICLR 2022
This work is funded in part by grants from the National Science Foundation and DARPA. This project received funding from the European Research Council (ERC) under the European Union's s Horizon 2020 program (grant agreement MAELSTROM, No. 955513). N.D. is supported by the ETH Postdoctoral Fellowship. We thank the Livermore Computing facility for the of their GPUs for some experiments.
Acknowledgements.
ETHICS STATEMENT
Neural Parameter Allocation Search and Shapeshifter Networks are broad and general-purpose. and therefore applicable to many downstream tasks. is thus challenging to identify specific cases of benefit or harm, but we note that reducing memory requirements can have broad implications. In particular, this could allow potentially harmful applications to be cheaply and widely deployed (e.g., facial recognition, surveillance) where it would otherwise be technically economically infeasible.
REPRODUCIBILITY STATEMENT
NPAS is a task which can be implemented in many different ways; we define it formally in Section 2 SSNs, our proposed solution NPAS, are presented in detail in Section 3 and Figure provides an illustration and example for weight generation methods. Appendi also provides a thorough discussion of the implementation details. To further aid reproducibility. we publicly release our SSN code at httpp:::///gtt //t:::////tttt com/ Bryan? Lummer/ee /SSN
REFERENCES
Ferran Alet, Erica Weng, Tomás Lozano-Pérez, and Leslic Pack Kaelbling. Neural relational inference with fast modular meta-learning In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Hessam Bagherinezhad, Mohammad Rastegari, and Ali Farhadi. LCNN: Lookup-based convolu-
tional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Pouya Bashivan, Mark Tensen, and James J. DiCarlo. Teacher guided architecture search. In The IEEE International Conference on Computer Vision (ICCV), 2019.
Andrea Burns, Reuben Tan, Kate Saenko, Stan Sclaroff, and Bryan A. Plummer. Language features matter: Effective language representations for vision-language tasks. In The IEEE International Conference on Computer Vision (ICCV), 2019.
Ernie Chan, Marcel Heimlich, Avi Purkayastha, and Robert Van De Geijn. Collective communication: theory, practice, and experience. Concurrency and Computation: Practice and Experience, 19(13): 1749-1783, 2007.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. 16440: 77400/1/44111 2016.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations (ICLR), 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
10

---

Published as a conference paper at ICLR 2022
Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurelio Ranzato, and Nando de Freitas. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems (NeurIPS), 2013.
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In International Conference on Robotics and Automation, 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (NAACL), 2019.
Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv: /70.0055000 2017.
Xuanyi Dong and Yang. One-shot neural architecture search via self-evaluated template network. In The IEEE International Conference Computer Vision (ICCV), 2019.
Leslie Pack Kaelbling Ferran Alet, Tomas Lozano-Perez. Modular meta-learning. In Conference Robot Learning (CoRL), 2018.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer.
A survey of quantization methods for efficient neural network inference. arXiv preprint MxXiv:2103.13633 2021.
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network:
Backpropagation without storing activations. In Advances Neural Information Processing Systems (NeurIPS), 2017.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A
survey. International Journal of Computer Vision, 129(6), 2021.
David Ha, Andrew Dai, and Quoc Le. Hypereetorrss. In International Conference on Learning Representations (ICLR), 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2016.
Byeongho Heo, Jeesoo kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In The IEEE International Conference on Computer Vision (ICCV), 2019.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241), 2021.
Gao Huang. Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals. and Joao Carreira. Perceiver: General perception with iterative attention. In International Conference Machine Learning (ICML), 2021.
Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Kurt Keutzer, Ion Stoica, and Joseph E Gonzalez. Checkmate: Breaking the memory wall with optimal tensor rematerialization. In Proceedings of the Third Conference on Machine Learning and Systems (MLSys), 2020.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferltGame: Referring to objects in photographs of natural scenes. In Empirical Methods in Natural Language Processing (EMNI.P), 2014.
11

---

Published as a conference paper at ICLR 2022
Jakub Konečný, H. Brendan McMahan, Felix X. Yu, Peter Richtarik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. In NeurlPS Workshop Private Multi-Paryy Machine Learning, 2016.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. In International
Conference Learning Representations (ICLR), 2020.
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. HRank: Filter pruning using high-rank feature map. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In The European Conference Computer Vision (ECCV), 2014.
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication--ffiiiee learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pp. 1273-1282, 2017.
Anh-Huy Phan, Konstantin Sobolev, Konstantin Sozykin, Dmitry Ermilov, Julia Gusak, Petr Tichavsky, Valeriy Glukhov, Ivan Oseledets, and Andrzej Cichocki. Stable low-rank tensor decomposition for compression of convolutional neural network. In The European Conference Computer Vision (ECCV), 2020.
Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30K entities: Collecting region-to-phrass correspondences for richer Image-to-sentenee models. International Journal of Computer Vision, 123(1):74 93, 2017.
Bryan A. Plummer, Kevin J. Shih, Yichen Li, Ke Xu, Svetlana Lazebnik. Stan Sclaroff, and Kate Saenko. Revisiting imae--lanuuage networks for open-ended phrase detection. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020.
Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. ZeRO-Infinit;: Breaking the GPU memory wall for extreme scale deep learning. In International Conference for High Performance Computing, Networking, Storage, and Analysis (Supercomputiing) 2021.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing (EMNLP), 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (ACL), 2018.
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. ZeRO-Offload: Democratizing billion-scale model training. In USENIX Annual Technical Conference, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
Cèdric Renggli, Saleh Ashkboos, Mehdi Aghagolzadeh, Dan Alistarh, and Torsten Hoefler. SparCML: High-performance sparse communication for machine learning. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (Supercomputing), 2019.
Pedro Savarese and Michael Maire. Learning implicitly recurrent CNNs through parameter sharing. In International Conference on Learning Representations (ICLR), 2019.
12

---

Published as a conference paper at ICLR 2022
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gated mixture-of-experts layer. In International Conference Learning Representations (ICLR), 2017.
Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning (ICML), 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. MnasNet: Platform-awan neural architecture search for mobile. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Matus Telgarsky. Benefits of depth in neural networks. In Conference Learning Theory (COLT), 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Liwei Wang, Yin Li, and Svetlana Lazebnik. Learning deep structure-preserving image-text embed- dings. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2016.
Liwei Wang, Yin Li, Jing Huang, and Svetlana Lazebnik. Learning two-branch neural networks for image-text matching tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), (1),,, 2018.
Jonatas Wehrmann, Camila Kolling, and Rodrigo C. Barros. Adaptive cross-modal embeddings for image-text alignment. In AAAI Conference on Artificial Intelligence (AAAI), 2020.
Chai Wah Wu. Prodsummet: reducing model parameters in deep neural networks via product-of-sums matrix decompositions, 2019.
Yunyang Xiong, Ronak Mehta, and Vikas Singh. Resource constrained neural network architecture search: Will a submodularity assumption help? In The IEEE International Conference on Computer Vision (ICCV), 2019.
Jing Yang, Brais Martinez, Adrian Bulat, and Georgios Tzimiropoulos. Knowledge distillation via softmax regression representation learning. In International Conference on Learning Representa- tions (ICLR), 2021.
Chengxi Ye, Matthew Evanusa, Hua He, Anton Mitrokhin, Tom Goldstein, James A. Yorke, Cornelia Fermuller, and Yiannis Aloimonos. Network deconvolution. In International Conference Learning Representations (ICLR), 2020.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics (TACL), 2, 2014.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference (BMVC), 2016
Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning. International Conference Learning Representations (ICLR), 2017.
13

---

Published as a conference paper at ICLR 2022
A DESCRIPTION OF COMPARED TASKS
A.1 IMAGE-SENTENCE RETRIEVAL
In bidirectional image-sentence retrieval when a model is provided with an image the goal is to retrieve a relevant sentence and vice versa. This task is evaluated using Recall @ K=(1, 5, 10) for both directions (resulting in 6 numbers), which we average for simplicity. We benchmark methods on two common datasets: Flickr30K (Young et al. 2014) which contains 30K/IK/IK images for taiiningttsting/vlliaatte each with five descriptive captions, and MSCOCO (Lin et al., 2014), which contains I23K/IK/IK images for baininngtesting//aliattioo each image having roughly five descriptive captions.
EmbNet Wang et al. 2016). This network learns to embed visual features for each image computed using 152-layer Deep Residual Network (ResNet) (He et al.) 2016) that has been trained on ImageNet (Deng et al. 2009) and the average of MT GrÖVLE (Burns et al.) 2019) language features representing each word into a shared semantic space using a triplet loss. The network consists of two branches, one for each modality, and each branch contains two fully connected layers (for total of four layers). We adapted the implementation of Burns el al. (Burns et al. 2019 I and left all hyperparameters at the default settings. Specifically, we train using a batch size of 500 with initial learning rate of lc-4 which we decay exponentially with a gamma of 0.794 and use weight decay of 0.001. The model is trained until it has not improved performance 10 epochs. This architectures provides a simple baseline for parameter sharing with our Shapeshifter Networks (SSNs), where layers operate on two different modalities.
the validation set over the last
ADAPT-T21 (Wehrmann et al. 2020). In this approach word embeddings are aggregated using a bidirectional GRU (Cho et al. 2014)) and its hidden state at each timestep is averaged obtain a full- sentence representation. Images are represented using 36 bottom-up image region features (Anderson) jet al. 2018) that are passed through a fully connected layer. Then, each sentence calculates scaling and shifting parameters for the image regions using pair of fully connected layers that both take the full-sentence representation as input. The image regions are then averaged, and a similarity score is computed between the centence-adapted image features and the fully sentence representation. Thus, there are four layers total (3 fully connected, GRU) that can share parameters, including the two parallel fully connected layers (i.e., they both take the full sentence features input, but expected to have different outputs). We adapted the author's implementation and kept the default hyperparameter/ Specifically, we use latent dimension of 1024 for our features and train with a batch size of 105 using learning rate of 0.001. This method was selected since it achieves high performance and also included fully connected and recurrent layers, as well as having a set of parallel layers that make effectively performing cross-layer parameter sharing more challenging.
A.2 PHRASE GROUNDING
Given a phrase the goal of a phrase grounding model is to identify the image region described by the phrase. Success is achieved if the predicted box has at least 0.5 intersection over union with the ground truth box. Performance measured using the percent of the time a phrase is accurately localized. We evaluate on two datasets: Flickr30K Entities (Plummer et al. 2017) which augments the Flickr30K dataset with 276K bounding boxes associated with phrases in the descriptive captions, and ReferIt (Kazemzadeh et al., 2014) which contains 20K images that are evenly split between training/testing and 120K region descriptions.
SimNet (Wang et al. 2018). This network contains three branches that each operate on different types of features. One branch passes image regions features computed with 101-layer ResNet that have been fine-tuned for phrase grounding using two fully connected layers. A second branch passes MT GrOVLE features through two fully connected layers. Then, a joint representation is computed for all region-phrase pairs using an elementwise product. Finally, the third branch passes these joint features through three fully connected layers (7 total), where the final layer acts classifier indicating the likelihood that phrase is present in the image region. We adapt the code
14

---

Published as a conference paper at ICLR 2022
from Plummer et al. (2020 and keep all hyperameters at their default settings. Specifically, a pretrained Faster R- R-CNN model (Ren et al. 2013) fine-tuned for phrase grounding by Plummer et al. (2020) on each dataset to extract region features. Then we encode each phrase by averaging MT' GrOVLE features (Burns et al., 2019) and provide the image and phrase features as input to our model. We train our model using learning rate of 5e-5 and final embedding dimension of 256 until it no longer improves on the validation set for 5 epochs (typically resulting in training times of 15-20 epochs). Performing experiments on this model enables us test how well our SSNs generalize to another task and how well it can adapt to sharing parameters with layers operating on three types of features (just vision, just language, and a joint representation).
A.3 IMAGE CLASSIFICATION
For image classification the goal is be able recognize if an object is present in an image. Typically this task is evaluated using Error@ K. or the portion of times that the correct category doesn't 't appear in the top & most likely objects. We evaluate our Shapeshifter Networks on three datasets: CIFAR-10 and CIFAR-100 (Krizhevsky] (009)). which are comprised of 60K images of 10 and 100 categories, respectively, and ImageNet (Deng et al. 2009), which comprised of 1 1.2M images containing 1 1,000 categories. We report Error for both CIFAR datasets and Error @5 for ImageNet. In these appendices, we also report Error@ I for ImageNet.
Wide Residual Network (WRN) (Zagoruyko & Komodakis 2016). WRN modified the traditional ResNets by increasing the width i of each layer while also decreasing the depth d. which they found improved performance. Different variants are identified using wRN-d-k. Following Savarese et al. (Savarese & Maire 2019). we evaluate our Shapeshifter Networks using WRN-28- 10 for CIFAR and WRN-50-2 for ImageNet. We adapt the implementation of Savarese el al and use cutout (De- Vries & Taylor 2017) for data augmentation. Specifically, on CIFAR we train our model using a batch size of 128 for 200 epochs with weight decay set at 5e-4 and an initial learning rate of 0.1 which we decay using gamma of 0.2 at 60, 120, and 160 epochs, Unlike the vision-language models discussed earlier, these architecture include convolutional layers in addition fully connected layer used to implement classifier, and also have many more layers than the shallow vision-language models.
DenseNet (Huang et al, 2017). Unlike traditional neural networks where each layer in the network computed in sequence, every layer in a DenseNet using feature maps from every layer which came before it. We adapt PyTorch 's official implementation juiin the hyperparameters as set in Huang er al. (Huang et al. 2017). Specifically, on CIFAR we train our model using a batch size of 96 for 300 epochs with weight decay set at le-4 and initial learning rate of 0. which we decay using gamma of 0.1 at 150 and 225. These networks provide insight into the effect depth has on learning SSNs, as we use a 190-layer DenseNet-Bee configuration for CIFAR. However, due to their high computational cost we provide limited results testing only some settings
EfficientNet (Tan & Le, 2019). EfficientNets are class of model designed to balance depth, width, and input resolution in order to produce very parameter-efficient models. For ImageNet, we adapt an existing PyTorch implementation and its hyperparametersS) which are derived from the official TensorFlow version. We use the EfficientNet-BO architecture to illustrate the impact of SSNs on very parameter-efficient, state-of-the-an models. On C10 we use an EfficientNet with Network Deconvolution (ND) (Ye et al., 2020). which results in improved results with similar numbers of epochs for training. We use the author's implementation!] and train each model for 100 epochs (their best performing setting). Note that our best eтror running different configurations of their model (35.88) is better than those in their paper (37.63), so despite the relatively low performance is comparable to results from their paper.
hiteer//sthaas.ooo///r/a///////e httpa////wnaneeaaassseeeeettttoe detection
15

---

Published as a conference paper at ICLR 2022
A.4 QUESTION ANSWERING
In question answering, model is given question and an associated textual passage which may contain the answer, and the goal to predict the span of text in the passage that contains the answer. We versions of the Stanford Question Answering Dataset (SQuAD), SQuAD v1.1 (Rajpurkar) et al. 2016), which contains 100K+ question/answer pairs on 500+ Wikipedia particles, and SQuAD v2.0, which augments SQuAD vI.1 with 50K unanswerable questions designed adversarially be similar to standard SQuAD questions. For both datasets, we report both the Fl score, which captures the precision and recall of the chosen text span, and the Exact Match (EM) score.
ALBERT (Lan et al. 2020) ALBERT is a version of the BERT (Devlin et al. 2019) transformer architecture that applies cross-layer parameter sharing. Specifically, the parameters for all components of a transformer layer are shared among all the transformer layers in the network. ALBERT also includes factorized embedding to further reduce parameters. We follow the methodology of BERT and ALBERT for reporting results on SQuAD, and our baseline ALBERT scores closely match those reported in the original work. This illustrates the ability of NPAS and SSNs to develop better parameter sharing methods than manually-desiee
systems for extremely large models.
B EXTENDED RESULTS WITH ADDITIONAL BASELINES
Below we provide additional results with more baseline methods for the three components of our SSNs: weight generator (Section B.I parameter upsampling (Section B.4), and mapping layers to parameter groups (Section B.3). We provide ablations on the number of parameter groups and templates used by our SSNs Section Cand Section D respectively.
B.1 ADDITIONAL METHODS THAT GENERATE LAYER WEIGHTS FROM TEMPLATES
Parameter downsampling uses the selected templates T% for a layer l to produce its weights w;. In Section 3.1.1 of the paper we discuss two methods of learning a combination the T to generate wy. Below in Section B.2 we provide two simple baseline methods that directly the candidates. Table [ compares the baselines to the methods in the main paper that learn weighted combinations of templates, where the learned methods typically perform better than the baselines.
B.2 DIRECT TEMPLATE COMBINATION
Here we describe the strategies we employ that require no parameters to learned by weight generator, i.e., they operate directly the templates T.k.
Round Robin (RR) reuses parameters of each template set as few times as possible. The scheme simply returns the weights at index & mod K in the (ordered) template set T. at the kth query of parameter group
Candidate averaging (Avg) averages all candidates in T, to provide a naive baseline for using multiple candidates. A significant drawback of this approach is that, if K is large. this can result in reusing parameters (across combiners) many times with no way to adapt to a specific layer, especially when the size of the parameter group is small.
B.3 ADDITIONAL PARAMETER MAPPING RESULTS
Table compares approaches that map layers to parameter groups using the same number of pa- rameters as the original model. We see small, but largely consistent improvement over using traditional (baseline) network. Notably, our learned grouping methods (WAvg, Emb) perform on par, and sometimes better than using manual mappings. However, our approach can be applied to any architecture to create a selected number of parameter groups, making them more flexible than hand-crafted methods. For example. in Table I0, performance when using very few parameters, but it not clear how to effectively create two groups by hand for many networks.
using two groups often helps improve
16

---

Published as a conference paper at ICLR 2022
Table Weight generation method comparison. See Section B.1 for description of the bascline methods (RR, Avg), and Section 3.1.1 of the paper for the learned candidates (WAvg, Emb). All models in the same row use the same (reduced) number of parameters. "Reduced Baseline" uses no parameter sharing, but adjusts the number and/or size of filters they have the same number of parameters as our SSNs. The original model's performance is reported the first column of results in Table
Onig Num % of Params
Reduced
SSN Variants (ours)
Method
Params (M)
Used
Dataset
Baseline
RR
Avg
WAvg Emb
Image-Sentence Retrieval
(f better)
F30K
72.8
74.1
73.5
74.0
74.3
EmbNet
57.1%
COCO
80.9
80.8
80.9
81.2
81.5
F30K
80.6
81.2
80.5
81.6
81.7
ADAPT-T2I
14
70.9%
COCO
85.2
85.6
85.8
86.2
85.9
Phrase Grounding
(1 better)
F30K Entities
71.1
72.3
72.1
72.3
72.5
SimNet
33.1%
ReferIt
59.4
60.5
60.2
60,5
60.4
Image Classification
(4 better)
C-10
4.22
4.09
4.19
4.00
3.84
WRN-28-10
36
11.3%
C-100
22.34
21.91
22.78
21.78
21.92
ImageNet H@5
10.08
6.69
7.61
7.38
6.69
WRN-50-2
69
27.5%
ImageNet H@!
28.68
23.32
25.11
24.15
23.25
Table 9: Parameter mapping comparison (described in Section 3.2 of the paper). Each method
uses the same number of parameters and the baseline represents no parameter sharing. Notably, these results demonstrate how our automatically learned groups can provide greater flexibility when sharing parameters while performing on par or better than manually created groups (which are illustrated in Figure
SSN Variants (ours)
Method
#Groups Dataset
Baseline Single Random Manual WAvg
Emb
Image-Sentence Retrieval
(f better)
F30K
74.1
74.4
74.0
74.3
74.2
74.3
EmbNet
COCO
81.4
82.1
81.5
81.7
81.7
81.9
F30K
83.3
82.1
81.9
82.0
82.6
82.9
ADAPT-T2I
2
COCO
86.8
86.1
86.3
86.1
86.4
87.0
Phrase Grounding
(f better)
F30K Entities
71.7
71.4
71.8
72.4
72.2
72.1
SimNet
ReferIt
61.1
60.9
60.0
60.2
61.0
60.5
Image Classification
(} better)
C-10
3.57
3.71
3.63
3.38
3.51
3.42
WRN-28-10
14
C-100
19.44
19.99
20.36
19.29
19.47
19.24
ImageNet E@5
5.84
6.18
5.91
5.82
5.86
5.96
WRN-50-2
32
ImageNet E@1
21.50
22.09
21.54
21.43
21.59
21.61
B.4 EXTENDED PARAMETER UPSAMPLING
In Table TO we provide extended results comparing the parameter upsamping methods. We addi- tionally compare with further naïve baseline simply repeating parameters until they are the appropriate size. We find that Mask upsampling is always competitive, and typically moreso when two parameter groups are used.
B.5 COMPARISON WITH HYPERNETWORKS
In Table II we compare our SSNs Wide ResNets (Ha et al. 2016) to the networks imple- mented using Hypernetoorss (Ha et al. 2016) for CIFAR-ID 10, using the results reported in their paper. We that, for the same parameter budget, SSNs outperform Hypernetworks.
17

---

Published as a conference paper at ICLR 2022
[A]
EmbNNet
[b]
SimNet
Text
300-2048
FC
204B-512
rC
300-1024
1020x1024
1024x512
512x1
Image
FC
204Bx512
FC
Image Region
1009N1004
ADAPT-122
WRN (CIRAR-
Text
600-1024
BI-GRU
1024-1024
FC
Image
Corvv 3x3x15
Convx2 3x3x150
Conve5
3x3-160
Convx2
S+3+320
Convi6
3x3-320
Conwx2
3*3x640
Convx6
3-3-640
FC 64D-100
Image
FC
FC
Region
2048+1024
1004HL024
1-1-320
Figure 5: Examples of architectures used evaluate SSNs. Fully connected (FC) and bidirectional GRU (Bi-GRU) layers give their in X out dimensions; convolutional layers (conv) give the number of layers. kernel size, and number of filters. Same colored boxes show "manual" (Table sharing. except for green boxes in WRN, whose layers share no parameters. Layers without parameters omitted.
Table 10: Parameter upsampling comparison. Most methods are described in Section 3.1.2 of the paper, except the method "repeat" that simply duplicates weights when upsampling. "Reduced Baseline" uses no parameter sharing, but adjusts the number and/or size of filters they have the same number of parameters as our SSNs. The original model's performance reported the first column of results in Table
Orig Num % of Params
Reduced Baseline Repeat
SSN Variants (ours)
Method
Params (M)
Used
Dataset
Inter Mask Groups+Mask
Image-Sentence Retrieval
( better)
F30K
56.5
64.8
67.5
67.3
67.9
EmbNet
6
6.9%
COCO
74.2
75.5
76.1
76.2
76.7
F30K
76.6
79.0
80.9
80.3
81.2
ADAPT-T21
14
34.7%
COCO
83.8
84.9
85.3
84.6
84.9
Phrase Grounding
(1 better)
F30K Entities
70.6
71.1
71.8
71.5
71.9
SimNet
6
7.8%
ReferIt
58.5
58.9
59.3
59.2
59,5
Image Classification
(J better)
C-10
6.35
5.43
6.05
5.18
5.07
WRN-28-10
36
1.4%
C-100
28.35
27.08
26.84 25.71
25.20
ImageNet E@5
15.20
14.16
14.83
14.19
14.58
WRN-50-2
69
2.2%
ImageNet E@
37.05
35.48
36.60 35.72
36.01
C EFFECT OF THE NUMBER OF PARAMETER GROUPS P
significant advantage of using learned mappings of layers parameter groups, described in Section 3.2 is that our approach can support any number of parameter groups, unlike prior work that required manual grouping and/or heuristies determine which layers shared parameters (e.g., Lan et al. 2020 Savarese & Maire 2019). In this section we explore how the number of parameter groups
Table 11: Comparison of SSNs to Hypernetworks Ha et al. 2016) on CIFAR-10 R-10.
Method
Error@
#Params (M)
Wide ResNet 40-1
6.73
0.563
Hyper Wide ResNet
8.02
0.097
SSN (ours)
7.22
0.097
Wide ResNet 40-2
5.66
2,236
Hyper Wide ResNet
7.23
0.148
SSN (ours)
6.56
0.148
18

---

Published as a conference paper at ICLR 2022
Table 12: Effect the number of learned parameter groups P has on performance for a SSN-WRN-28-
10 model when training on C- using the Emb strategy. See Section C for discussion.
#Groups
2
4
12
14
16
4M Params 21.92 0.30 21.47 = 0.23 21.31 = 0.13 22.01 0.26 22.43 = 0.37 23.88 = 0.40
24.99 = 0.39
36M Params 19.99 0.22 19.80 = 0.38
19.75 = 0.29 19.69 0.28 19.27 = 0.10 19.24
0.29
19,40 0.14
Table 13: Effect the number of templates K has on performance. Share type is set using the best results from the main paper for each method. Due to the computational cost, we the number of candidates for ImageNet experiments in our paper using CIFAR results. See Section D for discussion
Method #Candidates
2
4
6
8
10
Image-Sentence Retrieval
F30K
73.7 + 0.55
74.3 + 0.25
73.9 + 0.18
74.3 + 0.14
74.1 + 0.16
EmbNet
COCO
81.7 + 0.26
81.9 + 0.30
81.6 + 0.23
81.7 + 0.16
81.5 = 0.19
Phrase Grounding
F30K Entities
72.1 +: 0.18
72.2 + 0.34
72.0 + 0.33
71.6 + 0.29
71.8 + 0.25
SimNet
Referlt
60.7 + 0.37
61.0 + 0.43
60.8 + 0.39
60.4 + 0.45
60.5 + 0.43
Image Classification WRN-28-10 C-100
19.66 + 0.23 19.59 + 0.21 19.48 + 0.24 19.24 + 0.29 19.32 + 0.27
effects performance on the image classification task. We do benchmark bidirectional retrieval and phrase grounding since networks addressing these tasks have few layers, so parameter groups are less important (as shown in Table 7
Table 12 reports the performance of our SSNs when using different numbers P parameter groups. We find that when training with few parameters (first line) low numbers of parameter groups work best, while when more parameters are available larger numbers of groups work better (second line). In fact, there a significant drop in performance going from 4 to 8 groups when training with few parameters seen in the first line of Table 12 This is due to the fact that starting 8 groups some parameter groups had too few weights to implement their layers, resulting in extensive parameter upsampling. This suggests that may be able to further improve performance when there few parameters by developing better methods of implementing layers when too few parameters are available.
D EFFECT OF THE NUMBER OF TEMPLATES K
Table 13 reports the results using different numbers of templates. We find that varying the number of templates only has minor impact on performance most of the time. We note that more templstes tends to lead to reduced variability between runs, making results more stable. As reminder, however, the number of templates does not guarantee that each layer will have enough parameters to construct them. Thus, parameter groups only this hyperparameter when many weights are available it (i.e., it can form multiple templates for the layers it implements). This occurs for the phrase grounding and bidirectional retrieval results at the higher maximum numbers of templates.
E SCALING SSNS TO LARGER NETWORKS
Table 14 demonstrates the ability of our SSNs significantly reduce the parameters required, and thus the memory required to implement large Wide ResNets so they fall within specific bounds. For example, Table b4) shows larger and deeper configurations continue to improve performance even when the number of parameters remains largely constant. Comparing the first line of Table [4 and the last line of Table 14 c) we see that SSN-WRN-76-12 outperforms the pully-parameterized WRN- 28-10 network by 0.6% on CIFAR- 100 while only using just over half the parameters, and comes within 0.5% of WRN-76-12 12 while only using 13.0% of its parameters. We do note that using a SSN does not reduce the number of floating point operations, so although our CON-WRN-76-1 12 model uses fewer parameters than the WRN-28- 10, it is still slower at both test and train time. However, our results help demonstrate that SSNs can be used to implement very large networks with lower memory
19

---

Published as a conference paper at ICLR 2022
Table 14: Comparison of different Wide Resnet (Zaorrukko & Komodakis 2016) configurations under different Shapeshifter Network settings on the image classification task. (a) contains baseline results without parameter sharing, while (b) demonstrates that using SSNs enable improve performance by increasing the architecture width depth without also increasing memory consumed, and (c) shows that the gap between SSNs with few model weights and fully-parameterizedd baselines in (a) gets smaller as the networks get larger. Note that the number of parameters in (c) was set by summing together the number of parameters required to implement the largest layer in the learned parameter group mappings for each network (e.g., four parameter groups would result in summing four numbers). See Section for discussion.
Orig Num
#Params
% of Orig
% of WRN-28-10 10
CIFAR-100
Method
Params (M)
Used (M)
Params
Params
Error I
(a)
WRN-28-118
36
36
100.0
100.0
19.44 = 0.21
WRN-40-10
56
100.0
153.1
18.73 + 0.34
WRN-52-10
75
75
100.0
206.2
18.85 + 0.31
WRN-76-12
164
164
100.0
449.6
18.31 + 0.22
(h)
SSN-WRN-28-10
36
4
11.3
11.3
21.71 = 0.13
SSN-WRN-40-10
56
4
6,8
10.3
21.06 + 0.11
SSN-WRN-52-10
75
5.0
10.4
20.93 0.25
SSN- SSN-WRN-76-12
164
5
3.3
14.9
20.37 0.21
(c) SSN. WRN-28-10
36
10.3
10.3
21.78 + 0.16
SSN- MANN-WNN44--L1E
56
13.4
20.4
20.17 = 0.32
SSN- WRN-52-10
75
15
19.7
40.7
19.50 + 0.21
SSN-WRN-76-12
164
21
13.0
58.6
18.83 + 0.19
Table 15: Performance of SSNs vs. traditional networks of comparable size on image classification on CIFAR-100 (from Figure of the main paper). See Section for discussion.
Method
Performance
% orig params
1,4%
4.1%
5.5%
16.4%
24.6%
WRN (Huang et al. 2017
28.35
24.50
21.05 22.19
23.45
21.36
SSN-WRN-20
25.75
23.08
20.54 20.11
% orig params
32.8%
46.5%
SWRN-28- 10 Savarese & Maire
2019)
20.48
20.04
% orig params
5.5%
SSN-WRNN-76-12-99
19.16
% orig params
1.9%
3.8%
7.7%
DenseNet-BC (Haan et al.
2017
24.85
22.27
21.79
MON-DenseNet-Bee 19-
22.97
21.03
19.65
% orig params
9.7%
17.9%
50.0%
75.1%
EfficientNet-ND Ye et al.
2020
44.17
42.00
36.80 35.88
ND
35.72
35.75
35.57
34.99
requirements by effectively sharing parameters. This enables us to train larger, better-performing networks than is possible with traditional neural networks comparable computational resources.
F IMAGE CLASSIFICATION NUMBERS
We provide raw numbers for the results in Figure inn Table
15 (CIFAR-100) and Table 16 (ImageNet).
G PERFORMANCE IMPLICATIONS OF NPAS AND SSNS
Our SSNs can offer several performance benefits by reducing parameter counts; notably. they can reduce memory requirements storing model and reduce communication costs for distributed training. We emphasize that LB-NPAS does not reduce FLOPs. as the same layer operations are implemented using fewer parameters. Should fewer FLOPSs also be desired, SSNs be combined
20

---

Published as a conference paper at ICLR 2022
Table 16: Performance of SSNs vs. traditional networks of comparable size on image classification on ImageNet (from Figure 3 of the main paper). See Section 4.1 for discussion.
Method
Performance E @ (E@) I)
% orig params
5.0%
9.4%
15.2%
22.6%
WRN SSN-WRN-50-2
11.60 (31.81)
10.02
(29.22)
9.08 (27.58)
8.44 (26.30)
11.07 (30.58)
9.02 (27.46)
8.10 (25.64)
7.42 (24.28)
% orig params
4.2%
14.8%
23.9%
EfficientNet
10.84 (33.68)
7.98 (28.47)
7.53 (27.61)
BSN--
10.21 (32.42)
6.82 (26.52)
6.44 (25.18)
with other techniques, such as pruning. Additionally, we note that our implementation has not been extensively optimized, and further performance improvements could likely be achieved with additional engineering.
G.1 COMMUNICATION COSTS FOR DISTRIBUTED TRAINING
Communication for distributed data-parallel training is typically bandwidth-bound, and thus employs bandwidth-optime allreduces, which are linear in message length (Chan et al. (007). Thus, expect communication time to be reduced by a factor proportional to the parameter savings achieved by NPAS, all else being equal. However, frameworks will typically execute allreduces layer-wist as gradient buffers are ready to promote connunication/cooppune reducing communication that is already fully overlapped is of little benefit. Performance benefits thus sensitive to the model, implementation details, and the system being used for training.
overlap in backpropagation:
For CNNs, we indeed observe minor performance improvements, as the number of parameters is typically small. When using 64 V100 GPUs for training WRN-50-2 on ImageNet, we see 1 1.04 performance improvement in runtime per epoch when using SSNs with 10.5M parameters (15% of the original model). This is limited because most communication is overlapped. We also observe small performance improvements some cases because we launch fewer allreduces, resulting less demand for SMs and memory bandwidth on the GPU. These performance results are in line with prior work on communication compression for CNNs Renggli al. 2019).
For large transformers, however, we observe more significant performance improvements. The SSN-ALBERT-Large is about 1.4x faster using 128 GPUs than the corresponding BERT-Large model. This is in line with the original ALBERT work (Lan et al. 2020), which reported that training ALBERT-Large was 1.7x x faster than BERT-Large when using 128 TPUs. Note that due to the differences in the systems for these results, they not directly comparable.
We would also reiterate that for some applications where communication is more costly, say, for federated learning applications (e. McMahan et al. 2017); Konečný et al. (2016), approach would be even more beneficial due to the decreased message length.
G.2 MEMORY SAVINGS
LB-NPAS and SSNs reduce the number of parameters, which consequentially reduces the size the gradients and optimizer state (e.g., momentum) by the same amount. It does not reduce the storage requirements for activations, but note there is much work on recomputation
address this (e.g.,
Chen et al.) 2016 Jain al.) 2020). Thus, the memory savings from SSNs is independent of batch size. For ASN-ALLBERT-aagee we use 18M parameters (5% BERT-Large, which contains 334M parameters). Assuming FP32 is used to store data, save about 5 GB of memory in this case (about 1/3 of the memory used)
21


Reviews:
[{"review_id": "jaePe7hPcJ", "reviewer": "Reviewer_9sBX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The topic introduced in this paper is interesting and meaningful, which provides a novel idea for saving parameters. The proposed parameter upsampling and downsampling methods are promising.\n\nHere are some questions to be answered.\n1. The parameter number is indeed saved. However, how about the real performance on the hardware platforms, e.g., bandwidth, memory consumption, training speed? It is essential to report these results for evaluating the real value of the method.\n2. How much benefit does the NPAS method bring? It is necessary to perform an experiment similar to \"random search\" in NAS to validate the effectiveness. Noting that in Fig. 4 (b), differences between the manual and auto one are not so evident, where it seems only the downsampling layers require unique parameters as in the manual one.\n3. How are the parameters allocated in depthwise layers, e.g., in EfficientNet? Is there any difference between depthwise and plain convolutions?\n4. Is the proposed method specified to tasks? It is interesting to explore whether parameters are allocated differently in different tasks.", "summary_of_the_paper": "This paper proposes to solve an interesting but meaningful task, i.e., learning to allocate parameters between layers; in other words, searching for the parameter sharing strategy in a neural network. Parameter sharing between serial layers is useful for saving parameter numbers, which is beneficial for issues including memory consumption, bandwidth, etc. ", "summary_of_the_review": "The topic is interesting but the concerns listed above need to be cleared. The main concerns lie in the real-application value and real benefit of the methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ib5ghN4EbX3", "reviewer": "Reviewer_jt4B", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nI commend the authors on extensive evaluation across multiple datasets, models, and tasks are convincing of the broad applicability of this approach. Not an expert in the literature, but during my investigation, the PNAS approach seems like a reasonably novel mechanism for parameter allocation and automatically learning which weights to share. Methodologically interesting approach, with implications for studies on generalizability. \n\nWeaknesses\n\nThe practical impact of this approach is overclaimed in this paper. For example, while the SSN reach lower error compared to the EfficientNet family as the same # of trainable parameters, the EfficientNet family (and ALBERT, WRN, etc.) are designed to have lower inference time, which SSN does not achieve.\n\nFigure 1 marks both the author's method (NPAS) and previous work in cross-layer sharing (presumably Savarasee & Maire, 2019) as \"make training more efficient\", but the effects are dramatically different. NPAS's efficiency is in lower memory footprint, whereas prior work reduces the FLOPs and therefore the training time (Table 4, Savarase & Maire, 2019). \n\nWhile shared parameters does reduces the communication during data parallel training, the authors provide no data or reasoning for the magnitude of that effect on training time. Given that CNNs already have relatively low parameter count, I suspect that the effect on the model sync time during distributed training is minimal. \n\nA meta-comment: parameter counts may be the wrong metric to compare model capacity in these studies, as the methods you compare against (e.g. the EfficientNet_B0 family) have reduce parameters by removing from network, whereas NPAS reduced by sharing weights. For example, would path norm be more appropriate from a generalization studies perspective, as hinted in your conclusion? This is admittedly out of scope for this paper, as the paper's main claims are on efficiency.\n\nMy score could be improved by:\n* Figure 1: breaking out \"makes training more efficient\" into \"Reduces FLOPs\" and \"reduces memory footprint\" rows, and marking the methods appropriately. Citing the paper for prior work directly in the column or the caption.\n* In the discussion of results, acknowledging that these other methods reduce FLOps and training time, whereas the NPAS approach does not.\n* Providing data on the actual effect of this reduced number of trainable parameter counts on (1) model training times in a distributed training setting, and (2) memory usage on the GPU. Are these savings practically realizable?", "summary_of_the_paper": "Parameter sharing can reduce memory footprint of neural networks and memory bandwidth requirements, but existing methods require manually tuning the sharing strategy. This paper uses a small phase of training to cluster the learned layer representations by groups. This allows networks to be scaled from small to large parameter counts (to be clear, number of trainable parameters) without changing the model architecture. Importantly, this procedure does not change the number of FLOPs in the model. \n\nExperiments across a wide set of tasks and networks compare this approach with either (1) SWRN from Savarasese & Maire, 2019), or (2) existing hand-tuned parameter scaling from families of networks such as EfficientNet, DenseNet, or ALBERT.\n\n", "summary_of_the_review": "The proposed method is methodologically an interesting contribution, however the practical impacts on training efficiency and model efficiency are minimal and overstated, which limits my recommendation. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HKEUXTom3u2", "reviewer": "Reviewer_cCHH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Figure 1. shows comparison between distillation and pruning which seems to be a different class of methods. It is mentioned to be complementary to NPAS. It would be better to show competitive analysis on prior work or similar approaches.\n\nSlimmable networks is also a work presents a network that change parameters sizes automatically\n\nNPAS adds training cost per epoch, but the parameter sharing enables faster convergence. Demonstrating that the effective train time to reach same accuracy would improve the paper.\n\nThe explanation flow in section 3 seems non-intuitive, because the method uses section 3.2 first and then it uses section 3.1.\nAdding pseudo-code or algorithm would also improve clarity of the method.\n\npage 8, typo: covolutional\n\n", "summary_of_the_paper": "The paper presents a method to automatically select parameters to share between layers.\nIt proposes to use a shape shifter network to either increase or decrease the number of parameters in the model.\nThe parameters are mapped into parameter groups through a preliminary training step and k-mean cluster the layers.\nLayers in the same group share parameters. It will generate weights by downsampling or upsampling depending on the layer needs.\nThe method is tested in Low Budget and High Budget regimes and on different tasks.\nIt also shows that the method can be used together with distillation and pruning.\n", "summary_of_the_review": "The paper demonstrates SSN and NPAS and presents experimental results on various benchmarks.\nThe text could be improved for clarity and organization.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4cpiyywOSjg", "reviewer": "Reviewer_p84E", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### **Strong points:**\n1. This work is well-motivated. Parameter sharing is one of the most promising directions to reduce training and inference costs, but not addressed in AutoML community. So, the aim of the work is timely.\n2. The proposed method is effective and efficient. Also, the authors validate the proposed method across not only multiple datasets but also multiple tasks. Moreover, the authors show that other techniques to reduce the inference cost are orthogonally applicable.\n\n#### **Weak points:**\n1. The proposed method contains many hyperparameters to be tuned, such as the window size, embedding dimension, and # partition groups. How sensitive is the performance of the proposed method to them? Also, do all tasks and datasets share the same set of hyperparameters?\n2. The proposed method (LB-SSN) seems to reduce only the number of parameters. However, in the practitioner’s point of view, reducing inference time may be more critical. Are other pruning methods also applicable to the network configuration obtained by LB-SSN? It would be great to add the result to Table 4.\n\n#### **Detailed comments:**\nIt is not easy to understand the proposed method with Figure 2 at a glance. Simplifying Figure 2 or replacing it with an abstractive algorithm description may be better.", "summary_of_the_paper": "This work investigates how to automate allocating parameters within a fixed budget to each layer and generating its weights with assigned parameters, which is a generalized version of parameter sharing. The proposed method is validated with leading performances and efficiencies on multiple datasets across multiple tasks.", "summary_of_the_review": "This work is well-motivated, and the experimental results support claims well. However, major concerns lie in the practices; the proposed method additionally introduces hyperparameters and cannot help practitioners reduce inference time.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were mostly concerned about the practical impact/implications of the proposed methods. There was a long discussion across multiple threads of the benefits of the approach proposed in CNNs vs larger language models, dissecting the benefits in terms of training time (as opposed to memory or FLOPs, which may have a non-linear impact on running time). Overall, the authors did a good job of putting their contribution into context and addressing the reviewer concerns.
Decision: Accept (Poster)