LAuReL: Learned Augmented Residual Layer
Authors: Gaurav Menghani
Ravi Kumar
Sanjiv Kumar
Venue: ICML 2025 poster (2025)
Keywords: residual connection
skip connection
residual layer
resnet
residual
Primary area: deep_learning->everything_else
TL;DR: We introduce LAuReL, which is a novel replacement for the residual / skip connection. LAuReL significantly improves performance of vision and language models  while adding as few as 0.01% more parameters and negligible extra latency.

Abstract: One of the core pillars of efficient deep learning methods are architectural improvements, such as residual/skip connections, which have led to significantly better model convergence and quality. Since their introduction, residual connections have become ubiquitous not only in convolutional neural networks but also in transformer-based architectures, the backbone of LLMs.

In this paper, we introduce the Learned Augmented Residual Layer (LAuReL) --- a novel generalization of the canonical residual connection --- designed to serve as an in-situ replacement while outperforming it in both model quality and footprint metrics.  Our experiments show that LAuReL can enhance quality for both vision and language models while adding fewer parameters and incurring less latency and memory overhead than naively increasing parameter count.

For example, on the ImageNet-1K task, LAuReL achieves the same model quality improvements as naively adding an extra layer while using $2.6 \times$ fewer parameters. Similarly, when pre-training 1B and 4B parameter LLMs, LAuReL improves performance on a variety of challenging downstream evaluation tasks by 2.54\% to 20.05\%, while adding only 0.012\% and 0.1\% additional parameters, respectively.

LAUREL: Learned Augmented Residual Layer
Gaurav Menghani'
Ravi Kumar'
Sanjiv Kumar"
Abstract
size, peak memory consumption, etc. dictate if it can be experimented with and/or deployed in real-world settings. A large and slow model can be impractical to train and use, making it unsuitable for applications that require fast responses, no matter how well it performs on benchmarks.
One of the core pillars of efficient deep Iearn- ing methods are architectural improvements, such as residual/skip connections, which have led to significantly better model convergence and qual- ity. Since their introduction, residual connections have become ubiquitous not only in convolutional neural networks but also chitectures, the backbone of LLMs.
LLMs such as Gemini 1.5 Flash (Gemini-Tean et al., 2024), DeepSeek V3 (DeepSeek-A.L. al., 2024) have been explic- itly designed with these efficiencies in mind and consistently outperform much larger models that preceded them. Conse- quently, improving the Pareto-frontier of model quality and model footprint, via efficient learning methods has been an area of active research in the past few years. Areas of in- terests span from algorithmic techniques (Menghani, 2023) and efficient hardware (Sze al., 2017) best practices around model efficiency (Dehghani al., 2022).
measformer-bbased ar-
In this paper, we introduce the Learned Aug- mented Residual Layer (LAUREL) novel gen- eralization the canonical residual connection- designed to serve in-situ replacement while outperforming it in both model quality and foot- print metrics. Our experiments show that LAU- REL can enhance quality for both vision and lan- guage models while adding fewer parameters and incurring less latency and memory overhead than naively increasing parameter count.
One of the core pillars of efficient deep learning methods are architectural improvements such as the residual/skip connection, which had led to significantly better model con- vergence and quality (He et al.). The residual connection has become ubiquitous not only in convolutional neural net- works but also in transformer- -based architectures (Vaswani et al., 2017), which the backbone of today's LLMs.
For example, on the ImageNet- IK task, LAU- REL achieves the same model quality improve- ments as naively adding an extra layer while us- ing 2.6x fewer parameters. Similarly, when pre- training 1B and 4B parameter LLMs, LAUREL improves performance on variety of challenging downstream evaluation tasks by 2.54% to 20.05%, while adding only 0.012% and 0.1 % additional parameters, respectively.
In this paper we introduce learned augmented residual layer, LAUREL, which generalizes the canonical residual connec- tion. Recall that deep-learning models with residual con- nections have 'block' structure, with many blocks chained together between the input and final output; these could be convolution/identiiy blocks within ResNet, a transformer block in transformer encoder/decoder, etc. Within block, typical residual connection is given by:
1. Introduction
Zz+1 f(x,) + +
(1)
Model efficiency is of critical importance in the age of extremely large language and vision models. Even if a given model has impressive quality, its footprint metrics such as train-time compute, inference latency, resident memory
Here, f'()) can be any non-linear function such as attention, MLP, multiple non-linear layers, etc., I; is the input to the said non-linear function, and 44+ the combined output of the non-linear function and the residual component (Fig- ure 1). For simplicity, we ignore pre- preprrcessing functions such as layer norm, which can be folded into f(-).
'Google Research, Mountain View, gmenghani @merghann&@googlee com, ravi. kk5kk33mmmml.. con. "Google Research, New York, NY. sanjivkégoogle, com. Correspondence to:
CA.
@google
Gaurav Menghani
brtccnghann@gogglc.co
2. Learned Augmented Residual Layer
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this section we describe the main idea behind LAUREL. In its most general form, we reformulate the residual con-

---

LAUREL: Learned Augmented Residual Layer
Block
Block
f(x) glx, ... x3
f(x)
Stroom
LAuReL
Residual connection
f(x)
f(x)
Nonlinear function (Attention. MLP, etc.)
Nonlinear function (Attention. MLP, etc.)
x
Activations from previous blocks (x, X Xx)
Figure 1. A standard residual connection. We assume the model to be divided into logical 'blocks', which is true for most modern architectures including transformers. The residual connection combines the output of a non-linear function f and the input to the said non-linear function. Here, f can be attention, MLP, any other combination of non-lincar layers.
Figure 2. An illustration of the LAUREL framework; see (2). LAUREL can used to replace the regular residual connection Figure 1. Again, f can be any non- non-linear function such as attention, MLPs, groups of multiple non-linear layers.
nection to be the following:
Notice that this version assigns learnable weights to the fæx) and from (1). In practice, we found that cannot let and 3 grow unbounded, and using normalization function such as softmax or s sigmo helps. Clearly, this version will add only two new parameters per LAUREL layer. If necessary, we can replace these two parameters by single learnable parameter and use function such as gmo inniddd id define a, B in terms of this single parameter.
T++1 = a f(x:)) + g( 21-....,...
Here, a learned sealar parameter, and g() a learned linear function with F4, 24-11... . To as inputs, where zz is the output of the j-th residual connection.
The main intuition is that one can learn richer set of (linear) functions than just using as the residual component. One motivation behind seeking these richer linear functions the concept of "residual stream" (Elhage et al., 2021), where the residual connection is considered to part of a stream of information that passes through each layer without being exposed any non-linearities This allows the learning process to focus on the non-linear components better.
This variant can be useful for learning the relative impor- tance of the non-linear component (f(z:) and the residual input (zz). In the earlier layers the former might be more important, while the later layers, the latter could useful for mitigating problems such as vanishing gradients. This variant can help by adaptively learning these weights.
Each layer/operation can read from, and subsequently write to this residual stream. Since the residual connection has been shown to be important for model quality and conver- gence, we designed LAUREL to operate on this residual stream in learned fashion, while being light- in terms of the model size and latency changes.
2.2. Low-Rank Version (LAUREL-LR)
Here, we fix L, and g(x;) Wz, in (2) to obtain:
F++1 = [(x)) + Wxy.
As written, W is learnable D D matrix, where D is the model dimension for transformer- bassd models, more generally it is the last dimension of x;, Hence, the W matrix will add D new parameters (per LAUREL layer).
In this paper we study three specific versions of the LAU- REL framework; although as described in (2), the frame- work can generalized beyond these versions.
In practice, to reduce the number of new parameters added to the model and to help with convergence, we consider a low rank version of W. In particular, let W A x B + where and B" are D matrices and < D. Thus, can rewrite (2) as:
2.1. Residual Weights Version (LAUREL-RW)
In this version, we keep a learnable and set g(vi,..., 3r, Therefore, (2) can be rewritten as:
x++ = (x.)) + 3x,
F++1 f(x;) + BAr, + Z4.
(3)
2

---

LAUREL: Learned Augmented Residual Layer
Here, both A and B matrices are learnable. The number of new parameters is 2r D, per LAUREL. layer.
with LAUREL-RW follows:
ne(or--)
This variant helps with allocating learning capacity for the linear part of the network (the residual input, i.e., æ4 in
#++1 = af())) +
+
Σ 1=0
ni
he
Figure 2), such that the main network can use its capacity towards learning better nonlinear functions (f(x))), while LAUREL contributes the linear components (xy + BAr:) to the residual stream.
When using low-rank product for hi, LAUREL-RW+LR+PA variant follows:
create the
k-1 Enns j=0
Si+1 = af(())
3i +
ni D
A (.gBastti)))
2.3. Previous Activations Version (LAUREL-PA)
(6)
This is similar LAUREL-LR, except that we use activa- tions from the previous blocks. In particular, set
Yet another variant while slightly relaxing our above formu- lations would be to treat and |3 as vectors of length D. Here, we would learn fine-grained per-dimension weights when mixing Iy and f(x,) at the cost of 2D parameters per LAUREL-LR layer, instead of two per parameters per layer.
k
g(.....,,,, Ro) = Zt +
Ba
ha hx(
==
To summarize, LAUREL is inherently flexible and provides many possible cheap learnable augmentations on top of the vanilla residual connection. In the following section we demonstrate that these combinations are efficient and effective at improving the model quality of common vision and language models, while having minimal impact the number of parameters and latency.
where "........ - are learned scalar parameters and h;
is another linear function.' 1 This allows us to rewrite (2) as:
Zi+1 = f(x;) + 24 +
Σ 3=0
nhs ha(zi-))-
(4)
In practice, we replace h; by a low-rank product similar to the LAUREL-LR version, but using the identity function is also an option. When using a rank r product for h; the number of new parameters per REU layer is 2r + k, where & is the number of previous activations used.
3. Experiments
We experiment with LAUREL in two domains, namely, vision and language. For the first case, our goal is to improve the image classification accuracy of the ResNet-50 model the ImageNet-IK IK dataset (Deng et al., 2009). For the second case our goal is to improve the performance of two different large language models (L.LMs) of size 1B and 4B parameters respectively, evaluated after the pre-training stage, on common benchmarks.
We can consider this variant to be hybrid of the LAUREL- RW and LAUREL-LR variants, where multiple previous activations are used in weighted manner that is learned. This allows layers accelerated access to previous activations, along with learning their relative importance.
The underlying motivation behind these experiments not necessarily to improve the SOTA results, but to show how LAUREL can be easily integrated on top of common model architectures with residual/skip connections in order to achieve a better model quality and footprint trade off.
2.4. Other Derived Variants
All three proposed LAUREL versions are a combination of scalar and/or low-rank products on top of the vanilla residual connection in (1). This makes it especially fight-weight in terms of its impact model size and latency. We discuss the efficiency of the variants in more detail in Section 4.
3.1. ResNet-50 on ImageNet-1K
That being said, LAUREL is generic enough to allow com- binations of the above variants, as well as new variants. For instance, one straight- forward combination is LAU- REL-RW+LR, where the residual weights from LAUREL- RW can be applied along with the LAUREL-LR variant to rewrite (3) as follows:
In this setup we train a standard ResNet-50 model on the ImageNet 1K dataset (Deng et al., 2009) using 16 Google Cloud TPUvSe chips over one epoch with data- augmentation turned on. In order to obtain a strong baseline, we fine-tuned the model learning rate schedule and picked a schedule that maximized the average of the best accuracy @ values over 5 trials (which we simply refer to as accuracy in this subsection). The baseline model that we obtained achieves an accuracy of 74.95 + 0.016%.
Fi+1 = az((z)) + + ((BAx; + x:).
(5)
Similarly, LAUREL-PA as defined in (4) can be combined
In addition, we also find that if we simply add another layer to the ResNet-50 model (i.e., naive scaling), we can
For simplicity, we fix a 1.
3

---

LAUREL: Learned Augmented Residual Layer
4B allow lenient, yet modest increase in parameters (~ 0.1% extra parameters), and latency (1% 1 increase).
Table 1. Applying LAUREL on a ResNet-50 trained the Ima- geNet IK classification dataset. LAUREL-RW provides a signifi- cant boost with negligible extra parameters. LAUREL-RWWAR RR and LA meet and beat the naive scaling base- line while using 2.6x and 1.82x fewer parameters. Results that provide statistically significant boost over the bascline are high- lighted in green and bold.
Given the scale of LLMs today, both the budgets would be considered negligible. For instance, a 0.1% increase in parameters for a 4B model will only correspond to 4M more parameters. As demonstrated in the CesNet-500Imagehhh experiments (Section 3.1), LAUREL outperforms naive seal- ing; see Section 4.5 for a more detailed comparison.
MODEL
AVG. BEST ACCURACY @1 (%), 5 TRIALS
PARAMS ADDED VS BASELINE (%)
Our objective behind testing LAUREL in these conditions was to demonstrate its efficacy and ensure that it scales well across different LLM setups in the wild.
BASELINE
74.95 + 0.01 75.20 + 0.12
0.00
BASELINE + LAYER
4.37
(NAIVE SCALING)
3.2.1. LLM-1B: VERY LOW ADDITIONAL FOOTPRINT
LAUREL-RW
75.10 0.10
0.003
For our first baseline, we chose a 1B parameter decoder-only transformer- based model. We pre-trained both the baseline, and our experiment with LAUREL, from scratch; use the LAUREL-RW and LAUREL-LR versions (with 4), Both the models were trained using 256 Google Cloud TPU v5e chips for approximately two weeks each, using a pre- training mixture consisting of only text data that included webpages, books, code, and translations.
LAUREL-RW+LR
75.20 0.07
1.68
MAUREL-RW+LRHA
75.25 1 0.09
2.40
increase the model's accuracy by 0.25% to reach 75. 75., while adding 4.37% new parameters. With that in context, applying LAUREL the model improves it (Table l).
If we only use the LAUREL-RW version, we get an im- provement of 0. 0.15% on average with only 0.003% extra parameters, which is essentially negligible. When we try the LAUREL-AW+LR version from (5) with r = 16, we achieve 75.20% accuracy while adding only 1.68% new parameters; this matches the performance of the baseline with extra layer, while using 2.6x fewer extra parame- ters. Additionally, when we use the combined LAUREL- RW+LR+PA version from (6), we improve the accuracy to 75.25% while still using 1.82x fewer extra parameters than the baseline with one extra layer, demonstrating that LAUREL is superior to naively scaling the model. Notably, despite substantial changes to the residual connection, we did not find any training instabilities with LAUREL.
It is worth noting that the combined LAUREL-AW-LR variant adds only 0.012% more parameters as compared to the baseline model. Since we chose 4, the number of parameters added by LAUREL-LR is 8 D and the number of parameters added by LA is 2N,, for a total of 2N (4D+1) additional parameters. Typically N E [10, 100] and D E [500, 5000). For the sake of illustration, assuming N = 20 and D 1000 and using LAUREL-RW+LR leads to 160, 040 extra parameters in a 1B parameter model. Thus, the number of new parameters is dwarfed by that of the orig- inal model. Furthermore, the additional latency introduced by LAUREL-RW+LR was within the noise range.
We evaluated both the pre-trained baseline and LAUREL models on a host of common L.LM tasks such as Q&A, NLU, Math, Code, etc; see Table 2 for the results. The task type and individual tasks are listed in the first and second columns respectively, and a higher score is better for all the tasks. LAUREL outperforms the baselines on all tasks except on the MBPP dataset where it was neutral. To reiterate, these improvements were achieved with only 0.012% extra parameters and nearly no increase in latency.
3.2. Large-Scale LLM Pre-training
In this setup, our goal was test the performance of LAU- REL when applied on top of strong LLMs. During the course of our work we evaluated LAUREL on two separate LLMs, which we pre-trained from scratch. The first LLM (LLM-1BB is a 1B parameter model pre-trained with a data- mixture consisting of only text tokenss. The second LLM (LLM-4B) 4B parameter model that was pre-trained with a multi-modal and multi-lingua data mixture. Both LLMs were trained with ~ 0.5T tokens.
3.2.2. LLM-4B: Low ADDITIONAL FOOTPRINT
In this second setting, we experimented with a 4B parameter decoder-only model with a similar token budget, but trained on multimodal and multilingual corpus of tokens.
What we varied across the two LLMs was to allow for different budgets for increasing model footprint (parameters, latency, memory, etc.) when applying LAUREL. For LLM- IB we allow a very small increase in these metrics (~ 0.01% extra parameters, and nearly no latency increase). For L.MM-
To compensate for a 4x larger model, and also the fact that the dataset and the evaluation tasks are harder, we allowing
bigger footprint budget for LAUREL; 0.1% extra pa-
4

---

LAUREL: Learned Augmented Residual Layer
Table 2. Evaluation results on I.LM-BB described in Section
Table 3. Evaluation results L.IM-4B as deseribed in Section
3.2.1; a IB parameter decoder-only LLM pre-trained from scratch with (a) once with the baseline model architecture, and (b) once using LAUREL. on top. We evaluated both the models number of common evaluation benchmarks (higher is better for all task type and task combinations listed below). LAUREL variant outper- forms the bascline on all but one dataset while adding only 0.012% extra parameters. Results that provide 2 2% relative improvement over the baseline are highlighted in green and bold.
3.2.2; a 4B parameter decoder-only LLM pre- pre-trained from scratch with (a) once with the baseline model architecture, and (b) once using LAUREL on top. We evaluated both the models on number of common evaluation benchmarks (higher is better for all task type and task combinations listed below). LAUREL variant out- performs the bascline on all but two tasks, while adding only ~ 0.1 extra parameters. Results that provide statistically significant improvement over the baseline are highlighted in green and bold.
TASK TYPE
TASK
BASELINE
LAUREL
TASK TYPE
TASK
BASELINE
LAUREL
MATH
MATH
3.54
3.70
MATH
MATH
14.70
15,30
(HENDRYCKS AL., 20218)
(+4.51 %)
(HENDRVOKS 2021n)
(+4.08 (08%)
GSM8K-CoT -CT
8.34
8.79
MGSM
20.0
23.09
(COBBE ET AL., 20211
(+5.39 %)
(SHI ET AL., 2023)
(+15.45 %
GENERAL
MMLU
25.72
25.89
GENERAL REASONING
MMLU (HENDRYCKS ET 2021A)
49.85
51,12
REASONING
(HENDRYCKS AL., 2021A)
(+0.06%)
(2.54%) %
Q&A
BoOLQ
58.07
65.66
READING
BELEBELE
58.40
63.23
(CLARK ET AL., 2019)
(+13.07 %)
COMPRE-
(BANDARKAR ET AL., 2024)
(+8.27%
TYDI QA (GOLDP)
67.98
72.58
HENSION
BOOKQA (MIRAYLOY ET At., 2018)
50.36
60.46
(+6.76 %)
(+20.05%
(CEARK AL., 2020)
TRANSLATION
WMT23
68.32
68.24
SENTENCE COMPLETION
HELLASWAG
64.84
65.06
(KocMI At., 2023)
(-0.1 %
(ZHIEERS BT AL., 2019)
(+0.03%)
MULTIMODAL
MMMU
32.22
36.33
CODE
HUMANEVAL
18.29
18.90
(YUE AL., 2024)
(+ ++12.75% %) )
(CHEN ML., MBPP
(3.33 %)
COCO-CAPTION
95.69
99.15
27.00
27.00
(1.N BT AL., 2014)
(+3.61 %
(AUSTIN ET AI., 2021)
DOCVQA (MATHEW ET AL., 2021)
68.28
68.34
-SM8K-PAL
10.31
11.37
(+0.08%)
(CosBe 20211
(10.28% )
TEXTVQA
60.07
62.64
(SINCH ET AL., 2019)
(+4.27%%
rameters and 1% extra latency. Note that this is still negligible increase in model parameters and latency.
stronger than LLM-IB on both the common evals; 4.16x better on the Math task, and 1.93x better on the MMLU task. We attribute this to LLM-4B being 4x larger, and having more sophisticated pre- pre-training mixture. LAUREL using the LAUREL-AW+LR variant improves on both Math and MMLU, which is pleasantly surprising given how powerful is LLM-4B. For other tasks LAUREL improves significantly over the baseline as well, except for WMT23 and Doc, where it was neutral.
To match these budgets, set r 64, which provides fa- vorable trade-off of providing more capacity to the low-rank matrices as specified in the LAUREL-RWW+R formulation in (6), while also meeting the above-mentioned budgets.
In this setting, both the baseline and the LAUREL experi- ment were trained using 1024 Google Cloud TPU v4 chips for slightly more than two days each. See Table for the evaluation results of both the baseline model and the model with LAUREL. The task type and individual tasks are listed in the first and second columns respectively, and a higher score is better for all the tasks. LLM-4B had a sophisticated suite of evaluation tasks, including math, general reasoning, reading comprehension. translation, and multimodal tasks. All of the listed evaluation tasks are used by leading LLMs to evaluate model quality, further solidifying our confidence in the model evaluation setup.
In terms of costs, the model adds ~ 0.1% more parameters. This is still a very reasonable since it means additional 4M parameters on top of a 4B parameter model. In terms of latency, we tested on both server-side (Google CloudTPUU for cloud serving, and a leading smartphone for on-device inference. In both the benchmarks, we measure nearly 1-2% increase in latency for prefill and generation. There was no human perceptible difference in terms of time-to-firstttkken
We did not try the LAUREL-PA version for the above LLM experiments, as the LLM training was expensive. How- ever, we expect the LAUREL-PA results from the ResNet experiments to also hold this case as well.
To start off, two of the LLM-4B evaluation tasks, Math, and MMLU were common with LLM- IB (refer to Table 2 for LLM-1B results). It can be seen that LLM-4B is much
5

---

LAUREL: Learned Augmented Residual Layer
3.3. LAUREL-LR: Rank Accuracy
Effect rank (r) model accuracy.
76.6
We note that for the LAUREL-LR version on the ResNet- 50/lmageNet combination, there is a pattern in terms of the best accuracy achieved with different values of r. In
75.4
Fvg Weetlleet--ooo Rocuroysst
75.3
the combined LAUREL-RW+LL version, we experimented with different values of r, and computed the average of the best accuracy@) achieved over 5 trials; see Figure 3. From Table I, with the LAUREL-RW version alone we already achieve an average best accuracy @ of 75.10%, therefore for the combined to see the accuracy exceeding that.
75.2%
25.2%
76.12%
1:16
F:32
75.14%
76
25.1
78.0
74.9
BEUREL-RW+LL version we would like
74.0
Bank 01
We observe that when r is small (r E (4, 84), there is not a significant improvement over the baseline LAUREL-RW experiment. This could be because a very small r acts as an information bottleneck the low-rank product in (3). As r increases, the accuracy reaches the maximum for r E {16, 32); beyond this, the accuracy seems to drop though still higher than the LAUREL-RW baseline.
Figure 3. Average best accuracy @ vs the rank (r) in the LAC- REL-RRWLR variant.
versus quality trade-offs.
To achieve these favorable trade-offs, the above-mentioned models incorporate training techniques and architectural changes that can help improve quality while keeping training and inference costs low. Therefore, any proposed model ef- ficiency technique including LAUREL should demonstrate not just improvement in quality, but also training and infer- ence efficiency.
We believe this unimodal phenomenon could be due to the number parameters added to the model (which increases linearly in r), since this would also require appropriate tuning of hyperparameters such as the learning rate as well as the regularization penalty.
Another possible cause could be how the low-rank matri- ces A and B are initialized. For A E RDx,, we used the Xavier initialization for the ResNet experiment and a col- umn orthogonal initialization for the LLM experiments while was always initialized to zero; this is similar to the scheme used in Hu al. (2022). We found that initializa- tion made significant difference in the performance of the LAUREL-LR variant, and posit that further work study- ing and improving the initialization scheme of the low-rank matrices could lead to better performance of the variant.
4.1. Efficiency Metries
We now define some of the metries on which can mea- model training and inference efficiency, before describ- ing how LAUREL scores on them.
4.1.1. 1. NUMBER OF PARAMETERS
This is the most common metric designed to capture the cost of training and serving the model. A larger number of parameters implies larger forward and backward pass costs.
In terms of the tuning required, since r < D, and if D = 512, 768, 1024, as in typical LLMs, this leaves small range of discrete values for r (unlike real-valued hyperparameters such learning rate, weight decay, etc). In our experience E {32, 48, 64} work well for LLMs.
4.1.2. LATENCY (TRAINING AND INFERENCE)
For training, key metric to consider would be the number of steps taken per second. Similarly, another key metric would be the inference latency when deploying the model. For LLMs this can be broken down into the latency met- rics in the warm-up stage (c.g., time-to-fistttokenn which can be further refined to prefill latency), and the genera- tion stage (output tokens/second). For LLMs and other interactive models, both the time-to-firt token and output tokens/second are important for good user experience.
4. Efficiency
Large models (LLMs and beyond) have been scaled both in the number of parameters as well as the number of tokens to achieve better model quality that scaling laws (Hoffmann et al., 2022) promise. However, this is directly in conflict with keeping training and inference costs reasonably low (we describe these costs shortly). These competing forces have resulted in the emergence of models such as Gem- ini Flash and Nano (Gemini- Team et al., 2024), DeepSeek (DeepSeek-A..) A.I.e et al., 2024), etc., which have attractive cost We use A.,, = 1/VrD if mod r = and 0 otherwisc.
4. 1.1.3. PEAK MEMORY
Peak memory used during training is another key metric that tracked to ensure that accelerators can accommodate the model graph and have enough free memory available
6

---

LAUREL: Learned Augmented Residual Layer
for forward and backward passes. Inefficiencies can be compensated by strategies like rematerialization (Kumar et al., 2019), which can help reduce peak memory usage but need recomputing some activations. Similarly, inference- time peak memory usage is also a key metric to track.
Table 5. Performance of LAUREL variants small L.LM pre- training task. LAUREL experiments are based on top of BASE- LINE-24. LAUREL variants achieve better test loss than BASE- LINE-28, while having fewer parameters, requiring lesser memory, and being faster in average step time.
VARIANT
PARAMS TEST
PEAK AVG. STEP
4.2. Efficiency Analysis
(M) LOSS MEMORY
TIME
(GB)
(SEC)
LAUREL variants are designed with the above efficiency metrics in mind, and in this section we study the perfor- mance of the variants on these metrics.
BASELINE- 24
157.20 3.0159
11.65
0.095
BASELINE-28
179.23 2.9963
13.23
0.105
LAUREL VARIANTS (24 LAYERS)
LAUREL-RW LAUREL-LR LAUREL-PA
157.20 2.9557
11.93
0.095
Table 4. Analysis of extra parameters, memory, and latency in- curred for each LAUREL variant, per instantiation, except the LAUREL-PA case where at any time no more than & previous activations are in-memory, incurring total (k:D) extra memory cost. Also note that the lateney cost of LAUREL-LR can be tighter than (r D²), depending upon which matrix multiplication algo- rithm used. For simplicity, the latency bounds drop the batch and sequence dimensions.
158.40 2.9624
12.29
0.098
157.22 2.9512
12.55
0.100
LAUREL-RW+LR
158.40
2.9531
12.57
0.099
PAUREL-AW+LR++A PA 160.83
2.9499
12.90
0.104
introduces additional latency of O(kD).
4.3. Ablation Study of LAUREL Variants
LAUREL
PARAMS
MEMORY
LATENCY
VARIANT
To supplement the efficiency analysis in Section 4.2, we provide an ablation study of LAUREL variants on a small LLM pre-training baseline. We used the C4 corpus (Raffel et al., 2020) with ~ 10B tokens, and a 4 x 4 Google Cloud TPU v6e (Trillium) topology² for compute.
LAUREL-RW O
0(1)
O(1)
LAUREL-LR 2rD
@(2rD)
O(rD2)
LAUREL-PA
&
(kD)
O(kD)
LAUREL-LR+PA 2rkD @(2rkD + k) D + +D)
In order to simplify the comparison across many ablations (and also to avoid the noise in downstream evaluations at the 10B tokens scale), we report model performance using the test loss, a reasonable proxy for downstream model quality. For each model we report the number of parameters, test loss, peak memory reported by profiling tools, and the average step time. The last two metrics are proxies for the theoretical memory and latency bounds respectively, as mentioned Section 4.2. Lower is better for all metrics.
Table 4 lists the costs associated for each LAUREL layer. We assume that we are working with a vector of size D (the last dimension of the input). Note that the listed costs LeuRLA instantiation. Hence, if there are many transformer layers and if a particular LAUREL variant is used in each layer, then the relevant costs accumulate across the layers. We now examine the costs of the variants.
We trained our main baseline with 24 layers (BASELINE-224) and 157.2M parameters, along with larger baseline with 28 layers (BASELINE-28) and 179.2M parameters. We ran all the LAUREL variants and two combinations (RW+LR, RW+LR+PA) on top of BASELINE-22. For the LAUREL. LR variants and its combinations, we picked r = 32. Sim- ilarly for LAUREL-PA variant and its combinations, we chose k = 3. Table 5 shows the results.
LAUREL-RW is the cheapest, with one or two scalars as parameters, a constant memory cost (the constant depend- ing on the data type: fp32, fp1 / bf 6, uni sittt.. // i int8, etc.), and constant latency cost since the scalar multiplica- tion might be optimized by the compiler.
The LAUREL-LR variant has 2r parameters per instan- tiation (rD for each of the two rank-r matrices, A and B), and uses (rD) memory. The latency is upper bounded by O(r since the input a vector of dimension D (ignor- ing the batch and sequence dimensions), and there two matrix multiplication steps.
As seen in the results, all LAUREL variants have lower test loss than the BASEI.NEE-24, while having a negligible impact on additional parameters, peak memory, or average step time. In fact, LAUREL variants perform better than even BASELINE-28 in terms of the test loss while using much fewer parameters, lower peak memory, and lower average step time.
Finally, the LAUREL-PA variant uses k previous activa- tions. Assuming h; is the identity function, it uses k extra parameters, and uses (k:D) extra memory for the previous activations. Since requires extra vector additions, this
"We expect similar results with a comparable GPU setup.
7

---

LAUREL: Learned Augmented Residual Layer
4.4. Practical Recommendations
each, and are referred to BASELINE-40 and LAUREL-40 respectively.
Given the experiments and the analysis of individual vari- ants and their combinations, LAUREL-RW is clearly the first candidate to try. LAUREL-AW-LR offers further im- provements to model quality, and seems to provide the best trade-off in terms of quality improvements and the addi- tional overhead. We validated this in the ResNet (Section 3.1) and LLM experiments (Sections 3.2.1, 3.2.2, 4.3).
The naive scaling baseline had 41 layers, and is referred to as BASELINE-41 henceforth. In Table 6, we present the respective number of parameters and average step times of the three models when training (forward + backward pass), A lower average step time is better. For BASELINE-41 and LAUREL, we report the delta in number of parameters and average step time when compared to the original baseline.
The LAUREL-RW+LRAHE variant also leads to improve- ments over LAAUREL-AWALR, The main caveat is the ad- ditional ((+D) memory, which might need to be moni- tored. If it is concern, it can be mitigated to some extent by choosing a small value of k or by changing the model sharding/re-mattee materialization schemes.
Table 6. Comparison between BASELINE-40 (40 layers), BASE- LINE-41 (41 layers), and LAUREL in terms of parameters and average step time (forward and backward pass).
Given the above tradeoffs in loss, memory, step time, etc. we recommend trying the LAUREL variants in the following order: RW- + LR - RW+LR / PA + RW+LR+PA.
MODEL
PARAMS
AvG. STEP TIME (SEC)
(B)
BASELINE-40
4.400
1.65
4.5. Comparison with Naive Model Scaling
BASELINE-41
4.560 (+3.63%)
1.68 (+1.81%)
LAUREL-40
4.404 (+0.1%)
1.69 (+2.42%)
One of the advantages of LAUREL we would like to high- light is that it is competitive against naive scaling. Con- cretely, given a modest additional budget for parameters, latency, and memory, allocating that budget to LAUREL variants is likely to produce larger gains than using that bud- get in naive scaling methods such as additional layers. We first observed this in the ResNet experiments (Section 3.1), where LAUREL variants match the performance of naive scaling while adding 2.6x fewer parameters, thus showing the Pareto-efficiency of LAUREL (versus naive scaling).
Note LAUREL-40 adds only +0.1% 1% parameters and incurs step time penalty of 2.42%. In terms of extra parameters added, LAUREL-40 adds ~ 36x fewer parameters than the extra parameters added by BASELINE-41. The latency of LAUREL-40 is slightly higher than BASELINE-41 since each layer incurs minor additional overhead.
In the LLM experiments in Section 3.2, the gains achieved using LAUREL variants are at the cost of ~ 0.01-0.1 more parameters. A naive way to use this 'additional' bud- get would be to increase the model dimension (D) such that we match the extra parameter budget. However, this is may not be feasible due to hardware limitations and memory alignment issues. Another way to use this 'additional' bud- get would be to increase the vocabulary size. However, the gains would be limited since this will include more tokens only from the tail of the distribution and may not contribute much to the model quality. Interestingly, Karpathy (2023) reported that for the NanoGPT model (Karpathy, 2022) pre- training was sped up by ~ 25% when the vocabulary size was made divisible by 64. Therefore naive scaling might also hurt metrics like latency and memory by making the model setup suboptimal on the hardware.
Table shows the downstream quality of the baselines and LAUREL-40 on 10 tasks across Math, General Reasoning, Reading Comprehension, Translation, and Mutimodal do- mains. LAUREL-40 wins on all tasks, except WMT23 and DoA where it matches the baselines. It outperforms not just BASELINE-40, but also BASELINE-41, achieving between 2%-21 % improvement over BASELINE-40's per- formance.
We posit that naive scaling by adding an extra layer two in a very deep network does not always lead to correspond- ing improvement in performance out of the box. For such networks, we might require hyperparameter tuning (learn- ing rate, weight decay) to counter overfitting, or training on more tokens to realize the expected theoretical model performance predicted by the scaling laws.
Additionally, we also conducted a detailed study comparing LAUREL with naive scaling on the same LLM pre-training task LLM-4B as mentioned in Section 3.2.2. The baseline model and the the LAUREL experiments are similar to their counterparts in Section 3.2.2, except a configuration change, which allows adding a layer. Both have 40 layers
It is also well known that with deeper networks, the residual stream starts to play crucial role in model convergence and quality. We suggest that this is the primary reason why LAUREL performs well. It augments the residual stream with learned components, allocating capacity for the network to learn the linear components of the input better.
8

---

LAUREL: Learned Augmented Residual Layer
Table LAURREL-440's comparison with naive scaling. Results that provide statistically significant improvement the RASELINEE:40 are highlighted in green and bold. The percentages in the last row of the LAUREL-40 section indicate the relative improvement over the baseline.
GENERAL REASONING
READING COMPREHENSION
MATH
TRANSLATION
MULTIMODAL
METHOD
MATH
MGSM
MMLU
BELEBELE BOOKQA
WMT23
MMMU
COCO-CAP DOCVAA TEXTVQA
BASELINE-40
14.20
20.29
48.83
57.92
47.11
67.72
33.77
97.29
66.87
60.86
(4,40B PARAMS)
+ 0.88
3.16
+ 0.81
+ 3.42
4.06
0.220
3.11
4.4.1
2.67
+ 2.86
BASELINE-41
14.50
20.29
49.10
59.30
42.77
67.74
35.33
98.50
66.18
60.23
(4.56B PARAMS)
0.9
+ 3.15
0.82
3.34
4.15
+ 0.21
3.12
3.53
2.76
+ 2.87
LAUREL-40
15.11
23.12
50.32
62.65
57.22
67.71
37.57
99.27
66.92
63.15
(4.404B PARAMS)
1.01
3.51
0.82
3.15
3.81
+ 0.19
3.10
5.03
2.65
+ 2.82
(+6.48%)
(+3.05%)
(+8.16%)
(+21.46%)
(-0.02%%
(+11.25%)
(+2.03%)
(+0.07%)
(+3.76%)
5. Related Work
models and it works directly on the model weight matri- ces level by introducing low-rank 'adapter' weights that are learned during the fine-tuning stage, while other model weights are held constant. In contrast, LAUREL works at the residual connection level, which likely spans multiple weight matrices involved in the function f: furthermore, is applied during the pre-training stage.
Residual Stream. DenseNet (Huang et al., 2017) connects every pair of layers in the network and hence in the basic variant of DenseNet, all the activations need to be in memory. This is prohibitively expensive for deep LLMs and other modern transformers. When introducing dense-blocks, all previous activations within the block need to be visible to any given layer within the block; this requires refactoring the model architecture into dense blocks.
AltUp (Baykal et al., 2023) is designed to replicate the qual- ity improvements of a model with large model dimension, without having to pay the additional cost. It operates at the transformer-be block level, constructing parallel 'lightwiight" transformer blocks to approximate the model dimension scaling effect. In contrast, LAUREL does not aim to repli- cate the dimension scaling effect.
On the other hand, LAUREL requires minimal changes. In fact, in LAUREL-PA, which is the most similar to DenseNet, we make three design choices to achieve memory efficiency and performance. First, each layer only looks at the l past activations (k = seems sufficient in our experiments). Sec- ond, we use low-rank linear functions to further reduce mem- ory usage due activations. Third, we use learned scalars (1i, . weigh the previous activations (which we found to be crucial in practice), whereas DenseNet assumes a simple sum of the previous activations.
Interestingly, LAUREL can be applied in conjunction with both LoRA (during fine-tuning) and AltUp (during pre- training and fine-tuning). LAUREL can also be enabled at the same time as parameter-sharing techniques.
6. Conclusion
He et al. (2016) introduce variants of residual connections with different types of "gating', which look similar to the LAUREL-RW variant, except that they use a much larger number of parameters (O(D²) per layer), where LAUREL- two extra parameters per layer. Highway Nets (Srivastava et al., 2015) is similar to LAUREL-RW but they also use 2 + D parameters; furthermore, they incur additional latency due a full-rank matrix multiplication. Residual Gates (Savarese, 2016) also is similar to L-UREL RW, except they use ReLU as the gating function. However, LAUREL is more general formulation.
In this paper we introduce the LAUREL framework, which is novel architectural change and a generalization of the residual/skip connection aimed at improving the model qual- ity without significantly increasing the model size or la- tency. We study three versions (LAURRLL-RW, LAUREL.- LR, LAUREL-PA) that can be miixed-and-mateenn together.
RW uses
Through experiments, we demonstrate the efficacy of re- placing the conventional residual connection with LAUREL on both vision and language tasks, while also providing evidence for its advantages over naive model scaling meth- ods. For future work, we aim to further improve LAUREL by developing new variants with better trade-offs between quality and model footprint.
Architectural Changes. Our work is inspired by recent model architecture improvements such as LoRA (Hu et al., 2022) and AltUp (Baykal et al., 2023) amongst others. LoRA is designed to efficiently fine-tune large pre-trained
9

---

LAUREL: Learned Augmented Residual Layer
Acknowledgments
Dehghani, M., Tay, Y., Arnab, A., Beyer, L., and Vaswani,
A. The efficiency misnomer. In ICLR, 2022.
We thank Cenk Baykal, Erik Vee, Fotis Iliopoulos, Khoa Trinh, and Sushant Sachdeva (in alphabetical order) for many helpful discussions. We would also like thank Andrew Tomkins for suggesting the name for the work.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255, 2009.
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph,
Impact Statement
N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield- Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. A math- ematical framework for transformer circuits. Trans- former Circuits Thread, 2021. https:/transformer hwtpiissppu/2221//aaaeooot/////
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Austin, J.. Odena, A., Nye, M., Bosma, M.. Michalewski, H., Dohan, D., Jiang, E.. Cai, C., Terry, M., Le, Q., and Sutton, C. Program Synthesis with Large Language Models. arXiv,
Gemini- Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, I.., Gulati, A., Tanzer, G., Vincent, D., Pan, 7.,, Wang, S., Mariooryad, S., Ding, Y., Geng, X., Alcober, F., and et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv, 2403.05530, 2024.
2021.
Bandarkar, L., Liang, D., Muller, B., Artetxe, M., Shukla, S. N., Husa, D., Goyal, N., Krishnan, A., Zettlemoyer, L.., and Khabsa, M. The Belebele Benchmark: Par- allel Reading Comprehension Dataset in 122 Language Variants. ACL Antholggy pp. 749-775, 2024.
He, K., Zhang, X., Ren, S., and Sun, J. Deep Residual Learning for Image Recognition. In CVPR, pp. 27-30.
He, K., Zhang. X.,, Ren, S., and Sun, Identity mappings in deep residual networks. In ECCV, pp. 630-645, 2016.
Baykal, C., Cutler, D., Dikkala, N., Ghosh, N., Panigrahy, R., and Wang, X. Alterattnn updates for efficient trans- formers. In NewrlPS, pp. 76718-77777 66718-76736, 2023.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring Massive Multitask Language Understanding. In ICLR, 2021a.
Chen, M. Tworek, J., Jun, H., Yuan. Q., Pinto, H. P. Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., et al. Evaluating Large Language Models Trained on Code. arXiv, 2107.033772 2021.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. In NeurlPS, 2021b.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In NAACL-HLT, pp. 1924-2936. 2019.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskkya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbi, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J. W., and Sifre, L. Training compute-optimal large language models. In NeurlPS, 2022.
Clark, J. H. Choi, E., Collins, M., Garrette, D., K Kwiatkowski, T., Nikolaev, V., and Palomaki, J. TyDi QA: A Benchmark for Information-Secking Question An- swering in Typologically Diverse Languages. TACL, 8: 454-470, 2020.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y.. Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training Verifiers to Solve Math Word Problems. arXiv, 2110.14168 2021.
Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In CVPR, 2017.
DeepSeek-A.L. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Guo, D., Yang. D., Chen, D., Ji, D., Li, E., et al. DeepSeek- V3 Technical Report, arXiv, 19437, 2024.
Karpathy, A. nanoGPT, January 2022. URL https: //github. com hatm/kaappthh//nnoorr [Online; ac- cessed 27. Jan. 2025].
10

---

LAUREL: Learned Augmented Residual Layer
Karpathy, A. Andrej Karpathy X: "The most dramatic optimization nanoGPT' far ("25% speedup) is to simply increase vocab size from 50257 to 50304 (nearest multiple of 64). This calculates added useless dimensions but goes down a different kernel path with much higher occupancy. Careful with your Powers of 2." /x, January 2023. URL hwwps: x com/ http://oooo/aaaaaaaaee wwrpathrraaahaaa /stutus/ 162157835402677337 [Online; accessed 27. Jan. 2025].
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S.,
Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21(1), January 2020.
ISSN 1532-4435.
Savarese, P. H. Learning identity mappings with residual gates, arXiv, 1611.01266 2016.
Shi, F., Suzgun, M., Freitag, M., Wang, X., Srivats, S., Vosoughi, S., Chung, H. W. Tay, Y., Ruder, S., Zhou, D., Das, D., and Wei, J. Language Models Multilingual Chain-of-Thouge Reasoners. In ICLR, 2023.
Kocmi, T., Avramidis, E., Bawden, R., Bojar, O., Dvorkovich, A., Federmann, C., Fishel, M., Freitag, M., Gowda, T., Grundkiewicz, R., Haddow, B., Koehn, P., Marie, B., Monz, C., Morishita, M., Murray, K., Nagata, M., Nakazawa, T., Popel, M., Popović, M., and Shma- tova, M. LLMs are here but not quite there yet. In WMT, Findings, pp. 1 1-42, 2023.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards VQA models that can read. In CVPR, pp. 8309-8318, 201-
Srivastava, R. K., Greff, K., and Schmidhuber, J. Training very deep networks. In NIPS, 2015.
Kumar, R.,. Purohit, M., Svitkina, Z., Vee, E., and Wang. J. Efficient rematerialization for deep networks. In NIPS, 2019.
Sze, V., Chen, Y., Yang, T., and Emer, J. S. Efficient pro- cessing of deep neural networks: A tutorial and survey. Pruc. IEEE, 105(12):2295-233)) 2017.
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., and Dollár, P. Microsoft COCO: Common Objects in Context. In ECCV, pp. 740-755, 2014.
Vaswani, A., Shazeer, N. M., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. NIPS, 2017.
Mathew, M., Karatzas, D., and Jawahar, C. V. DoeVQA: A dataset for VQA on document images. In WACV, pp. 2021.
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y.. Huang, W., Sun, H., Su, Y., and Chen, W. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In CVPR, 2024.
Menghani, Efficient deep learning: A survey on making deep learning models smaller, faster, and better. ACM Computing Surveys, 2023.
Mihaylov, T.. Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, pp. 2381-2391, 2018.
Zellers, R., Holtzman, Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can machine really finish your sentence? In ACL, pp. 1791-4800, 2019.


Reviews:
[{"review_id": "OMoSQkuMQM", "reviewer": "Reviewer_gnJr", "summary": "The paper introduces LAUREL (Learned Augmented Residual Layer), a new generalization of residual connections that can replace standard skip connections in neural networks. LAUREL outperforms traditional residual connections in both model quality and efficiency across vision and language tasks. When tested on ImageNet-1K, LAUREL achieved the same improvements as adding an entire extra layer while using 2.6× fewer parameters. In large language model experiments with 1B and 4B parameter models, LAUREL improved performance on downstream tasks by 2.54% to 20.05% while adding only 0.012% and 0.1% additional parameters respectively.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions.  \n\n#### __Ablation study with practical footprint metrics__\n\nThank you for suggesting this experiment.  For the purpose of comparing different LAuReL variants on the LLM pre-training task, we set up the following baseline. We pre-trained on the C4 corpus with $\\approx$ 10B tokens. We used a $4 \\times 4$ Google Cloud TPU v6e  topology, but we expect similar results with a comparable GPU setup.\n\nIn order to simplify the comparison across many ablations (and also to avoid the noise in downstream evals at 10B token scale), we report model performance using the test loss, which is a good proxy for downstream model quality. \n\nWe train our main baseline with 24 layers and 157.2M params, along with a larger baseline with 28 layers for comparison. We run all the LAuReL variants (RW, LR, PA) on top of the regular baseline with 24 layers. We also run two combinations of variants (RW+LR, RW+LR+PA). We report the number of params, test loss, peak memory as reported by profiling tools, and average step time. Lower is better for all metrics. The table below shows the results; note that all LAuReL experiments use L=24.\n\n| Variant                 | Params(M)    | Test Loss  | Peak Mem(GB)  | Avg. Step(sec) |\n|-------------------------:|:---------:|:---------:|:-----------:|:--------------:|\n| Baseline (L=24)         | 157.20  | 3.0159    | 11.65     | 0.095         |\n| Baseline-Large (L=28)   | 179.23  | 2.9963    | 13.23     | 0.105         |\n| LAuReL-RW                | 157.20 | 2.9557    | 11.93     | 0.095         |\n| LAuReL-LR                 | 158.40  | 2.9624    | 12.29     | 0.098         |\n| LAuReL-PA                | 157.22  | 2.9512    | 12.55     | 0.100         |\n| LAuReL-RW+LR         | 158.40  | 2.9531    | 12.57     | 0.099         |\n| LAuReL-RW+LR+PA | 160.83  | 2.9499    | 12.90     | 0.104         |\n\nAll LAuReL variants perform better than the large baseline in terms of the test loss while using much fewer parameters, lower peak memory, and lower average step time. Given the above tradeoffs in loss, memory, step time, etc. we recommend trying the LAuReL variants in this order: RW $\\rightarrow$ LR $\\rightarrow$ PA / RW+LR $\\rightarrow$ RW+LR+PA.  We will add these numbers in the revision.\n\nWe found that r={32, 48} work well for LAuReL-LR, and k={3, 4} works well for LAuReL-PA. We went with r=32, and k=3 respectively.\n\n#### __Relationship to DenseNet__\nThank you for the reference.  DenseNet connects every pair of layers in the network and hence in the vanilla version, all the activations need to be in memory. This is prohibitively expensive for deep LLMs and other modern transformers. When introducing dense-blocks, all previous activations within the block need to be visible to any given layer within the block; this requires refactoring the model architecture into dense blocks.\n\nOn the other hand, LAuReL requires minimal changes. In fact, in LAuReL-PA, which is the most similar to DenseNet, we make three design choices to achieve memory efficiency and performance. Firstly, each layer only looks at the past $k$ activations. For the above experiments, $k=3$ was sufficient. Secondly, we also propose using low-rank linear functions to further reduce memory usage due to activations. Thirdly, the LAuReL-PA variant uses learned scalars ($\\gamma_{i}$, $\\gamma_{i-1}$, …) to learn the weights of the previous activations (which we found to be crucial), whereas DenseNet assumes a simple sum of the previous activations. \n\nAdditionally, as seen above, LAuReL-RW and -LR variants provide significant improvements over naive-scaling, and can be combined with the -PA method. \n\nWe will include these discussions in the revision.\n\n#### __ViT__\nUnfortunately these experiments are still in-progress at the time of the rebuttal deadline; we will add them as soon as the runs finish.  However, we expect LAuReL to provide improvements on top of ViT baselines, given LAuReL has shown improvements on ResNet as well as three LLM baselines (LLM-1, LLM-2, and the new small LLM baseline mentioned above). Note that the latter three are transformer-networks, very much like ViT.\n\nWe hope we have addressed your concerns and questions and we hope you can reconsider your assessment/score.  Thank you.", "claims_and_evidence": "To support the claim, the authors conduct experiments on ResNet-50 on ImageNet-1K, and LLMs.", "methods_and_evaluation_criteria": "The proposed methods make sense for the problem or application at hand. However, the authors only shows theoretical extra memory, and latency incurred for each LAUREL. Do the authors have practical numbers?", "theoretical_claims": "There is no theoretical claims in the paper.", "experimental_designs_or_analyses": "The experimental designs are reasonable, but why only conduct experiments of ResNet on ImageNet, why not use ViT? ViT is very popular in computer vision.", "relation_to_broader_scientific_literature": "This paper is related to DenseNet, but iit does not discuss DenseNet-like works.", "essential_references_not_discussed": "This paper is related to DenseNet, but iit does not discuss DenseNet-like works.", "other_strengths_and_weaknesses": "Strengths:\n1. Figure 2 clears show the idea of the paper and it is eaay to follow.\n2. Experiments on ResNet and LLMs demonstrate the effectiveness of the proposed paper.\n\nWeaknesses:\n1. Do not report the pratical memory usage and latency compared with other methods? Especially on GPUs.\n2. Lack of detailed ablation study for the design choises. How does different design choises influence pratical memeory usage and latency?\n3. Lack of discussion with previous related methods, like DenseNet. In related work, the authors discuss Architectural Changes, Compression Techniques and Learning Techniques. Acutally, they are not very related to this paper. It is necessay to discuss the realtionship between this work and other DenseNet-like works.", "other_comments_or_suggestions": "Plese see weaknesses.", "questions_for_authors": "Plese see weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VrhtzMVxcC", "reviewer": "Reviewer_jdcS", "summary": "The paper introduces a new method for residual connections with an additional layer called Laurel. Their method involves introducing learnable parameters into the residual stream, which the authors argue might be too restrictive in its original form. The learnable parameters allow the authors to decide how much information might be incorporated from different parts of the residual stream. \n\nThe authors introduce three methods for Laurel. Laurel-RW is the simplest version that only introduces two learnable weights. Laurel-LR introduces a low-rank approximation of the residual stream. Laurel-PA applies the same approach as LR except over activations from prior layers. The authors test variants that apply these methods in tandem. The authors show that their method allow for better scaling, where their methods adds fewer parameters but leads to a larger increase.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions.  \n\n#### __Tuning $r$__\nSince is $r$ the rank of $A$, $B^T$, we expect $r \\ll D$. Indeed, if $D = 512, 768, 1024, …$, this leaves a small range of discrete values for $r$ (unlike hyperparameters such as learning rate, weight decay, that can take continuous values). In our experience $r = {32, 48, 64}$ work well for LLMs.  \n\n#### __LLM Depth / Parameter Scaling with Error Bars:__\nSimilar to the comparison with naive depth-scaling with ResNet (Sec 3.1), we ran scaling experiments with LLM-2. Originally, both baseline/LAuReL had 40 layers. Adding the 41st layer required turning off a minor architectural change (which enforced the number of layers to be divisible by 2). To be fair, we re-ran the baseline and LAuReL.\n\nWe present the number of params and average training step times.\n\n| Model | Params(B)   | Avg. Step (sec)|\n|--------------:|:---------|:---------------|\n| Baseline (40 layers)      | 4.40    | 1.65          |\n| Baseline$^{+1}$  (41 layers)  | 4.56 (+3.63%)   | 1.68  (+1.81%)        |\n| LAuReL        | 4.44 (+0.1%)   | 1.69 (+2.42%)           |\n\nNote LAuReL adds only 0.1% parameters and incurs a step time penalty of 2.42%. On latency, it is slightly above Baseline$^{+1}$ . This is because LAuReL is invoked for each layer, but the per-layer invocation cost is small.\n\nThe table below shows the downstream quality of the baselines and LAuReL.\n\n| Model                                                                    | Math             | MGSM             | MMLU             | Belebele         | BookQA           | WMT23            | MMMU             | Coco-Cap         | DocVQA           | TextVQA          |\n|:------------------------------------------------------------------------------------|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|:-----------------:|\n| Baseline (40 layers) 4.40B Params                                     | 14.20 ±0.88     | 20.29 ±3.16     | 48.83 ±0.81     | 57.92 ±3.42     | 47.11 ±4.06     | 67.72 ±0.20     | 33.77 ±3.11     | 97.29 ±4.41     | 66.87 ±2.67     | 60.86 ±2.86     |\n| Baseline$^{+1}$ (41 layers) 4.56B Params                                   | 14.50 ±0.9      | 20.29 ±3.15     | 49.10 ±0.82     | 59.30 ±3.34     | 42.77 ±4.15     | **67.74** ±0.21     | 35.33 ±3.12     | 98.50 ±3.53     | 66.18 ±2.76     | 60.23 ±2.87     |\n| Laurel (40 layers) 4.44B Params     | **15.11** ±1.01 | **23.12** ±3.51 | **50.32** ±0.82 | **62.65** ±3.15 | **57.22** ±3.81 | 67.71 ±0.19     | **37.57** ±3.10 | **99.27** ±5.03 | **66.92** ±2.65 | **63.15** ±2.82 |\n| _Laurel %Change wrt Baseline_                                                          | _(+6.48%)_       | _(+13.94%)_      | _(+3.05%)_       | _(+8.16%)_       | _(+21.46%)_      | _(-0.02%)_       | _(+11.25%)_      | _(+2.03%)_       | _(+0.07%)_       | _(+3.76%)_       |\n\nLAuReL wins on all tasks except WMT23; it is more parameter-efficient than naive-scaling.\n\n#### __When to use Laurel-PA?__\nLAuReL-PA works well for deep networks where there is a risk of vanishing gradients.  A suitable rematerialization and sharding,  given the extra activations in memory, would help; the new ablation results (see Reviewer gnJr) are positive. We expect LAuReL-PA to do better than works such as DenseNet (Huang et al. '16), where the number of activations in-memory is quadratic.\n\n#### __Comparison with Parameter Sharing__\nTechniques like param-sharing are complementary, since they can be applied in conjunction with LAuReL.\n\n#### __Comparison with Highway Nets/Residual Gates__\nHighway Nets (Srivastava et al., 2015) is similar to LAuReL-RW, except that the Transform Gate requires ($D^2$ (weight matrix) + $D$ (bias)) params, in addition to the latency incurred by a full-rank matmult.  But we use 1-2 scalars per Laurel-RW layer, with no significant latency impact. Similarly, the Residual Gates (Savarese et al., 2017) also is similar to LAuReL-RW, except they use ReLU as the gating function. However, LAuReL is more general including variants like -LR, -PA, which can be combined.\n\n#### __Green Percentages in Table 2 / 3__ \nApologies. Green and bold font indicated statistically significant improvements for ResNet and LLM-2. The percentages for a couple of tasks (TyDiQA, MGSM, etc.) had a typo, but the absolute values were correct. \n\n#### __Intuition for asymmetrical improvement in tasks__\nDeep networks generally do better with reasoning/math/coding. However, residual connections are crucial and LAuReL helps augment them with learned components. So we expect LAuReL to improve such tasks. \n\nHowever, for certain other tasks the network might be bottlenecked on the number of parameters in the MLP layers. Thus it is hard to pinpoint why a particular task improves more than others with LAuReL.\n\nWe hope these address your concerns and you can reconsider the assessment/score - thanks!", "claims_and_evidence": "In general, authors report consistent improvements with Laurel in comparison to naive scaling of their base architectures. But, in general, I have the following concerns:\n\n* In most settings, Laurel seems best with Laurel-RW+LR or Laurel-RW+LR+PA settings. In general, using the low-rank approach seems tricky to me because you now have a new hyperparameter to tune. The authors generally address this with Figure 3 -- but I remain concerned with the trend of selecting the best r. The Figure doesn't show consistent results on when the best r is actually applicable. In general, the results also seem sensitive to r if I read Figure 3 correctly.\n* I found the description in Section 4.4 for LLMs to be vague. For instance, I think the argument with ResNets is convincing where the authors show that the addition of a single layer compared to Laurel saves 2.6x parameters. Could the authors provide the same numbers for language models? If I add an additional layer to language model i.e. their 1B/4B model, how many parameters am I adding? The authors should discuss depth/parameter scaling in LLMs as well. It would be useful to have these upper bound memory costs in the table if possible. \n* Under what setting would I use Laurel-PA? It seems to add additional overhead for using sharding, etc.\n* The results of Laurel in Table 2/3 is really confusing to me. See questions. Overall, could error bars be added to results to make significance more obvious? Beyond this, do authors have any intuition on why certain baselines improve significantly while others stay relatively similar.", "methods_and_evaluation_criteria": "Yes; Authors use traditional computer vision and NLP benchmarks for their evaluation.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "* If the authors' aim is to have a parameter efficient scaling technique, why not compare with techniques that allow for scaling like parameter sharing or the methods that save over transformer blocks? I'm curious to see how much these methods in save in comparison to Laurel.", "relation_to_broader_scientific_literature": "The authors compare against other parameter efficient scaling methods, highlighting that theirs is parameter efficient.", "essential_references_not_discussed": "The authors may want to reference the many modifications that have been introduced to residual connections that overlap with their work in the related work. I think a section referring to HighwayNets or ResidualGates. Many similar ideas have been proposed and Laurel should discuss and cite them. Space is available so I recommend an additional related work subsection.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "didn't find any typos", "questions_for_authors": "* What do green percentages mean in Table 2/3? I found this very confusing since the percentages didn't match improvements and some were highlighted green. This isn't explained in the text as far as I can tell although I may be missing something.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FICGEnE5cu", "reviewer": "Reviewer_qGJS", "summary": "The paper introduces Learned Augmented Residual Layer (LAUREL), a novel enhancement to residual connections in CNNs and Transformers. LAUREL enriches the residual stream by incorporating learned scalar parameters and low-rank transformations, improving efficiency and expressivity.  \n\nKey Contributions:\n1. Three variants (LAUREL-RW, LAUREL-LR, LAUREL-PA) balancing expressivity and efficiency.  \n2. Improves performance in vision (ResNet-50, ImageNet-1K) and language models (1B, 4B LLMs) with minimal parameter, latency, and memory overhead.  \n3. Matches naïve scaling accuracy on ImageNet-1K while using 2.6× fewer parameters.  \n4.  Boosts reasoning, math, reading comprehension, translation, and multimodal tasks with negligible overhead.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind comments and valuable feedback!\n\n#### __Justification into why specific variants perform better under certain conditions__\nDeep networks generally do better with reasoning, math, coding etc. tasks. However, residual connections are crucial for such networks, and LAuReL helps augment these residual connections with learned components. So at a high-level we expect LAuReL to help with reasoning/math/coding tasks. \n\nSpecifically we have the following intuition for why the different variants work:\n\n- LAuReL-RW: Helps with learning the importance of the residual input ($x_i$). This might not be as useful in the earlier layers, but would be important in other later layers to tackle the vanishing gradient problem using a higher learned weight for the residual input.\n\n- LAuReL-LR: Helps with allocating learning capacity for the linear part of the network (the residual input, i.e., $x_i$ in Figure 2), such that the main network can use it’s capacity towards learning better nonlinear functions ($f(x)$), while LAuReL contributes the linear components ($x_i + ABx_i$) to the residual stream. \n\n- LAuReL-PA: A hybrid of -RW and -LR variants, where multiple previous activations are used in a learned weighted manner, along with learning linear functions on top of them. This allows layers accelerated access to previous activations, along with learning their relative importance.\n\nOverall, LAuReL provides a general formulation for the residual connection, along with practical variants that can operate on the residual stream in different ways as highlighted above. LAuReL variants can be combined with each other, as seen in the ResNet experiments, as well as the small-scale ablations on an LLM reported in response to Reviewer gnJr. \n\nInterestingly, LAuReL variants consistently show better performance than naively adding a layer on top of the baseline (ResNet experiments in Table 1, small-scale LLM ablations in response to Reviewer gnJr, and LLM-2 naive scaling experiments in response to Reviewer jdcS). This demonstrates that operating on the residual stream using LAuReL has a non-trivial impact on model convergence, which cannot be matched by adding a full additional layer whereas LAuReL variants add $\\leq$ 0.1% params in these experiments.\n\n#### __Practical considerations for very deep networks (e.g., scaling to 100B+ parameter models)__\nWhile it is hard to experiment with 100B+ params, we can extrapolate from the ablations done in the paper as well as in response to Reviewer gnJr and Reviewer jdcS.\n\n- The -RW variant is straightforward to try and does not have a tangible impact on latency and memory. We expect initialization of these scalars to also be important. For earlier layers, the initial weight given to the non-linear component can be higher.\n- The -LR variant is cheap enough to also be included. We expect a slight increase in memory and latency footprint. $r$ should be scaled up as $D$ grows. In our experiments up to 4B params, the ideal ratio of $D / r$ was between 24--32. \n- The -PA variant is also helpful if the network is very deep. Using $k={3, 4, …}$ was helpful in smaller-scale LLM experiments. When keeping a larger value of $k$, the accelerator memory usage should be monitored and appropriate rematerialization strategies should be employed. Although we do not see a large increase in memory usage with the -PA variant in the smaller-scale experiments.\nIf the above variants work well with some headroom available in latency and memory, a combination of RW+LR or RW+LR+PA can be tried.\n\nWe hope these address your concerns - thanks!", "claims_and_evidence": "The claims presented in the submission are strongly supported by experimental evidence provided. The key claims of superior performance and efficiency are substantiated through well-structured experiments comparing variants of LAUREL against standard baseline models and naïve scaling. Results across ResNet-50 and LLMs convincingly support the authors' assertions about improved model efficiency.", "methods_and_evaluation_criteria": "The claims presented by the authors—namely, LAUREL’s efficiency in parameter utilization, computational latency, and memory overhead relative to naïve scaling—are convincingly supported by clear empirical evidence presented through systematic experiments. The authors conducted extensive experiments, comparing multiple LAUREL variants with standard baselines across tasks from image classification (ImageNet-1K) and various LLM evaluation benchmarks (MATH, GSM8K, BoolQ, MMLU, etc.), thus substantiating their claims effectively. No problematic claims were identified.", "theoretical_claims": "The paper does not contain any explicit theoretical proofs that require verification. Instead, the contributions are algorithmic and empirical. Mathematical formulations provided (equations defining LAUREL variants) appear correct and straightforward.", "experimental_designs_or_analyses": "The experimental design is sound and well-structured. The evaluation on ImageNet-1K and multiple well-established LLM benchmarks is methodologically rigorous. The authors used clear baselines, performed multiple trials to report statistical significance (mean and standard deviation), and conducted parameter sensitivity analyses. One minor area for deeper exploration could be a broader ablation on initialization schemes, but the current treatment is already thorough and sufficient for the current claims.", "relation_to_broader_scientific_literature": "The paper is appropriately situated within the broader scientific literature on deep learning efficiency, clearly citing key related work such as LoRA, AltUp, transformer block simplifications, model compression techniques, and distillation methods. The authors articulate how LAUREL differs and complements these approaches, providing a clear perspective of novelty and positioning their contribution clearly as a generalization of residual connections with lightweight augmentations.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n1. The proposed method is conceptually straightforward yet impactful, providing clear theoretical motivation and intuitive interpretation.\n2. Significant empirical evidence across multiple model architectures (ResNet and transformers) and domains (vision and NLP) convincingly demonstrates general applicability.\n3. Clear, thorough, and insightful experimental analysis, including ablation studies and analysis of the trade-offs regarding rank (r) parameters.\n\nWeaknesses:\n\n1. The paper could provide deeper insights or theoretical justification into why specific variants perform better under certain conditions.\n2. Practical considerations for very deep networks (e.g., scaling to 100B+ parameter models) could be discussed to better position the approach for current state-of-the-art LLM scale.\n\nOverall, the strengths outweigh the minor weaknesses mentioned above.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MJKYoXsxHV", "reviewer": "Reviewer_MkJ5", "summary": "Authors propose extensions to ResNet blocks that can improve performance with minimal addition of parameters. The extensions modifies the residual connection by adding a learnable transformation to it and/or utilises the output of previous layers.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions. \n\n#### __Ablation study with practical footprint metrics__\nThank you for suggesting this experiment. This section demonstrates improvements of LAuReL variants individually, as well as when they are combined.\n\nFor the purpose of comparing different LAuReL variants on the LLM pre-training task, we set up the following baseline. We pre-trained on the C4 corpus with $\\approx$ 10B tokens. We used a $4 \\times 4$ Google Cloud TPU v6e  topology, but we expect similar results with a comparable GPU setup.\n\nIn order to simplify the comparison across many ablations (and also to avoid the noise in downstream evals at 10B token scale), we report model performance using the test loss, which is a good proxy for downstream model quality. \n\nWe train our main baseline with 24 layers and 157.2M params, along with a larger baseline with 28 layers for comparison. We run all the LAuReL variants (RW, LR, PA) on top of the regular baseline with 24 layers. We also run two combinations of variants (RW+LR, RW+LR+PA). We report the number of params, test loss, peak memory as reported by profiling tools, and average step time. Lower is better for all metrics. The table below shows the results; note that all LAuReL experiments use L=24.\n\n| Variant                 | Params(M)    | Test Loss  | Peak Mem(GB)  | Avg. Step(sec) |\n|-------------------------:|:---------:|:---------:|:-----------:|:--------------:|\n| Baseline (L=24)         | 157.20  | 3.0159    | 11.65     | 0.095         |\n| Baseline-Large (L=28)   | 179.23  | 2.9963    | 13.23     | 0.105         |\n| LAuReL-RW                | 157.20 | 2.9557    | 11.93     | 0.095         |\n| LAuReL-LR                 | 158.40  | 2.9624    | 12.29     | 0.098         |\n| LAuReL-PA                | 157.22  | 2.9512    | 12.55     | 0.100         |\n| LAuReL-RW+LR         | 158.40  | 2.9531    | 12.57     | 0.099         |\n| LAuReL-RW+LR+PA | 160.83  | 2.9499    | 12.90     | 0.104         |\n\nAll LAuReL variants perform better than the large baseline in terms of the test loss while using much fewer parameters, lower peak memory, and lower average step time. Given the above tradeoffs in loss, memory, step time, etc. we recommend trying the LAuReL variants in this order: RW $\\rightarrow$ LR $\\rightarrow$ PA / RW+LR $\\rightarrow$ RW+LR+PA.  We will add these numbers in the revision.\n\n#### __Can α and β be learned by the network__\nThat’s a good question. Since $f(.)$ is non-linear, absorbing $\\alpha$ within the function is not equivalent to having an explicit $\\alpha$  outside $f(x)$. Additionally, the normalization would apply on top of the weighted combination of $f(x)$ and $x$, so the individual scalars would still be useful in learning the relative weights of the two components. We also demonstrate that the -RW variant is useful on its own in the above ablation study.\n\n#### __Relevant Work__\nThank you for the references.  DenseNet (Huang et al., 2018) connects every pair of layers in the network and hence in the vanilla version, all the activations need to be in memory. This is prohibitively expensive for deep LLMs and other modern transformers. When introducing dense-blocks, all previous activations within the block need to be visible to any given layer within the block; this requires refactoring the model architecture into dense blocks.\n\nOn the other hand, LAuReL requires minimal changes. In fact, in LAuReL-PA, which is the most similar to DenseNet, we make three design choices to achieve memory efficiency and performance. Firstly, each layer only looks at the past $k$ activations. For the above experiments, $k=3$ was sufficient. Secondly, we also propose using low-rank linear functions to further reduce memory usage due to activations. Thirdly, the LAuReL-PA variant uses learned scalars ($\\gamma_{i}$, $\\gamma_{i-1}$, …) to learn the weights of the previous activations (which we found to be crucial), whereas DenseNet assumes a simple sum of the previous activations. \n\nAdditionally, as seen above, LAuReL-RW and -LR variants provide significant improvements over naive-scaling, and can be combined with the -PA method. \n\nIdentity Mappings in Deep Residual Networks (He et al., 2016): This paper introduces variants of residual connections with different types of 'gating', which look similar the -RW variant, except that they use a much larger number of parameters for either the exclusive gating  or the 1x1 conv gating  ($D_in \\times D_out$ params per layer), which is much more expensive than the 1/2 params per layer of the -RW variant. For the rest of the paper, the authors introduce and focus on the pre-activation residual connection which places the activation functions such that it helps with optimization.\n\nWe will include these discussions in the revision.\n\nWe hope these address your concerns - thanks!", "claims_and_evidence": "OK, see below.", "methods_and_evaluation_criteria": "OK, see below.", "theoretical_claims": "Does not apply.", "experimental_designs_or_analyses": "OK, see below.", "relation_to_broader_scientific_literature": "OK, see Strengths and Weaknesses.", "essential_references_not_discussed": "There is a study from Kaiming He et al from 2016 (Identity Mappings in Deep Residual Networks) which also quite extensively considerers different versions of ResNet blocks. The authors should check it and cite in their paper. \n\nAnother study which might be relevant is Huang et al, Densely Connected Convolutional Networks, 2018. They also describe approach to use previous activations, but not in ResNet style.", "other_strengths_and_weaknesses": "The authors mainly consider augmented learned residual layers. Specifically they propose three variants: 1) residual weight, 2) low-rank approximation, and 3) one using previous activations. \n\nAbout the first one (RW), I am not sure there is enough novelty. He et al 2016 proposes quite similar form, even thought they do not train the parameters in similar manner as in this Laurel version. Also, He et al 2016 proposes to use 1x1 convolution for the residual layer, which is especially useful if $f(x)$ part changed dimension (e.g. different number of input and output channels in convolutions). This can perhaps be thought to be extended version of this Laurel but of course is also much more expensive and not exactly same. On the other hand, I am not fully confident that these introducing these two learnable parameters are meaningful in all cases. I would guess that $\\alpha$ and $\\beta$ could be either be included to the weights in $f$ of this or next layer or would the effect would disappear if layer or batch normalisation is applied. Anyway, the results show some kind of improvement, but I am not sure if this depends on the version of ResNet block used (see He et al 2016 for different options). The authors could perhaps experiment with it more. \nThe second one (low rank LR) seems novel and interesting and there are clearly gains shown by results (albeit their results are for combination of the first and second, but considering above, I think this LR is the key factor). \nThe third one (Laurel PA) is again a question. They did not have experiments using PA version only, but only for the version which combines all of these version. It might be that all gains are coming from LR. \n\nTo summarize: as a whole, I think this study is interesting and results looks good. However, I would wish to see Hu et al 2016 to be cited and reflect their work against this study. Also I would like to see experiments with LR and PA alone (e.g. in ResNet case) to see which one contributes has the highest contribution.", "other_comments_or_suggestions": "The manuscript is well written. I didn't find any typos.", "questions_for_authors": "No questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes to use previous activations to learn an "augmented" residual at the current layer. The review ratings are strong (3 accepts, 1 weak reject) overall. The results are general and strong,  with small overhead and simple designs. The AC checked the remarks of the weak reject review and found the author's rebuttal to be sufficient in addressing the concerns, especially regarding related work and missing metrics. The AC encourages the authors to include these content into the revision, and recommends acceptance.
Decision: Accept (poster)