DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging
Authors: Matteo Pagliardini
Amirkeivan Mohtashami
François Fleuret
Martin Jaggi
Venue: NeurIPS 2024 poster (2024)
Keywords: Transformer architecture
Large language models
Primary area: deep_learning_architectures
TL;DR: We propose a novel architecture called DenseFormer which outperforms the classical Transformer in many ways

Abstract: The transformer architecture by Vaswani et al. (2017) is now ubiquitous across application domains, from natural language processing to speech processing and image understanding. We propose DenseFormer, a simple modification to the standard architecture that improves the perplexity of the model without increasing its size---adding a few thousand parameters for large-scale models in the 100B parameters range. Our approach relies on an additional averaging step after each transformer block, which computes a weighted average of current and past representations---we refer to this operation as Depth-Weighted-Average (DWA). The learned DWA weights exhibit coherent patterns of information flow, revealing the strong and structured reuse of activations from distant layers. Experiments demonstrate that DenseFormer is more data efficient, reaching the same perplexity of much deeper transformer models, and that for the same perplexity, these new models outperform transformer baselines in terms of memory efficiency and inference time.

DenseFormer:
Enhancing Information Flow in Transformers via Depth Weighted Averaging
Matteo Pagliardini EPFL
Amirkeivan Mohtashami
EPFL
matteo. pagliardini@epfl.cd
wwirrkeiaa..nohtahhamiiee
Francois Fleuret
Martin Jaggi
University of Geneva Francois. F www.retuunggeeec..mmmm
EPFL
jagtin.jaggieepfl...c
Abstract
The transformer architecture by Vaswani et al. [31] is now ubiquitous across application domains, from natural language processing to speech processing and image understanding, We propose DenseFormer, simple modification to the standard architecture that improves the perplexity the model without increasing its size- size-adding at most few thousand parameters for large-scale models. Our approach relies on an additional averaging step after each transformer block, which refer to this -veerae (DWA). The learned DWA weights exhibit
computes weighted average of current and past representations operation as Depth- coherent patterns of information flow, revealing the strong and structured reuse activations from distant layers. Experiments demonstrate that DenseFormer is more data efficient, reaching the same perplexity of much deeper transformer models, and that for the same perplexity, these new models outperform transformer baselines in terms of memory efficiency and inference time.
Introduction
The transformer architecture [31] the workhorse modern natural language processing. Recent leaps in the state of the art can be attributed in a large part to efforts scaling this architecture, from millions of parameters [3] to large pillien-parameter models [1, 18, 23, 29, 30). Unfortunately, those larger models come with an increased computational cost, and large memory footprint. This renders them impractical to use in a wide range of use-cases, limiting who can benefit from them to a handful of big corporations. As attempt mitigate this issue, Touvron et al. [29] propose training smaller model for more steps. However, longer training requires larger datasets which becomes challenging as reaching scales where extremely large datasets fall short of sufficient amounts of data [32].
Furthermore, recent observations suggest that we reaching the state of diminishing returns where increasing the depth of the model beyond a certain point does not significantly improve performance [21]. Interestingly, similar state of diminishing returns has been observed in the field of computer vision focused on the training of deep convolutional neural networks. Various solutions were proposed address this issue, including DenseNets [1] which alleviated the problem by allowing subsequent layers to directly access outputs of earlier layers.
In this work, using similar intuition as DenseNets, we propose the DenseFormer architecture. In particular, instead of only having skip connections from one block to the next, in DenseFormer,
"Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Traudiena Blok
Transfier Wick
Dilation
Dilation
Dilation =
Xa
DWA tndex
TWW
ITO
(a) DenseFormer Architecture.
(b) DWA Weights with Dilation.
Figure 1: DenseFormer architecture. The diagram in (a) shows the DenseFormer architecture
with two transformer layers and dilation of 1. After the first (resp. second) block, the past and current intermediary representations {Xo, {1} (resp. {Xo, X1, X2)) are averaged using the first (resp. second) DWA weights [01100 01,1] (resp. [0-20, (02,0,002,1,4 0 ). The DWA weights are supported by red arrows. Those weights are represented in matrix form in (b), for a 12 layers DenseFormer. A DWA module at depth has i + I weights, represented in red. Increasing the dilation sparsifies this matrix, reducing the computational overhead without degrading the perplexity, see Section 3.2 for more details.
weighted average of the outputs of all previous blocks is given as the input of the next block. The approach is visually summarized in Fig. la.
We show that Fenserormers can perform the same as a much deeper standard Transformer model while at the same time being smaller in size, faster, and consuming less memory at inference. More importantly, this achieved without increasing the amount of required data. As such, DenseFormers are also more data efficient, obtaining much better performance when trained on the same amount of data than a standard model with a similar number of parameters. Our results establish the DenseFormer architecture as improved version of Transformers for language modeling, encouraging their future use.
In addition to providing experimental results on Fenserormer's performance, we also provide additional insights and intuition for their success. Looking the learned weights of the DWA modules we observe surprisingly stable pattern in the learned weights that emerges at multiple depths and generalizes across random seeds (see Fig. 4). Overall, similar to Huang et al. [11], we hypothesize that the inter-block connectivity enables the model to more directly re-use early features, without requiring allocate much bandwidth to propagate them through multiple layers. Intuitively, this seems to help resolve an ambiguity caused by skip-connections which force the deeper representations to maintain the current token representation while at the same time having to predict the next token.
Contributions. Our contributions can be summarized as follows:
Introducing DenseFormer architecture by adding a Transformer block.
aveergge module after each
Demonstrating over different settings (e.g. datasets, batch sizes, sequence lengths) the sig- nificantly superior performance of DenseFormer over deeper Transformers, yielding a better speed-performance trade-off during both inference and training.
Providing additional empirically grounded insights and intuition that support the benefits of using DenseFormer.
2 Related Work
While larger models have shown great promise in delivering better capabilities, recent results suggest that the gain from using deeper models faces diminishing returns [22]. Interestingly, the same challenge presented itself earlier for scaling convolutional neural networks [9]. Methods that allow a better flow of information from earlier to later layers such as Residual connections [10] and Highway Networks [27] have been proposed as a successful solution to tackle this challenge, stabilizing training and increasing the threshold where a gain can be observed from increasing depth. Taking it to the extreme, DenseNets [11] demonstrate the benefit of having access to the output of previous layers. We propose DenseFormer by building on similar intuition, allowing each block to directly access the output of all previous blocks.
Some advantages of attending to representations from earlier layers have already been explored in prior work. Desth-wise Attention [4] suggests adding an attention- like layer before Transformer's

---

final projection layer. This new layer applies attention across the outputs of the Transformer's blocks for the current token (instead of over different tokens as in the standard attention layer). This operation is similar to the weighted averaging block introduced in this work which mixes the outputs of earlier blocks. However, whereas in our proposal the weights are learned during training, Depth-wise Attention computes the weights using the dot product (similar to the attention) which levies higher overhead. In our experiments, we also show that only a single DWA step before the last layer does not yield the same performance as full DenseFormer. In another recent relevant work Mohtashami et al. [17] suggest interleaving current and past representations. This allows current tokens to attend to previous representations of themselves and past tokens. Our DenseFormer can be seen as a crude and much more efficient approximation of this mechanism in which we restrict each token to only attend to past representations of themselves, using static (as opposed to dynamic) attention weights. Most relevant to us-in the context of low data training-the concurrent work of Charpentier & Samuel [2] also adds links from the input of each block representations from earlier layers. An important difference is their use of softmax weights. We conjecture it is important to let the model subtract representations using negative weights (see & 5), and implementing their methods only outperformed the baseline in low-data regime (see App. B.7). Moreover, they do not provide an efficient implementation of their method.
Since its original design by Vaswani al. [31], the Transformer architecture used in most applications changed surprisingly little. LLM training is costly, and architecture choices are often conservative. Small variations include changing the activation function [26], adopting RMSNorm instead of Lay- enNorm. or computing the feed-forward and attention in parallel [20, 29, 33]. More progressive proposals have been made to alleviate computational challenges of the self attention module, such as using kernel methods or other linear approximations [12, 13, 34J, or removing redundant operations without impacting performance [8, 25]. These proposals only affect the internal structure of Trans- former blocks. As DenseFormer only adds DWA modules that operate between blocks, we expect that it can be readily used together with these existing proposals.
Recent explorations also have shown gains from using multiple language models instead of one. An example is using a mixture of experts, which rely on a routing mechanism [5] to select which expert(s) to use a given context. Other examples include deploying the same instance of a model in different roles allowing them to debate or provide feedback to each other leading to performance improvements [14, 16]. As these approaches mostly retain the structure of the Transformer architecture and focus on the communication structure between multiple models (or sub-modules), we also expect them to be adaptable to use DenseFormers.
3 Method
Setup Notations. We consider the standard Transformer architecture. Given a depth d, it consists in a succession of d Transformer blocks BI...., Ba, each composed of self-attention module followed by a single hidden layer Baati-Layer-Perceptee (MI.P). We name Xo...., Xa the different intermediary representations, with Xo being the embedded token sequence, and X, for i 2 being the output of block B,.
We summarize the Transformer architecture as follows:
Xo == Embedding( (X)
Vi = 1,... d, X; = 1x(X.-1)
Transformer(X) == Xx.
DenseFormer. The only change to the original architecture is the addition of a Depth Weighted Average module (DWA) after each transformer block. A DWA module at depth performs weighted average between (i) the output from the current block B,, (ii) the output of all previous blocks Bjci, and (iii) the embedded input Xo. The weights of the weighted-average for the DWA, module at depth i are 0s.0..... Or.i. A visual summary can be seen in Fig 1a. The elements of the a matrix are the only additional parameters of our method. More formally, our DenseFormer model can be
3

---

summarized as follows:
Xo = Embedding
Yo = Xo
= l....dd. d, X; = B;(Y,-1)
Vi = I,...d, d, Y = ((
Σ 1=
aij Xj
DenseFormer((X) == Ya.
In Section 4 we demonstrate that the DenseFormer architecture can outperform the standard Trans- former architecture. In particular, it obtains a much stronger performance (in terms of perplexity) than model of the same depth, matching the performance of much deeper model which is both slower at inference and larger in size, leading to a much higher memory footprint than DenseFormer. We further demonstrate the importance of the improved inter-block connectivity brought by the DWA modules in Section 5. We do so by comparing our architecture to a variety of baselines with constrained connections and show those do not perform as well as DenseFormers.
Initializing the DWA modules. We note that if a; set to I while others are set to the DWA module acts as an identity function, reducing DenseFormer to the standard Transformer architecture. Therefore, we start our training from this initialization.
3.1 Impact on Resources
Negligible Model Size Overhead. At depth i the DWA module has + weights. Therefore, for a DenseFormer of depth d, the total number of additional parameters is ン==1 (j ( 1) d(d+3) For
typical model depths (less than 100 blocks), this represents most an order of 10s parameters, which is negligible when compared to the full size of the models.
Negligible Memory Overhead. We also emphasize that while DWA requires access to the output of blocks and embedded input Xo....; Xd, these values are stored even when using the standard Transformer architecture. During training, the outputs of the blocks are kept in memory to allow backpropagation, while inference, only the blocks' outputs for the current token need be stored. Therefore, the total memory overhead of DenseFormer is negligible.
Computational Overhead. Computing the output of the DWA modules increases the computational cost since it requires averaging over multiple large tensors of size (batch size sequence length x hidden dimension). In this work, we provide an efficient implementation of DWA to reduce the overhead and avoid unnecessary data movement. addition, we introduce two architectural hyperpa- rameters, which allow building set of DenseFormer variants that approximate the full DenseFormer. These hyperparameters are DWA dilation and DWA periodicity, respectively introduced in Sec- tions 3.2 and 3.3. We refer to DenseFormer variant with the dilation factor k and the DWA periodicity p as Foy-DerseForrmer. this notation, the full Denselorrerr is la FormeerrsFFrrer..
3.2 Dilated DenseFormer
In order to further reduce the computational overhead, we introduce a dilation parameter which sparsifies the DWA weights by periodically setting them In particular, each DWA module is now given the output of every k-th block, where is called the dilation factor. More formally, given a DWA module at depth i, dilation factor of k implies DWA; only computing weighted average over {Xylj < i.j = i(mod })) See Fig. 1b for visual explanation. Our dilated DenseFormer can be described as:
Xo Embedding (X)
Yo = Xo
Vi 1, d, X, = B.(Y:-1)
Vi Y, DWA, dilatlated ((Xyli S i,j i(mod k)))
(om = Ya-
As shown in Section 4, we observe no noticeable performance degradation for small values of (e.g. 2 or 4) while the computational overhead is significantly reduced, leading to much faster inference

---

4xl Denselormer
18.5
Trunsformer
Dilation (2
Diacica 4,D Period
Dilaton DWA Parsod
14
18.0
Paplesdo
DWA index
DWA index
DWA index
48
17.5
101
17.0
X2
Xn
104
(a) DWA weights with dilation and DWA period.
16.5
104
Batches Per Second
(b) Speed and performance trade-off.
Figure 2: (a): DWA weights with dilation and DWA period: For 12 layers DenseFormer, the weights sparsified using dilation k and DWA periodicity p (referred to kxp). Compared to Fig. lb, only certain rows have some weights other than the upper diagonal weights (which correspond to the regular transformer information flow). Increasing the dilation and period sparsifies the a matrix, reducing the computational overhead without degrading the perplexity, see Sections 3.2 and 3.3 for more details. (b): Speed and performance trade-off:: Comparison of speed and performance trade-off between the standard Transformer architecture and DenseFormer. The number of blocks in each architecture is reported next the data-point. All DenseFormer models on this plot use dilation factor of 4. We show results using DWA period of 1 and 5. Comparing perplexities: Considering only the perplexity (y-axis), 48 block DenseFormer performs similarly as much deeper 72 block Transformer. Comparing trade-offs: A 48 block 9K5-DenseFormer matches the better perplexity of 72 block Transformer while being 1.4x faster at inference.
and training. More precisely, a dilation of reduces the computational overhead induced by DWA modules by a factor of 1/k.
3.3 Periodic DenseFormer
An alternative method dilation for sparsifying DWA weights is adding DWA modules to the architecture less frequently. In particular, we can consider only adding the DWA after every blocks (instead of after every block as in the standard DenseFormer). We refer to p as the DWA period. A standard DenseFormer has period 1. A FenseFormer with dilation and DWA period p p--referred to kxpp-DenseFormer--oe be formalized similarly DenseFormer with the following distinction:
DWA, dilated dilated
(lx, S i.j i(mod k)})) p
Vi 1, d, Y,
Xi
pta
visual representation of the matrix of weights can be seen in Fig. 2a. By increasing the periodicity we further reduce the computational cost of FenseFormer. In Section 4 we evaluate the effect of increasing the period for 4-dilated DenseFormer that using small values larger than for the period can provide noticeable boost in speed without noticeable performance degradation. Using period of p reduces the computational overhead by factor of l/p. Hence, a Exp-DenseFormee only has 1/kp of the computational overhead of a regular Denserormer. In Section 5 we also provide results with other sparsity patterns for the DWA weights (a) but show that using dilation and periodicity works most favorably.
both performance and speed. We observe
Interplay between A and The ideal value of can depend on the dilation k used. For instance, using i p = 4 implies the DWA module after block will look at {Xo, X²). Its output, Y,, will be sent through block 5 to yield Xş. However, the next DWA module after block 8 will only look at (Xo, X., Xs} (and not at Xs). This means that Y. will have to go through blocks 5, 6, 7,8 before being accessible by later DWA modules. In contrast, using k = 4 and to propagate much faster since DWA modules always have access to the processed output of the previous DWA module. This interplay can be visualized in Fig. 2a as well as in Appendix B.2.
5 allows the information
5

---

Table 1: Performance of DenseFormer and the standard architecture of different sizes WpenWeeTexx2 dataset. The number millions of parameters is reported as well as the final perplexity. Additionally the number of batches of size 64 that can be processed in second is reported as measure of inference speed. The results are based on three runs with different seeds. The mean value is reported with the standard error reported in parenthesis. In terms of perplexity, DenseFormer clearly outperforms a standard Transformer of the same depth as well as standard Transformers with similar inference speed. While sometimes a deeper model with the standard architecture can match the performance of shallower DenseFormer (e.g. 72 block standard architecture and 48 block DenseFormer), inference using the shallow DenseFormer remains much faster. The inference speed is significantly improved with negligible effect on perplexity when increasing the dilation factor and DWA period. Adding a scaling factor all skip connections in the standard architecture (named Skips with Gains) does not yield the same performance boost as Dense highlighting the importance of inter-block connectivity DenseFormer.
Model
Dilation x Period
Depth
Parameters # (M)
Perplexity (L)
Inference BPS (1)
Transformer
48
378.45
18.61 (0.02)
5.94 (0.00)
Skips With Gains
48
378.45
18.45 (0.03)
5.72 (0.01)
X
48
378.45
17.84 (0.00)
4.65 (0.00)
DenseFormer
4 X
48
378.45
17.86 (0.02)
5.31 (0.01)
4 x
48
378.45
17.87 (0.02)
5.72 (0.00)
64
491.72
17.94 (0.01)
4,57 (0.00)
Transformer
72
548.35
17.82 (0.04)
4.08 (0.00)
1x
72
548.36
17.12 (0.02)
2.93 (0.00)
DenseFormer
548.35
17.17 (0.00)
3.60 (0.00)
72
548.35
17.21 (0.01)
3,90 (0.00)
633.31
17.48 (0.01)
3.54 (0.00)
Transformer
90
675.78
17.44 (0.01)
3.32 (0.00)
Results
We demonstrate the effectiveness of DenseFormer through experiments on language modeling tasks. We compare the performance of DenseFormer architectures with the standard Transformer architecture using model size, inference time, training time, and final perplexity (sometimes abbreviated PPL) as metries. For each metric, consider a baseline that performs the same as DenseFormer on that metric. Concretely, we include the following baselines:
Same Depth Baseline. A standard architecture with the same depth as the DenseFormer. It roughly has the same number of parameters as DenseFormer given the negligible number DWA parameters.
Same Inference Time Baseline. standard architecture that has the same inference time as the DenseFormer. Since adding DWAs to the architecture has computational overhead. this baseline has more layers (i.e. more capacity) than DenscFormer.
Same Perplexity Baseline. standard architecture with roughly the same PPL as the DenseFormer. It usually is much deeper than the Former, showcasing the benefits of using DWAs.
Same Training Time Baseline. A standard architecture that has the same training time as the DenseFormer. Since adding DWAs the architecture has a computational overhead, this baseline is trained for more iterations than DenseFormer.
Skips with Gains. It can be observed that DenseFormer, among other things, allows scaling the output of the previous layer, providing more control than the original skip connections. Therefore, we provide an additional baseline to show this is not the only benefit offered by DenseFormer and emphasize the importance of having direct access the outputs of earlier layers. particular, we consider modified version of the standard architecture where each skip connection also contains learned scaling factor which is applied to the the values moving through the skip connection before being summed with the output from different layer (e.g. self-attention).
Experimental setup. We use models with 8 heads, each having 64 dimensions, and train them using batches of 400 sequences of length 256. We use rotary positional encoding [28]. We optimize the model using AdamW [15] with 31 0.9 and B2 0.95 with weight decay factor 0.1.

---

18.6
18.6
eq-Daemrrorrrse Trandomus
24-
Trazakonax (41 iin)
Transeuue
Gob-Dreellieuee (lke ners
18.1
18-1
Poogledity
Perpledit
Perplesity
-2
18.2
18.2
e-4r20
20-
tell
knill
11xl
off/
111
105
18-
17.8
17.6
3.0
5.3
6.0
1.0
1.4
Dotehre Per Scoond tchnring infrennce
Batches Per Second (durin training)
Time Shoum)
(a) PPL vs. Inference speed (BPS) (b) PPL. vs. Training speed (BPS) (c) Comparison under same training time
Figure 3: Training and inference efficiency of kxp--Dnnseoorer vs. Transformer. For 48 block models, we compare in (a) the different perplexityinffrence speed trade-offs reached by a regular Transformer and kxp-Dense Formnerrrrrr.s. In the top right corner, the Transformer baseline is the model with the worst perplexity but the fastest at inference. In contrast, the FoomDenseorrmer in the bottom left corner, is reaching the best perplexity but incurs a cost in inference speed. By varying the dilation k and DWA periody p. some Exp-DenseFormn Former models (c.g. 4x5) provide most of the perplexity improve- ment of the original DenseFormer while significantly reducing the time overhead. A similar analysis holds when looking at the training speed in (b). In (c), we show the perplexity decreasing during training. The x-axis is time. To compensate for the computational overhead of DenseFormer, we train the Transformer baseline for more iterations, such that the two methods have the same training time budget. We observe how our 4x5- For-merseFormer is reaching a better perplexity faster than the baseline.
We perform most of our experiments on the Open CebTextoorpus [7] with around 17B tokens. We train all models for 40k steps, thus keeping the number of data points used in training fixed. We use learning rate 0.001 with a cosine scheduler and do warmup in the beginning 5% steps.
dataset [6], an enhanced version of
We present the result of training 48 block and 72 block DenseFormers along with baselines of various sizes in Tab. 1. We make the following observations based on these results:
Better perplexity than same depth baseline. When comparing with a baseline of the same depth, DenseFormer significantly outperforms the standard architecture. Moreover, it can be seen in Fig. 2b, the perplexity of a 48 block DenseFormer is only matched by a 72 block Transformer baseline.
Faster than a baseline with the same perplexity. The performance of a 48 block Fonser is on par with 72 block standard architecture. Still, the 48 block DenseFormer is much faster at inference (measured in batches per second) than the 72 block standard architecture. Moreover, the number of parameters and memory footprint of the 72 block baseline is 45% larger than the one of the 48 block DenseFormer.
Better perplexity than a baseline with the same inference time. Comparing the 48 block Dense- Former without dilation, with 64 block standard architecture (which has the same inference speed), shows a wide gap between the higher performance of DenseFormer (17.8 844 and the standard Trans- former architecture (17.94). Considering DenseFormer models with a dilation of 4 and/or a DWA period of 5 would increase this gap further.
Weighted skip-connections are insufficient. DenseFormer is changing the flow of information in the model. We can wonder whether it leverages the additional expressivity or whether the performance gains could be explained by making it easier to rescale the contribution of each block. When compar- ing the 48 block Transformer baseline the 48 block skip-with-gee -aiin baseline, it seems adding tunable weights to each skip connection does not lead to a significant improvement. When compared with the 48 block DenseFormer, this showcases the importance of having direct access all previous layers.
Faster with dilation and DWA period. Finally, our Tab. results show that for small dilation factors k and DWA period p. kxy-DensefFormer perform comparably while significantly boosting speed. Indeed, as can also be seen in Fig. 2b, using For-DenseFormers or 4x5-DenneeFormers allows pushing the Pareto frontier on speed and performance trade-off forward.
More efficient during training. We train 48 block 4x5-DenseFormme and compare it against a 48 block Transformer baseline trained with the same time budget. The baseline therefore trained for more iterations (41.5k vs. 40k) compensate for the time overhead of DenseFormer. In Fig, 3c
7

---

30
10
1780 hedre
DMN Intre
00
60-
(a) 48 block DenseFormer.
(b) 72 block DenseFormer.
(c) 48-block 4x1 -or-eenseFormer.
Figure 4: Visualization of DWA Learned Weights. Each row shows the weights a learned by a DWA module at given depth. While the heatmaps averaged across runs with different seeds, those patterns are very consistent across seeds. (a) and (b), strikingly similar patterns can be observed in both 48 and 72 layer DenseFormers. In (c), show the learned weights for 48 block DenseFormer trained with a dilation of 4. Despite the sparsity, we still observe a very similar pattern to those learned by the non-dilated models.
Table 2: Same training time comparison. Comparison of JX5-DenseFormer's performance against standard Transformer trained for iterations. The number of training steps of the standard architecture is chosen such that the training time is roughly the same (and always more than) that of the DenseFormer. Both architectures have 48 blocks and are trained with 2000 warmup steps. Even though the Transformer is trained with more steps, it is still outperformed by the DenseFormer.
Model
Steps
Train time (h)
Perplexity
Standard
41500
8.09
18.33 (0.00)
ForrmerseFormer
40000
8.04
17.87 (0.02)
visualize the perplexity (approximated on a small subset of the validation set) dropping as function of the training time. The DenseFormer's perplexity is dropping faster than that of the Transformer baseline. This shows the superior efficiency of DenseFormer during training. While the Transformer is trained for more steps, thus using more data points, it is still outperformed by the DenseFormer. The final perplexities on full validation set reached by the two models can be seen in Tab. 2.
4.1 Additional Experiments
We perform additional experiments to show that our results are general and extend to different settings and larger scales. We also study the impact of the dilation factor l and DWA period p the efficiency of our eop-DenneFormer architecture.
Experiments with longer sequences. Due to computation limitations, we can not run all experiments large scale. We however repeat limited set of experiments with longer sequences of 512 tokens using a smaller batch size of 128. A 48 block Transformer baseline reaches a final perplexity of 18.28 + 0.03 against 17.73 + 0.02 for the DenseFormer. This result shows that the gap between the two architectures persists for longer sequences.
Effect of Dilation and DWA period. Fig. 3a and Fig. 3b show the impact of using different combinations of dilation & and DWA period p on the final perplexity, training and inference speed. As can be seen, small values of the dilation factor (e.g. up to 4) have a negligible effect on the perplexity. However, increasing the dilation factor further affects the performance more adversely while the gain in both training and inference speed starts to plateau. Increasing the DWA period also provides a similar trade-off, with the perplexity being barely affected for p S. From those figures, we conclude that dilation of 4 and a DWA period of 5 seem to offer the best compromise between speed and perplexity. In Appendix B.2, we provide more detailed results, including showing how increasing dilation yields more pronounced speed-up for deeper models, making larger dilation factors more effective those scales.
PG-19 experiments. We also show the superior performance of DenseFormer on the PG- 19 dataset [24]. At any depth, DenscFormers outperform Transformers in PPL. See App. B.3 for details.

---

0.20
DenseFormer
Table 3: Alternative DWA Sparsity Patterns. We compare 48 block architectures with different sparsity patters. In Last K. DWA can only access the output of the last k blocks as well as the input embedding vectors. Connect to Last includes only single DWA module in the architecture placed after the last layer, i.e. connecting every block to the last block. Neither of these patterns achieves the same perplexity boost as our proposed Dense- Former. 12x1 and sities of respectively 92% and 95% ootperforr other sparsity patterns, which signifies the impor- tance of pairwise inter-block connections.
Cosine similarity
Transforner
0.15
0.10
0.05
0.00
6
12
18
24
30
36
42
48
Depth
spar-
Figure 5: Cosine similarity between the output of each DWA module and the initial embedding
vectors. The results are averaged over three seeds, for DenseFormer models with 48 blocks and no dilation (corresponding to the weights in Fig. la).
The model initially maintains a high correlation between the input and the output of each DWA module. That correlation decreases in later layers. Intuitively, we can hypothesize that this is the stage where the model is preparing to output the next token. A very similar plot can be observed for 72 block models in Appendix B.I.
Sparsity Pattern
Perplexity
Sparsity
Baseline Transformer
18.61 (0.02)
Last K (K = 4) 18.28 (0.01)
84%
Connect Last
18.33 (0.02)
96%
12x1-DenneFormer
17.96 (0.01)
92%
FormmerseFormer
17.87 (0.02)
95%
5 Analyzing the Information Flow
In this section, we investigate the learned DWA a weights to gain more insight and intuition on the reason behind the superiority of DenseFormer.
A stable weight pattern emerges. We start by visualizing the learnt DWA weights for 48 and 72-block DenseFormer models (with a dilation and period of 1) in Fig. 4. Interestingly, the a weight patterns learned at both depths are very similar:
High weights are on the diagonal (corresponding to the normal information flow as in a standard Transformer) as well as on the immediate previous blocks.
High weights are given to the initial embedding vectors. Those weights are positive in earlier layers while later layers assign negative weight.
An aggregation block is observed near the final layers where a high weight is given to all previous layers in the block (seen as high-weight triangle near the diagonal in the lower right corner).
Finally, Fig. 4c shows similar patterns persist to some extent when using dilation factors higher than 1. Similar results hold for Rxp-DenseFormers as can be seen in Appendix B.1.
Other sparsity patterns during training. Alternative pruning after training, also consider imposing different sparsity patterns during training. We already have experimented with such sparsity patterns induced through dilation and DWA period. In Fig. 4, we observe that in many cases the largest weights are given to the few previous blocks as well as the first block. As such, we experiment with a sparsity pattern allowing DWA to access previous k blocks as well as the input embedding vectors calling it "Last K". Furthermore, given the large magnitude of weights on the last layer, we also experiment with only having a single DWA in the architecture which placed after the last layer, calling it "Connect to Last", Tab. 3 shows the perplexities when using these sparsity patterns and shows that they do not achieve the boost in performance obtained with DenseFormer. This observation further strengthens the importance of small DWA weights both during training and inference.
Correlation with Input Embeddings. Based on the special pattern of weights given to the embedding vectors especially the negative weights given to the input by the last layers (see Fig. 4) 4) we hypothesize that the model tries to use the information in the input in earlier layers while removing the influence of the current token as it tries to predict the next token. In order to test this hypothesis we plot the average cosine similarity of each token's vector after each DWA block with its input embedding Fig. 5. As expected based on the weight pattern, we observe that the similarity is high in the earlier layers. At the later stage, the model decreases this similarity significantly.
eececal-es

---

We hypothesize this final decrease is due to the model moving from processing the current token building the next token's representation. In contrast, the similarity drops down very early for standard transformer and remains low for the rest of the layers.
6 Future Work & Conclusion
In this paper, we introduced the DenseFormer architecture. This architecture adds an averaging module called DWA after each block which allows it to directly access the outputs of previous blocks. We established the superiority of this architecture over Transformers in terms of perplexity/speed trade-off through experiments in a variety of settings, Additionally, we provided dilation and DWA periodicity as simple methods to improve speed without significantly hurting performance. Finally, we provided insights about the learned weights, revealing patterns persisting over different depths.
As the next steps, finding more efficient implementations of DenseFormer is grounds for future work. One possible direction is finding better sparsity patterns that also can be implemented efficiently. The weights visualization in Fig. suggests such patterns might exist. Furthermore, finding efficient ways to shard DWA across multiple nodes is important to allow large-scale distributed training.
References
[1] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A.,
Shyam, P., Sastry, G., Askell, A., Agarwal, S., Beerbert-osss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language Models are Few-Shot Learners, July 2020. URL httt:://trrstt [cs].
org/abs/ 14165. arXiv:2005..4111
[2] Charpentier, L. G. G. and Samuel, D. Not all layers are equally as important: Every layer counts BERT. CoRR, abs/2311.022 1.02265, 2023.
[3] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
[4] EINokrashy, M., AlKhamissi, B., and Diab, M. Depth-wise attention (dwatt): A layer fusion method for data-efficien classification. arXiv preprint arXiv:2209.15/ 2022.
[5] Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120): 1-39, 2022.
[6] Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang. J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, Open WebText2 dataset, as part 'the Pile: An 800gb dataset of diverse text for language modeling'. arXiv preprint arXiv:2101.0002 2020.
[7] Gokaslan, A. and Cohen, V. Openwebtext corpus. http://Skkyiinn07-.itthht io/ OpenWebTextCorpuss 2019.
[8] He, B. and Hofmann, T. Simplifying transformer blocks. CORR, abs/2311.01904 2023.
[9] He, K. and Sun, J. Convolutional neural networks at constrained time cost. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5353-5360, 2015.
[10] He, K., Zhang, X., Ren, S., and Sun, Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778. 2016.
[11] Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In CVPR, pp. 2261-2269. IEEE Computer Society, 2017.
[12] Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are RNNs: Fast
autoregressive transformers with linear attention. In Proceedings of the International Conference on Machine Learning (ICMI), pp. 5294-5303, 2020. URL. https: // /tte://///lett.eee oeaeei.rrgpppppppeee papers/ katharopoulos-et- al-icm12020000 pdf.
10

---

[13] Kitaev, N., Kaiser, L., and Levskaya, A. Reformer: The efficient transformer. In ICLR. OpenReview.net, 2020.
[14] Liang T., He, Z., Jiao, W. Wang. X., Wang, Y., Wang, R., Yang, Y., Tu, Z., and Shi, S. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint 2r0.v:2305.191 2023.
[15] Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv.1711.05101, 2017.
[16] Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with eeff-feedback. arXiv preprint w3w. 2023.
[17] Mohtashami, A., Pagliardini, M., and Jaggi, M. Cotformer: More tokens with attention make up for less depth. In Workshop on Advancing Neural Network Training: Computational Efficiency. Scalability, and Resource Optimization (WANT@ @ NeurlPS 2023), 2023.
[18] OpenAl. GPT-4 Technical Report, March 2023. URL http:////aaxxi org/a//// /a//abb//222000 08774. MiXiv:2303.08774 [cs].
[19] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang. E., Vito,o, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in pytorch. 2017.
[20] Penedo, G., Malartic, Q.. Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., and Launay, The refinedweb dataset for falcon LLM: outperforming curated corpora with web data, and web data only. CoRR, 0bs/2306.0011/// 2023.
[21] Petty, J., van Steenkiste, S., Dasgupta, I., Sha, F., Garrette, D., and Linzen. T. The impact of depth and width on transformer language model generalization. CoRR, abs/2310.19956 2023.
[22] Petty, J., van Steenkiste, S., Dasgupta, I., Sha, F,, Garrette, D., and Linzen, T. The im- pact of depth and width on transformer language model generalization. arXiv preprint arXiv:2310.1992 19955....9999.... 2023.
[23] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners, 2019.
[24] Rae, J. W., Potapenko, A., Jayakumar, S. M., Hillier, C., and Lillicrap, T. P. Compressive transformers for long-range sequence modelling. ICLR. OpenReview.net, 2020.
[25] Shazeer, N. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02111 2019.
[26] Shazeer, GLU variants improve transformer. CoRR, abs/2002.05202. 2020.
[27] Srivastava, R. K., Greff, K., and Schmidhuber, J. Highway networks. arXiv preprint arXiv:1505.0038/ 2015.
[28] Su, J., Lu, Y., Pan. S., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. CoRR, abs/2104.09864, 2021.
[29] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models, 2023.
[30] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra. S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes. J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami. V., Goyal, N., Hartshorn, Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, Liskovich, D., Lu, Mao, Y. Martinet, X., Mihaylov, T., Mishra, P.. Molybog, I., Nie, Y. Poulton, A.. Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models, 2023.

---

[31] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. CoRR, abs/1706.03762, 2017. URL http: //arxiv. wwt./a/g//////////o///..ot..eeet abs /106. 03762.
[32] Villalobos, P., Sevilla, J., Heim, L., Besiroglu, T., Hobbhahn, M., and Ho, A. Will we run out of data? an analysis of the limits of scaling datasets machine learning, arXiv preprint arXiv:221..04:3 2022.
[33] Wang, B. and Komatsuzaki, A. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.
com wtwp:ithwb.oommiiioooooooooooo.
toath--ttraaofmmr---ite jaxx May 2021.
[34] Wang. S., Li, B. Z., Khabsa, M., Fang. H., and Ma, H. Linformer: Self-attention with linear complexity. CoRR, abs/2006.04766 2020.
12

---

Contents
1
Introduction
2 Related Work
3 Method
3.1 Impact on Resources
3.2
Dilated DenseFormer
3.3
Periodic DenseFormer
4
Results
4.1
Additional Experiments
Analyzing the Information Flow
Future Work Conclusion
10
Implementation details
14
A.1 Naive Pytorch implementation
14
A.2 More Optimized Pytorch implementation
15
A.3
Hyperparameters and Hardware Used
16
Additional Results
17
B.1
Information Flow
17
B.2
Analysis of Dilation and DWA Period
18
B.3
PG-19 Experiments
19
B.4
Delaying the Training of DWA Weights
19
B.5
Rank Analysis
19
B.6
Experiments with a batch size of 128
19
B.7 Comparison with ELC-BERT
23
B.8 Effect of varying the width
23
13

---

A Implementation details
In this section, we provide several implementations in Pytorch [19]. The first implementations we propose are very simple and rely looping over the past representations to compute each DWA output. As result, those naive implementations are slow when dilation and DWA periodicity are not used. In second time, we propose for a Formeer with 48 blocks, the naive implementation takes 67 per training itcration against 657ms for the later implementation. In Appendix A.2, we provide denseformer python package to help turn Transformers into DenseFormers in only simple steps.
optimized and faster implementation. As comparison,
TL;DR: you want to implement your own DenseFormer, simply check the densef ormer python package: https: //// thub. com/ cow/epfml//eccccccccccccccccc wwtp////esseemmmaaaaaamma
A.1 Naive Pytorch implementation
Naive Ixo-DenseFormer implementation. naive implementation in Pytorch would consist of storing the output of each block during the forward, and feeding those representations to DWA modules after each block. pseudocode would look like the following:
Listing Naive Ixommrr implementation
import torch
claxs (torrh en. Nadula):
def __isit_. (aulf,, n_alphea);
super ....intt...
self a__iphas n_alphas
alphas terrh.serox(ffeeephaa
10
alptar[-]] [-1] 1.0
11
self. aelhaaiphas Sorch....Paraneteeeeaap...
aiphas
12
13
def forvard(self,, pll.previons_x)
14 veighted_avg all_previous_4le
15
for weighred_arg alphasip (11
self selt-n_alphani]
16
17
veighsed_avg
18
20
GPTBann (torch Modula):
21
22
def ...ist... (aulf, config):
23
super .n....nt......
24
self. coafig = config
25
mnlf-dva_endul. selt
for range )llookk))) config.n. n.ebb)) Modullllloddiee ((loeiiii(iieeecaaaaaiiii
torch KndulaLiat (IDVA
26
Snbedding
self. blocks Layarliors Linear (contig contiggnn_ssbl a_onbd, config.voce vocab, size, biac-Falce)
range (eggeeeeti....bleeee
28
b bias=cantig.bing
lo_bead
aulf. transforner. ste..err..t....i....... self. he, bead. neight seight tying
31
22
def forvard(seif, idx):
33
x self. vte velf.vte
34
all.previous.3
[x] This stores all the internediary representatione for i ia (self. a.
35
16
self.bincka [i]
37
<ll_previous_= append
38
des.f.dda_aocee (alf lagits salf.1a_haade return logits
wtw../ooieeeltllallllllcccoo (alllpootts(llll.rrrrtnee.. Computing zeighted
39
Naive Fen-Derneerrmmr implementation. Here we introduce naive implementation with dilation and DWA-frequency. The DWA module in the following implementation would be the same as for the 1x1 FereDerseFrrmer naive implementation.
Listing 2: Naive kxp--DenseFormer implementation
class
Module))
def iiiit... (aulf, config):
super ()...intt... self.
wnlffdva_nndulew
Mn..shha...odaiitittttt.... Kodulaliitt wtw.beii.........aaahaoaarrririd
for in range 2tw.eecc.a.g..occcee..
aelf.vte torch.
labeddiag(coofiig vocab. size, config.n anebbd) for in range
swif.blos blocks sarch-mn
self. .i_.f LayorSors (eoforrncoonig.... exbd, sarfcosfig.bias
10
aelf. awlf.in han_head torch. en.ch..n..fe Linear Cennearroonfi (config, iigg.,, n_snbd, config. aaffig.vccab vicabbaaige biss-Falas)
11
selt. traneforner. 9te. self. head. neight. seight tying
12
13
def forward 18x): self, (1dx)
14
15
all_previous_4 [x]
range config.n_blocke)" intersediary representations
belk.blocke
14

---

18
all.previous_ ax)end(x)
19
self den.gerddd
evary dva_pariod
all, all.previous_ dilated [x. foz 1. osuzerate Soueerste(lll.provvsse..
(1ea(ul). pre.pprrrosss)-1e- celf.config dilation
21
I welf.ddn.nndulllee (al.ppeeooee- ct Computing the veighted average 91th
dilation
22
x self.in_((s)
23
logite = (x)
24
return logits
A.2 More Optimized Pytorch implementation
The point of this more optimized implementation is to remove the for loops of the naive implemen- tation, and instead use tensor operations. To achieve this, need to create tensor containing the previous representations instead of a list. Given the structure of this tensor which will simply accumulate all the past representation during the forward pass, would want pre-allocate an accumulator and update this accumulator with new representation after each block. However, this implies using in in-place operations that conflict with Pytorch's 's autograd. We find workaround by defining the InPlaceSetSlice class. We implement a helper module DWAModules es built on top of this class:
Listing 3: Content of the denseformer python package
import torch
class InFlaceSetSlice (torch (torch. autograd. Function):
oetaticnethod
def forvard(etz,
last_alice, x_idx,
ctx.x_idx x_1dx
ret torch. Tensor () (full, [ix_ddz 13)
11
12
zetera ret
13
14
0esaticnethod
15
def bacavard grad_eut):
17 stx.z_idz
16
Foze, [ctx.x_
18
gua x,112. None, guad_ x,1dx]
22 def sntay_inplacc_eetteo x_1dx, x.val);
23 teliotensor, last_alice 2,ac0
24 nev_xlsce
app 25 return sell_tezsor, mer_elice
last_alice, s_idr, x_val)
26
27
28
DWAModules (torch. Module):
29
..1nit.. (self, a_blocke, dilstion=l, period=1); auper(). (u..rt....
wilf-n_blaske n_blocks
33
dila-iin dilation
34
self peried peried
35
self.
torch. sa, Roduletiia ((tercb((Crrhaaa as Linear (fri-ddiation)//anaa) 1, bisg-False) if (iri))pariod =- alas None for in range (n_bloccs))) self accumulatore Jene
36
37
awlf. aell._.iiit.wiightac)
38
39
def siit__veghht (self):
salf salf allh.a)
module deiaat. 2ero sodule.veight. weiil...eigtt.datn -13
42
43
14
45
def init.eccusniattrssaalll x):
46
x_acce -
47
for i in rangooeoif.diatteee d:latios):
48
current girapp (aulf bllock
self. dtlationnooon:
19
(celf b.blok
current.group.
x_acea. appand Noze))
((eprrent..groo..ata 4x.shaps),
dtype=r.dtype).
52
x_acca [0] apply_inplace_ (x_ 0, self. accumulatore I_aces
53
34
55
def forvard(swif.. t, idock_ids):
56
assert self, accumplators not Bone, "'init accutttaccu (x) needs to be called firet" self. accunulators [(block, idxt1) solr.dilatonn apply_inplace, inplace__cc
37
58
salt accurnaaaaat [(black sulf dilasion),, /1oo//.ixx:))//ool dilation,
39
(block_idx+1) pelfoderiod
torch. Lansordet (awlf.ulphal wlppaaapalle [block_idx). ved)).veight.vieecee.
celt accunulatore bllock.idxt1)eee dilation] (1], dims-1)
64
zetera
15

---

Lenseformer python package. The above module is available in the densef ormer package which can be installed through the following link: https: /tt://gitt /github..... www/eppfll/eessaorraaaaaa
It provides the DWAModules class which orchestrates all the DWA logic given number of blocks, dilation factor, and DWA period. After installing the package, Transformer can turned into DenseFormer in three simple steps:
Listing 4: Faster kxp-Densee Eep-DenseFormer implementation using the denseformer package.
import torch Irom densoforzor import DVAKodules
class BenseFormer (torch. Modolle)..
-_init_.. config):
super (p..rtt...ntt_...
sulf-coafiz config
selt. melff-daa-oduuler DVAModules (oontig.. ha.iiig.g..ooks.... confsg.dilation, contig. dva.period) Step
10
cost self.2te torch. mn Babeddn.igggg config, vocab_ size, config.n_enabe
11
aelf. bhlcck sn.ch..nn.aee Modulai.ia ( (canfig) ( for in range (eondfffngggnbloo n_exbd, biac=contig.bbl blas)
12
LayerBorr
13
self. ln_head torch an. Lineer (config.n, a,eend, config. vocab_aize, biagmFalse)
14
swlf tra.aaaaarrrrr sta.veight self falf.ln_hee head snight
15
16
def Iorvard(au11,, idx):
17
self.ete(1er)
in range(aelf config.n.blocke. Stop
20
selt.blocke [i) (x)
21
sell. welf.dva_aoduleade
block_idx=1) e step
22
== xalf-in_ff in_ff(ff
23
logite celf. self.la_bead(xx
24
ruturn logita
A.3
Hyperparameters and Hardware Used
all our experiments we trained on A100-80GB GPUs. In most of our experiments we trained using pata-parallelism, distributing the batches over up to 4 GPUs.
All of our models share the following hyperparameters:
Max learning rate of 0.001. For different depth, we tested values in {0.0001, 0.0003, 0.0005, 0.0007, 0.001, 0.002] and found 0.001 to be consistently better.
2000 warmup steps, chosen between (1000, 2000}
A weight decay of 0.1
Adam W hyperparameters 31 0.9 and 32 0.95
We use the GPT-2 tokenizer with a vocab size of 50304
12 attention heads of dimensions
An embedding size of 768
16

---

B
Additional Results
B.1
Information Flow
Small weights matter. In the visualized weight matrix of Fig. 4, most of the weights the inter-block connections are small. This observation raises the question as to whether it is possible to drop most of these connections or not. To answer this question, a portion of the smallest DWA weights and report the results in Fig. 6. It can be seen that even though large portion of weights in Fig. are small, dropping beyond 15% of the weights leads to significant increase in perplexity. Therefore, even though these weights are small, they seemingly play important role in predicting the next token.
plot the perplexity after dropping
20.0
19.5
Porplexity
19.0
18.5
DenseFormer
18.0
Transformer
17.5
20
Spansity (% of a dropped)
Figure 6: Performance after dropping small DWA weights. The figure shows the performance when the model is trained with no sparsity induced (dilation and period of 1) and the DWA weights are later sparsified based on their magnitude at inference. One can see that the perplexity quickly explodes after only sparsifying 15% of the weights. This observation suggests that even though many of the DWA weights are small (as can seen Fig. 4) they still play an important role the output of the model.
Visualizing as with periodicity. Similarly to Fig. 4, we show in Fig. the DWA weights for 4x3, 4x4, and For-eesseFormers. We observe patterns similar the Ixl-DenseFormer Fig. 4 but at lower resolution.
10
10-
190% beler
pROA bolox
D00A Prg Aoosknn
100
100
(a) 48-block Formmer
(h) 48-block Re--DenseFormer.
(c) 48-bloock For-mer
Figure 7: Visualization of DWA Learned Weights. Each row shows the weights learned by a DWA module given depth. Those patterns are very consistent with the ones learned by 1x1 ForrDen seen in Fig. 4.
Correlation with input embeddings at 72 blocks. As in Fig. 5, we analyze the cosine similarity between the output of each DWA module and the initial embedding vectors for models of 72 blocks. The results Fig. are very consistent with those obtained with shallower models (Fig. 5).
Evolution of DWA weights during training. In Fig. 9, we plot the DWA weights of 48 block Former during training. We observe how the pattern is learned relatively fast, within the first 5000 iterations.
17

---

0.25
DeuseFFormer
Cosine sitalariy
0.20
Transfomne
0.15
0.10-
0.05
0.00
12
30
36
42
48
60
06
72
Depth
Figure 8: Cosine similarity between the output of each DWA module and the initial embedding vectors. The results are averaged over three seeds, for DenseFormer models with 72 blocks and no dilation. The model initially maintains a high correlation with the output of each DWA modules, but reduces that correlation towards later layers. Intuitively, where the model is preparing to output the next token.
hypothesize that this is the stage
Step
(b) Step 1000 (c) Step 2000 (d) Step 3000 (e) Step 4000 Step 5000
(g) Step 6000
Figure 9: Rapid convergence of DWA weights during training. The DWA weights are rapidly converging their final pattern. After 5000 iterations, the weight pattern already looks very similar to the one in Fig. 4.
B.2 Analysis of Dilation and DWA Period
More detailed analysis of dilation. For 48 block models, we study the impact of varying the dilation factor k, we do not vary the DWA period which is set to 1. The results of this experiment are Fig. 10. We observe how small dilation coefficients do not significantly deteriorate the perplexity yet increase the inference speed.
1.2
DenaeFortss Transfermee
1.1
180
DesueFoeseer (68 Blods)
Deneltemer (12 Blods)
17.
1.0
Dilatinn Factor
Dilatan Factur
(a) Perplexity of 48 block Dex-FeenseFormrr OWT2
(b) Relative speed improvement over dilation
factor 1
Figure 10: Efffect of the Dilation Factor k on Speed and Performance. Part (a) shows the degradation in perplexity as we increase the dilation factor of Foo-Denseoormmrr models. noticeable drop in performance occurs for larger dilation factors, c.g. after & = 4. However, surprisingly, 12-Dilated DenseFormer still outperforms the Transformer baseline. shown in (b), while the perplexity is not so impacted by dilation, the inference speed is significantly improved. Interestingly. the speed gain also plateaus for larger values of e.g. roughly increases with the depth of the DenseFormer, and the plateau threshold occurs later for deeper models.
for 48 blocks. The gain
More detailed analysis of the DWA period. For 48 block models, we study the impact of varying DWA period p. We not vary dilation which is set In Fig. 11, we observe the impact of increasing p on the perplexity. Interestingly, the perplexity profile is non-monotonic in p. which exposes the interplay between k p. and the depth of the model. Moreover, increasing the DWA period further increases the inference speed over increasing the dilation.
18

---

185
1.2
15.4
DrreeFonu 4xp
13.2
2.1
1500
17.5
10
12WA perted p
19A1 perted p
(a) Perplexity of block Fop-Derseeormer on OWT2
(b) Relative speed improvement over 1x1- DenseFormer
Figure 11: Effeet of the DWA period p on Speed and Performance. Part (a) shows the degradation in perplexity as we increase the DWA period of Axp-DenseeFormer models. Surprisingly, Foreerrrs still outperforms the Transformer baseline. As shown (b), while the perplexity is not so impacted, the inference speed is significantly improved.
B.3 PG-19 Experiments
PG-19 experiments. Tab. shows the performance of DenseFormer against standard Transformer on PG-19, which consists of a large collection of full-length books from Project Gutenberg [24]. We trained both architectures for 48k steps and used a batch size of 128 instead of 400. All the other training parameters are kept the same for OWT2 experiments. On this dataset, we can clearly see the superior performance of DenseFormer.
Table 4: Comparison on PG-19. Comparing DenseFormers and Transformers on the PG19 dataset. The results show similar improvements as the ones observed on the Open WebTex2 dataset. This demonstrates the generality of our results. Those results were obtained using batch size of 128.
Model
Depth
Perplexity
Transformer
24
20.13
Formmer
24
19.60
Transformer
48
18,94
48
18.43
Transformer
72
18.44
B.4 Delaying the Training DWA Weights
In this section, we study what would happen if we started training the DWA weights at different training iterations. As seen in Fig. 9, the DWA weights are rapidly converging their final values within the first 5000 iterations. Moreover, the initialization of the DWA weights corresponds to the same flow of information as normal transformer. This raises the question of whether training the DWA weights during the first training iterations is important, or whether a pre-trained model would still gain from adding the DWA weights later. To answer this question training the DWA-weights after N iterations. We do not modify the learning rate scheduler or any hyperparameter besides N. Results Tab. show a diminishing return as N increases. seems important to tune the DWA weights from the beginning. A possible hypothesis could be that the iterates commit to a valley in the loss landscape relatively early during training. Once deciding to go to the valley where DWA weights are not used, it is difficult to recover and ultimately benefit from newly added DWA weights. We believe this phenomenon could mitigated using better learning rate scheduler. We leave this investigation as future work.
experiment with
B.5 Rank Analysis
In this section, compare the ranks of matrices learned using DenseFormer and Transformer architectures. Our main result in Fig. 12 is that there no significant difference rank between the two approaches.
B.6 Experiments with batch size of 128
In this section, we revisit experiments from the main paper but use small batch size of 128 instead of 400 during training.
19

---

Table 5: Start training the DWA weights after N iterations. At initialization, DenseFormer the same as a Transformer. We experiment with tuning the DWA weights only after N iterations. This means the model is trained as a Transformer for N iterations, and as DenseFormer from N 40k iterations.
Model
N
Perplexity
Bascline Transformer
18.61 (0.02)
FooeeerseForrmer
0
17.87 (0.02)
Formerr
1k
17.99
4x5 Ferrrmr
2k
18.07
3x5--DeneFFormn
4k
18.13
3.55DDnscFormet
6k
18.17
De-neeForrrrr
10K
18.23
Forsmmrrrrr
20k
18.33
Formerr
30k
18.40
10
Tranforuu
Transfomer DeneeForau
Taundorner
DinoeFocteer
Ranked singulaz
singular
(a) QKV attn matrix
(b) Output attn matrix
(c) MLP up proj.
Transonee Denaaereems
Transfemmr
DonerFemmer
Tatradte
Iatanstt
5.00
2.5
2000
800
250
500
750
Ronked
valure
Banked singguie valuos
(d) MLP down proj-
Emheddings
Figure 12: Ranked singular values averaged across blocks. For 48 block models, we average the singular values for each matrix across blocks (except for the embedding matrix). We observe no significant differences between Transformers and DenseFormers. Results are averaged over 3 seeds.
Speed and performance trade-off. In Fig. 13 we show the trade-off between inference speed and perplexity for different numbers of blocks. Similarly Fig. 2b, DenseFormers reach better perplexity than much deeper Transformer models. Interestingly, the perplexity gap is larger than when using larger batches (compared to batch size of 400 used in Fig. 2b). A 48 block DenseFormer is performing on par with 90 block Transformer. This might indicate that the DenseFormer is more robust large gradient noise compared Transformers. DenseFormers reach better trade-off's in terms of inference speed and perplexity. Those results are expected to improve if we were to train Lx5-DenseFormer instead of Jxl-DenseFFormr. Detailed results can be seen in Tab. 6.
Results with other sparse patterns. In Tab. 7 we reproduce the experiments of Tab. 3 but with batch size of 128. Similar conclusions follow.
Visualizing the DWA weights (trained with a batch size of 128). In Fig. 14 we plot the DWA weights obtained when training with batch size of 128. The learned patterns very consistent with the ones of Fig. obtained with a larger batch size.
20

---

DemseRomact Trunsbormet
Puplecky
23.0
50
22.5
45
22.0
Balches Second
Figure 13: Speed and performance trade-off. Comparison of speed and performance trade-off between the standard Transformer architecture and 4x1-DenseFormer. The number of blocks in each architecture is reported next the data-point. All DenseFormer models on this plot use dilation factor of 4. Comparing perplexities: Considering only the perplexity (y-axis), 48 layer DenseFormer strikingly outperforms much deeper Transformer baselines. Comparing trade-offs: 48 layer 4-Dilated DenseFormer matches the better perplexity of 90 layer Transformer while being 1.6x faster at inference.
Table 6: Performance of DenseFormer and the standard architecture of different sizes OpenWeeTexxt2 dataset. Using a batch size of 128, and DWA period 1. DenseFormer clearly outperforms standard architecture of the same depth as well as standard architecture with the same inference speed. While sometimes deeper model with the standard architecture can match the performance of shallower DenseFormer, inference using the shallow DenseFormer remains much faster.
Model
Dilation
Depth
Parameters # (M)
Perplexity
Inference BPS
Transformer
48
378.45
23.67 (0.09)
5.98 (0.00)
Skips With Gains
48
378.45
23.78 (0.19)
5.72 (0.01)
48
378.45
22.61 (0.05)
4.67 (0.00)
DenseFormer
48
378.45
22.60 (0.04)
5.15 (0.01)
378.45
22.68 (0.06)
5.36 (0.00)
64
491.72
23.21 (0.07)
4.59 (0.00)
Transformer
72
548.35
23.10 (0.02)
4.12 (0.00)
548.36
21.81 (0.00)
2.93 (0.00)
DenseFormer
72
548.35
21.92 (0.04)
3.39 (0.00)
72
548.35
22.03 (0.04)
3.62 (0.00)
84
633.31
22.84 (0.07)
3.56 (0.00)
Transformer
90
675.78
22.67 (0.04)
3.35 (0.00)
Table Alternative DWA Sparsity Patterns. We compare 48 block architectures with different sparsity patterns. In Last K. DWA can only access the output of the last & blocks well as the embedding vectors. Connect Last includes only a single DWA module in the architecture placed after the last layer, i.e. connecting every block to the last block. Neither of these patterns allows achieving the same perplexity boost as the original DenseFormer. Even with a dilation of 12, which implies sparsity of 92%, the Denseformer models outperform other sparsity patterns, which signifies the importance of pairwise inter-block connections.
Sparsity Pattern
Perplexity
Sparsity
Baseline Transformer
23.67 (0.09)
Doxl-DenseFormer
22.91 (0.06)
92%
Last K (K 4)
23.23 (0.07)
84%
Connect Last
23.45 (0.05)
96%
21

---

30
10
10
DWA Exlex
D0S Budoe
30
(a) 48 block DenseFormer.
(b) 72 block DenseFormer.
(c) 48-block For-eerseFormer.
Figure 14: Visualization of DWA Learned Weights. Each row shows the weights learned by DWA module at a given depth. While the heatmaps are averaged across runs with different seeds, those patterns are very consistent across seeds. In (a) and (b), strikingly similar patterns can be observed in both 48 and 72 layer DenseFormers. DenseFormer trained with dilation of 4. Despite the sparsity, still observe very similar pattern to those learned by the non-dilated models.
(c), we show the learned weights for a 48 block
22

---

B.7 Comparison with ELC-BERT
In this section we compare work with the concurrent work of Charpentier & Samuel [2]. We implement the different variants they suggest and show the results we obtain Fig. 15. We observe that while their architecture is more data efficient in the low data regime, DenseFormers catch-up quickly (after about 1.2k iterations). In the non low data regime, ELC-Bert performs similarly as the baseline, while DenseFormer is consistently better.
30
Transformer
Transformer
ELC Bert Skips
250
Bert
28
ELC Bert Skips O-Init
ELC Bert Skips 0-Init DenseFormer
Denseformer
Validation (Subset) Perplexity
ELC Bert
Validation (Subset) Perplexity
ELC Bert
200
24
150
22
100
18
0
5
20
25
30
35
600
700
800
900
1000
1100
1200
1300 1400
Steps
Steps
(a) ELC-BERT vs. DenseFormer
(b) ELC-BERT vs. DenseFormer in low data regime
Figure 15: Comparison with ELC-Bert. Part (a) Shows that ELC-Bert does not improve upon the baseline, while DenseFormer is consistently better. In (b), we observe that ELC-Bert is better low data-regime, which the setting for which it was developed. Yet the DenseFormer is catching up fast and outperforms
after approx. 1 1.2k iterations.
B.8 Effect of varying the width
Information capacity bottleneck. We can postulate that the larger the width of our model, the easier it is to pass information sequentially from layer to layer. Therefore, the wider the network, the less we expect to gain from adding inter-layer residual connections. We experiment with varying the width of 24 block models, using the following values: (384, 768, 1536]. Results are summarized Table We also train a transformer model with a larger hidden size of 1664. For the larger widths of 1536 and 1664, we tuned the learning rate {0.001, 0.0005, 0.0003) and found 0.0005 to be best. use batch size of 128, learning rate of 0.001, sequence length of 256, and train for 40k iterations. We observe-as as expected- the gap becomes smaller for wider models, yet does not disappear. A DenscFormer with smaller hidden size of 1536 performs better than Transformer with a hidden size of 1664. general, as the complexity of the task increases, followed by the need for more capacity, it is more compute and memory efficient to DenseFormer instead of Transformer. This is especially relevant when we think of deploying LLM on devices with hardware constraints.
Table 8: Effect of varying the width.
Width
Transformer PPL
DenseFormer PPL
Parameters
384
31.039
30.120
62M
768
25,020
24.631
209M
1536
21.597
21.279
757M
1664
21.313
881M
23

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The superior performances of DenseFormer are reflected in Fig. 2b and Fig. 3. Those figures show that (i) during training, DenseFormers are reaching a lower perplexity faster than a transformer baseline, as well as (ii) during inference, DenseFormer provide better perplexity for given inference time budget. See 4 for more detailed results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Addressing the limitations of the proposed architecture is an important part of our work. In 3, discuss the impact on the resources, which tackle through introducing sparsity in the DWA weight matrix. Finally, in the conclusion, we mention how future works should focus understanding how this architecture can be efficiently distributed in the context of large scale training. Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This work is empirical in nature.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: addition to providing the source code, we give great details on how to implement approach in App. A. The method is detailed at length in the main paper, $ 3.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
25

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
We provide the code as well as the general command line to run the experiments. One of the datasets, we used: Open Weenee, publicly available. Unfortunately this is no longer the case. Given that someone has access to this data, they can use our code for processing it.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/ cablic/guides// (CCooeeSaiissioonn.... moyyccy)) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/ http:/pullgggguiieee (odee///oomdmmiii ss wsisnnooll onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The hyperparameters for training discussed in the main paper in 4 as well as in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars are always given, results averaged across three seeds.
Guidelines:
26

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe the hardware used in App. A, as well as discuss the computational time required extensively in the main paper (see. 4).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attps:/ http///eerripp.eeeee cctppblic//ettcccccee httplaice/hhccscccieleieeeee
Answer: [Yes]
Justification:
No special consideration.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
27

---

Justification: Our work proposes a generic replacement for the Transformer architecture. While we focus on language modeling, it does not target any specific domain of application. therefore there is no direct path to negative application.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite all datasets and libraries used in our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
28

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [No]
Justification: We share source code and guidelines to reproduce our experiments, but we do not share any other asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
29

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "omjBrbOzcp", "reviewer": "Reviewer_Ljsg", "summary": "The paper proposes to construct current transformer block input by weighted average of all inputs from previous transformer blocks. The weights are static and learned during training process. To reduce computation complexity, the method comes with a dialated version controlled by modulo and division relations. Experiments are conducted to observe the changes in model size, train/test speed, PPL. Models are trained from scratch on OpenWebText2. Results show better PPL, or similar PPL with smaller/faster model.", "questions": "1. the abstract is a little misleading, L5 says \"100B parameters\", but experimetal models are at the level of millions of parameters. \n2. the proposed method should consider a generalization into triangular/pyramid structure. what happens if there is DWA on top of DWA and again on top of DWA? \n3. the sparsity design by modulo and division is rule based (with no intuition on hyperparameter setting), not learned from training. is it possible a simple regularization can be much better?\n4. the whole design is meant for traning models from scratch. it becomes not useful given many well trained transformers are released.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "the method is well defined, and experimental results are good", "weaknesses": "not a rigorous study regarding experiment comparison with related work, dialation mechanism design", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our work. We hereby answer the reviewer’s questions in order:\n\n1. The number of additional parameters required by DenseFormer increases quadratically with the model’s depth. We mentioned 100B parameter models as those large scale models are typically the deepest and represent a worst case scenario. This serves to emphasize that even in this worst case, the number of added parameters is negligible. \n2. If we understand correctly, it is suggested to add DWA on top of other DWAs. Given that the current DenseFormer architecture is already mapping any block output to all future block inputs, adding additional connections on top would not make the model more expressive. Please let us know if we misunderstood your statement.\n3. We opt for rule based sparsity for simplicity and to maintain hardware compatibility. Other types of sparsity do not easily yield speedups or memory savings and require modifying the loss, possibly introducing additional training overheads. Furthermore, as can be seen in Fig 3.a., the rule based sparsity is already doing quite well, allowing 4x5 DenseFormers to achieve a very close performance to 1x1 DenseFormers. The remaining gap seems too small to justify deploying adaptive sparsity techniques. \n4. The reviewer mentions it is not relevant to do research on better transformer training given open-source models are being released. We strongly disagree with the reviewer on this statement. Just because pre-trained LLMs are open sourced does not mean it is not worth researching more efficient ways to train LLMs. The field is extremely fast-paced and new models are continuously trained and released. Our research---by making models more parameter efficient---increases the expected return from pre-training models, and therefore promotes the training of more models.\n\nWe hope we have answered your concerns, in which case we would appreciate it if you could raise your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qSOV5tKUiE", "reviewer": "Reviewer_VTZd", "summary": "The paper introduced DenseFormer -- a simple and effective architecture that boost transformer language model's performance by adding trainable residual connections to all the previous layers. The paper discusses intuitions behind the architecture (that it enhances information flow between earlier and latter layers) and design choices for quality-cost tradeoff. Empirical results on the language modeling tasks validates the effectiveness of the architecture. The paper also presents empirical observations of the residual weights which exhibit consistent patterns that might be tied to the effectiveness of the architecture.", "questions": "Regarding the memory consumption of DenseFormer, the paper mentioned that it has \"Negligible Memory Overhead\" since the previous layer's output are stored in the KV-cache. This seems to be at odd with the most common practice, in which the actual key and value vectors are stored in the KV-cache, not the layers' outputs. The reviewer assumes that the authors are referring to a difference way of caching -- caching the layers' output and re-compute key, value projections at inference time, which will be consistent with the negligible memory overhead claim, but introduces additional computation. A clarification on this point would be helpful.", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Originality: while the idea of residual connection or trainable residual connection have been proposed, it is still surprising that a simple trainable combination of residual streams can be both effective and has consistent weight patterns. The design of this architecture and the findings of the effectiveness and weight pattern are novel.\n\nClarity: the paper presents the idea in a clear and easy-to-follow fashion. Various design choices are discussed regarding quality-cost trade-off; and important issues such as whether the architecture remains effective if introduced in latter training stages are discussed.\n\nSignificance: the simplicity and effectiveness of the method is quite appealing for consideration in practical language models.", "weaknesses": "A minor weakness is the robustness of the architecture across tasks: the paper focuses on the language modeling task, while transformers have been used for vision, speech and general time series data. A discussion on the robustness of Denseformer on other tasks would strength the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your consideration of our work and your valuable feedback. We make the following comments to provide further clarifications:\n\n1. We completely agree on the ambiguity regarding the negligible memory overhead claim and thank you for bringing it to our attention. We point out that at the decoding stage during inference, we do not need to store the intermediate outputs for already processed tokens. The overhead of storing the intermediate outputs for a single token remains negligible in comparison with the KV cache. However, a more careful implementation is needed to avoid additional memory usage during the pre-fill stage as you suggested. We will clarify this in the revision of our paper.\n2. While we expect our results to extend to other tasks as well, we decided to focus on text for this work to save on resources. We agree that extending the results to other tasks would be an excellent direction for future work. \n\nThank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SKNEZI1aIg", "reviewer": "Reviewer_1YUF", "summary": "The work introduces DenseFormer, a variant of Transformer architecture using special connections between blocks. Each transformer block in DenseFormer may look at a weighted average of all previous blocks' outputs (instead of the simple output of the previous block). Authors run extensive experiments showing the performance improvement of such a design and show that this simple change beats the baseline model on the Pareto-frontier of model quality and speed. Authors also run good ablations and showcase different variants of DenseFormer.", "questions": "See weaknesses. The primary concern is the architecture shape, and the secondary is hyperparameter tuning. With this primary issue resolved, this would be outstanding work.\n\nAnother question is, can we get a more detailed analysis of how exactly the technique differs from previous work, apart from being applied to Transformers? My understanding is that it has multiple novel parts that were adapted specifically, but more detailed comparison would be welcome.", "limitations": "Properly addressed, except for the architecture shape (see weaknesses).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. **The idea.** The idea behind the work is simple, in a very positive way, with a good story. The implementation is straightforward (even if optimizations are useful), the methods aren't costly during training, and what costs it incurs it makes up for in performance. Moreover, I believe the technique introduced could not only improve Transformer-based language models but Transformers in different modalities as well, and even other kinds of residual networks apart from Transformer - not only \"classical\" CNN-based ResNets, but also new architectures like Mamba.\n2. **Experimental setup.** Experiments are extensive, and results show consistent improvement. Every metric that I'd want to see is present and measured, like inference throughput, training speed, and comparison to models equal in inference/training time (both through an increase in size and through an increase in training iterations).\n3. **Improvements on Pareto-frontier.** The significant Improvements achieved with DenseFormer compared to the \"vanilla\" Transformer should interest both the research community and industry applications.\n\nWhat is more, **the authors provide excellent, easy-to-understand code**, so even when I had doubts about how things were implemented, I could easily check the source code; thank you so much.", "weaknesses": "**Non-standard model shape (issue essentially resolved during the rebuttal).** By a model shape, I mean the relative dmodel and nlayers. I am extremely worried that results would hold up on standard model shapes. The work uses a dmodel of 768, with the smallest models being 48 layers deep and the biggest model being 90 layers. This is extremely deep and thin.\n\nFor reference I will use GPT3 model shapes, as they are reasonably tuned and still used by the community today ( see Table 2.1 on page 8 of https://arxiv.org/pdf/2005.14165 ). The only model with a dmodel of 768 has just 12 layers, 4x shallower than **the shallowest** model used by the authors. This ratio 768/12 was also used by BERT-base. The common ratio used by the community, shared by GPT3 6.7B, llama7B, Mistral, is 4096 dmodel with 32 layers - still shallower than any models used by the authors, and 6x as wide! Chinchilla 80B and LLama 80B use 8192/80, with depth comparable to the authors' models. Only when we get to \"the GPT3\", 175B, we finally get to the model deeper than any of the authors' models, but with a whopping 12288 dmodel.\n\nThe above issue concerns me because one possible interpretation of DenseFormer improvements is essentially increasing the information throughput/capacity of the residual stream, as the layers can communicate directly without using the limited residual stream. With those hyperparameters artificially bottlenecking the throughput of the baseline model's residual stream (compared to the optimal hyperparameters), every technique that effectively circumvents the bottleneck gains an unfair advantage, and the results may not reflect the improvement in the real-world models.\n\nI am unsure how big of an impact this nonstandard architecture has (maybe it doesn't have any impact?), but it leads me to trust the results less. While I trust that running wider models isn't feasible with computational constraints, I would suggest the authors run models of \"standard\" shape, especially when it comes to the baselines. For example, seeing the results of a smaller model, like 768/12 (and/or less than 12 layers while keeping the dmodel of 768), would showcase the impact of changing the width/depth ratio of the model on the relative improvement achieved by DenseFormer. Those experiments would also be relatively cheap (I'm not suggesting authors pre-train 7B models, of course).\n\nThe saving grace for now is that the code exists and everything is easy to reproduce by the community, so I'm leaning accept even with this issue standing. However, running those experiments would be perfect.\n\n**Learning rate, hyperparameter tuning.** Those are less important than the depth/width ratio, but still important. Learning rate (and other parameters, if tuned). How was it chosen? Was it tuned for the baseline? Was it tuned at all? This, again, is important - especially for those deep models, which are generally harder to train and less stable.\n\n**Minor**\n1. Number of heads. In line 198, \"We use models with 8 heads\", while in line 602, \"12 attention heads\". One of those lines is incorrect (I assume 12 heads is used, though).\n2. I needed to see the code to see the details about the residual connection inside the block and where the layer norm is (is Pre-LN used, in particular). Adding a \"Block\" implementation to the paper's naive implementation code would be nice to make it easier for future readers to check this. That said, the code is very nicely written.\n3. It would be worth adding that the memory overhead (line 130) is only negligible if no activation checkpointing is used (and it is commonly used). Also, again, for the inference, the overhead is negligible if no optimizations are used to shrink the KV cache (and again, things like grouped attention are commonly used).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your careful consideration of our work and your valuable feedback. We provide the following comments to further clarify our contributions:\n\n\n1. **On the non-standard model shapes.** As you have correctly pointed out, using DenseFormer “alleviates the information capacity bottleneck of the residual stream”. It is also true that increasing the hidden dimension is another possibility for resolving this bottleneck. Indeed, we conjecture that given a DenseFormer it is possible to build a standard model with a much larger hidden dimension and show that it can theoretically perform the same operations as the original DenseFormer. As such, it is possible that the gap becomes less pronounced as we increase the width. However, increasing the hidden dimension comes with significant overheads on the resources, leading to larger models and additional memory footprint (e.g. larger KV cache). To further strengthen our claims, we additionally ran experiments with a $24$ layer DenseFormer and compared it at different widths of $384$, $768$, and $1536$ with a standard Transformer. The perplexities for those models are presented in the following table:\n\n    | Width      | Transformer | DenseFormer | Parameters  |\n    | -----------| ----------- | ----------- | ----------- |\n    | 384        |    31.039  |    **30.120**   |    62M      |\n    | 768        |    25.020   |    **24.631**   |    209M     |\n    | 1536       |    21.597   |    **21.279**   |    757M     |\n    | 1664       |    21.313   |       -     |    881M     |\n\n    For the larger widths of $1536$ and $1664$, we tuned the learning rate in $\\{0.001,0.0005,0.0003\\}$ and found $0.0005$ to be best. We can see that the gap between the two architectures still persists. We also validate the above conjecture by measuring the perplexity of a transformer model with a larger hidden size of $1664$, which is worse than a DenseFormer with a smaller hidden size of $1536$. This, despite a $124$M parameters difference. Furthermore, given the above discussion regarding the trade-off between capacity, width, and using DenseFormer, we expect the gap to widen when moving to more demanding settings which require additional capacity (e.g. longer sequence lengths, or training on more tokens). \n2. Moreover, while current architectures are using larger hidden dimensions, this can be because until now, increasing the hidden dimension has been the only avenue to address the said information bottleneck. In our work, we point out that using DWA additionally allows operations such as subtracting the first layer’s output which we have demonstrated is actually learned by the model.\n3. **On the learning rate and hyperparameters tuning.** We tried our best to tune the hyperparameters for the baseline models at different model depths given the available resources. We then used the same settings when training DenseFormers. We tested learning rate values in $\\{0.0001,0.0003,0.0005,0.0007,0.001,0.002\\}$, and found $0.001$ to be systematically better. We also tuned the number of warmup steps for a $48$-layer standard Transformer, and used the best values found in all subsequent experiments. We will include these clarifications in our next revision. \n4. **On the memory overhead.** We agree that when activation checkpointing is used, the overhead might become more noticeable at training and thank the reviewer for pointing this out. We mention that at the decoding stage during inference, we do not need to store the intermediate outputs for already processed tokens. The overhead of storing the intermediate outputs for a single token remains negligible in comparison with the KV cache.  We will include these clarifications in the next revision of our paper.\n5. **On the difference with relevant prior works.** The most relevant prior method might be DenseNet. DenseNet is composed of a succession of Dense blocks. Within each block, convolution operations are interleaved with concatenation operations, which concatenate the output of all the previous convolution blocks. Therefore, within a dense block, the size of the representation is increasing. The hidden size is reduced using transition blocks consisting of convolution and pooling. Compared to DenseNet, our approach is minimalist: we simply do a linear combination of past layer’s outputs. This removes the need for transition blocks, only adding a very small number of parameters. We will include this discussion in the related work section of our next revision and try making the distinction with prior work more clear.\n\nWe hope the above comments address your concerns and that you would consider raising your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C99J55hFKR", "reviewer": "Reviewer_EGgo", "summary": "The authors propose to use a weighted average of previous layer outputs as an input for the successive layers. The weights are learned parameters and can also take negative values. The weighting module can be coarse: it is enough to insert it only in some layers, and it is enough to attend to a subset of previous layers, which significantly lowers the overhead resulting from the weighting. The authors show perplexity gains on language modelling on OpenWebText2. They analyze the learned weights, and uncover interesting patterns.", "questions": "In line 133, the authors write, \"the outputs are stored in KV cache to facilitate decoding.\". Depending on the implementation, this is not what is typically done. To avoid recomputation, the K and V values are stored instead of the residual. However, I agree with the authors that this is not an issue since only the state of a single column has to be stored, which is typically negligible.", "limitations": "Already discussed in the weaknesses section. The authors should add some discussion on the limitations of the evaluation.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The method is easy to use, improves perplexity, and shows interesting and consistent weight patterns. The overhead is minimal, less than for depth-wise attention, and the perplexity in a large data regime is better than that of the concurrent DenseNets.", "weaknesses": "The evaluation could be stronger: It would be nice to evaluate the zero-shot performance on some downstream tasks (e.g., BLiMP, CBT, PIQA). This should be easy to implement and does not require additional training.\n\nThe evaluated transformers are narrow ($d_{model}=768$) and very deep (48 layers for the 378 and 72 for the 548M parameter models). It is unclear how the gains would transfer to a more traditional setup (e.g a 350M GPT-3 has 24 layers). The effects might be smaller with fewer layers if the main role of the DWA module is to provide additional shortcuts for the gradients. The unusually narrow $d_{model}$ can provide an additional advantage for DWA because there is less space to store everything in the residual directly, and the shortcuts can help with that. There are two ways to improve this: either show the gains with a more standard aspect ratio or show that the narrow+deep DWA model is better than a more traditionally shaped DWA model (both of these should be paramer-matched).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our work.\n\n1. **On the KV cache comment (line 133).** We thank the reviewer for bringing this to our attention and agree the statement line 133 is incomplete. As the reviewer rightfully pointed out, while we do not need to store the past layers’ outputs, a small price has to be paid to store the intermediary states for the current token. This cost is small and does not grow with the sequence length. We will make this clear in our next revision. \n2. **On the width used in our experiments.** We agree that our DenseFormer architecture helps to solve the capacity bottleneck of the residual stream typically present in transformers. As such, a narrow DenseFormer can match the performances of a wider Transformer. This is an important advantage as increasing the width also increases the training and inference costs in terms of both compute and memory. To investigate the impact of the model’s width on the perplexity, we trained multiple $24$ layer models with different widths of $384$, $768$, and $1536$. The results are summarized in the following table:\n\n    | Width      | Transformer | DenseFormer | Parameters  |\n    | -----------| ----------- | ----------- | ----------- |\n    | 384        |    31.039  |    **30.120**   |    62M      |\n    | 768        |    25.020   |    **24.631**   |    209M     |\n    | 1536       |    21.597   |    **21.279**   |    757M     |\n    | 1664       |    21.313   |       -     |    881M     |\n\n    We observe that the gap—while smaller for wider models—does not disappear. Moreover, a $1536$ wide DenseFormer performs better than a $1664$ wide Transformer, despite having $124$M fewer parameters. In general, as the complexity of the task increases, followed by the need for more capacity, it is more compute and memory efficient to use a DenseFormer instead of a Transformer. This is especially relevant when we think of deploying LLM on devices with hardware constraints.   \n\n\n    Additionally, other hypotheses exist that could explain the advantage of DenseFormers over Transformers. For instance, we mention in our work how the DWA weights can learn to subtract earlier representations from later representations, which potentially helps to disentangle between processing the current token, and predicting the next token. Evidence for this can be seen in Fig.4 and Fig.5.\n\nWe hope the above comments address your concerns and that you will consider raising your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Given the overall positive reviews, strong empirical results, and the authors' thorough responses to the reviewers' concerns, I recommend accepting this paper. DenseFormer presents a simple yet effective modification to the Transformer architecture, demonstrating its potential to improve the performance and efficiency of various models.
Decision: Accept (poster)