From Overconnectivity to Sparsity: Emulating Synaptic Pruning with Long Connections

Authors: Vaggelis Dorovatas, Georgios Paraskevopoulos, Alexandros Potamianos
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: machine learning architectures, sparsity, residual connections, redundancy, long connections, pruning, synaptic pruning

Abstract:
During brain development, an excess number of synapses are initially created, which are progressively eliminated through a process known as synaptic pruning. This procedure is activity-dependent, shaped by the brain's experiences. While creating an overabundance of synaptic connections only to later remove many might appear inefficient, research suggests that pruned networks demonstrate significant efficiency and robustness. Inspired by this biological process, we propose a neural network architecture utilizing long connections instead of traditional short residual connections. When long connections neural networks (LCNs) are trained with gradient descent, information is naturally "pushed" down to the first few layers, leading to a sparse network. Even more surprising is that this simple architectural modification leads to networks that exhibit behaviors similar to biological brain networks, namely: early overconnectivity to later sparsity,
enhanced robustness to noise, efficiency in low-data settings and longer training times. Specifically, starting with a traditional neural network architecture with initial depth $d$ and $k$ connections, long connections are added from all layers to the last layer and summed up. During LCN training, 30-80% of the top layers become effective identity mappings as all relevant information is concentrated in the bottom layers. Pruning the top layers results in a refined network with a reduced depth $d'$ and final connections $k'$, achieving significant efficiencies without any loss in performance compared to residual baselines. We apply this architecture to various classification tasks and show that, in all experiments, the network converges to utilizing only a subset of the initially defined pre-training connections, and the amount of compression is dependent on the task complexity.

Decision: reject
Reviewer scores: 6, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1729860329902

thread_type:
official_review

<thread_object>
creator: reviewer_fzkp

<content>
summary:
The paper explores a new neural network architecture inspired by biological brain processes, specifically synaptic pruning. The paper proposes Long-Connected Neural Networks (LCNs) as an alternative to traditional deep learning architectures that primarily use short residual connections. 
This simple architectural modification leads to networks that exhibit behaviors similar to biological brain networks, namely: early overconnectivity to later sparsity, enhanced robustness to noise, efficiency in low-data settings and longer training times.

strengths:
- Good Writing. The paper clearly explains the motivation for the proposed work, drawing a strong connection between biological processes and deep learning archutectures.
- The paper proposes a novel architecture. By pushing information to the lower layers and pruning upper layers, the architecture addresses the common issue of overparameterization in deep learning models. It is interesting.
- The paper demonstrates that LCNs lead to sparser networks and improved efficiency. This can be particularly valuable in real-world applications where computational resources and memory are limited.

weaknesses:
- The paper provides some mathematical intuition, but a deeper theoretical analysis of why this architecture works so well (e.g., in terms of information bottleneck theory or gradient dynamics) could strengthen its claims.
- Across the experiments, I cannot find the comparative experiments with state-of-the-art pruning techniques.

questions:
- Please explain why your method can have effect.
- Please compare with some state-of-the-art pruning techniques.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730587375274

thread_type:
official_review

<thread_object>
creator: reviewer_zpq5

<content>
summary:
The authors propose a method for encouraging more task-informative representations in the lower layers of ANNs and thereby improving efficiency.

strengths:
- the text is generally well written

weaknesses:
- I did not feel that the comparison to similar models was adequately made. There is a related work section which many relevant models are correctly brought up, but 1. there is no explicit description of how LCNs are different from these models and 2. apart from residual connections, none of these models are quantitatively compared to this previous work. One is left to ponder whether LCNs makes conceptual/quantitative *advances* in the field. 

- the link to biology and the brain is interesting, but after the initial abstract it is hardly addressed. The authors report longer training times as more akin to 'biological brain networks', but I'm not sure which result even supports this (in Fig 8 it trains faster?). That LCNs handle noise better is interesting, but an intuitive explanation/link to previous literature as to why this occurs would be very helpful in its interpretation. 

- It appears from the figures that all results are derived using a single random initilisation seed? If so, I would strongly recommend re-running these analyses over several seeds, otherwise it is very difficult to draw any strong conclusions. 

- Some of the segways between sections could be smoother. For example short introductions to sections 3.1/7 would help. Some figures could also be clearer. For example a few instances where the axis labels were confusing. E.g. what are the axis labels in Fig 2? I presume w and the error gradient with respect to w? in fig 3 is weight of hidden1 = w1, and wouldn't it be interesting to look at w2/w3 as well to check if they are sparser for LCN, and maybe absolute values instead of raw values to better show sparsity? In Fig 7 I would prefer an explicit mathematical term relating to an equation than 'std' and 'noise_level'. In section 3.1 a schematic (even if for the appendix) would be helpful to the reader

questions:
- line 54: 'Contemporary deep learning architectures aim for overparameterization' - what does this mean?
- is the residual network given by equation (3) a specific type of residual network? It is introduced there rather abruptly for a non-expert reader
- for the toy example the models are trained for 500 epochs (1000 examples per epoch). This seems like a lot for such a simple problem?
- line 259: 'possible indication of overfitting' - is Fig 5 showing the training or validation accuracy?

- typos:
line 210: brackets around citation
line 326: -> architectures
line 411: -> directly
paragraph 460: brackets around (multiple) citations
line 492: -> effectively
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648978439

thread_type:
official_review

<thread_object>
creator: reviewer_qqay

<content>
summary:
The authors add long-range connections to standard transformers and show that this enables effective pruning, perhaps like that in the real brain.

strengths:
This is an interesting and creative idea.  It is also true that it is supported by the neuroscience literature is a reasonable possibility.

weaknesses:
The evaluations are too simple.  Like many things in machine learning, something that works for simple versions of tasks might break down for larger tasks.   I am most knowledgeable about vision.  I would like to see similar results for imagenet rather than just CIFAR-10 to be really sure that these results are actually going to be impactful.  (Apologies if I missed it if this was included.) I am not competent to juge the meaningfulness of the language results.

questions:
Can you illustrate results on a stronger task like imagenet? Or perhaps CIFAR-100?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711875843

thread_type:
official_review

<thread_object>
creator: reviewer_rknz

<content>
summary:
Inspired by brain synaptic pruning, they proposed long connection neural networks that eliminate top layers without sacrificing performance, resulting in a significant reduction in inference time.

strengths:
1. This work accelerates inference and reduces memory usage without sacrificing performance.
2. This architecture can be used complementarily with pruning or dynamic inference techniques.

weaknesses:
1. The validity of the structure is verified only on the base ViT, but there is a lack of experimental and theoretical validation for larger model structures.
2. If the author claims that this is a fine-tuning method, the time spent during model training is typically included in the overall time assessment. Fine-tuning often involves adjusting a pre-trained model on a specific dataset, and this process can require significant computational resources and time. However, if the focus is on inference speed or deployment efficiency, the discussion (time, GPU memory and other computation cost) might be reported during the fine-tune process .
3. It sounds like you' re noting that the determination of the final model layer relies heavily on experimental validation, which can indeed be resource-intensive and time-consuming. This approach can lead to significant computational costs, especially when iterating through different configurations to find the optimal structure.
4. While verifying the effectiveness of a structural design on a classification task provides important insights, it doesn't necessarily guarantee that the same design will perform well across different tasks, such as object detection, segmentation, or generative modeling. To better justify the generalization to other tasks, the authors could provide theoretical justifications or include empirical evidence.

I am glad to raise my score if the authors can address the above points.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732584316952

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the Reviewers for their rigorous and productive feedback, which has substantially improved our manuscript. We have addressed their recommendations and outlined both the major changes and our responses to individual comments. Below, we detail the specific improvements made to the manuscript's technical content and presentation:

-  Included in Appendix G a more comprehensive theoretical analysis of the training dynamics of LCN, as requested by R1 (RKnz), R2 (Qqay) and R4 (Fzkp). The analysis provides an explanation for the “layerwise-training” behavior observed in the LCN experiments and why this behavior generalizes to different tasks and training regimes. 

-  To further demonstrate the general applicability of the proposed LCN architecture we added a pretraining experiment on BERT as suggested by R1 (RKnz) and implied by R3 (Zpq5) (Appendix F).

- Revised the manuscript for minor errors, enhanced the introductions, clarified the figures as pointed out by R3 (Zpq5). 

-  Added multiple run results (Table 1) as requested by R3 (Zpq5).

-  Enhanced the literature review as requested by R3 (Zpq5).  

-  Presented some state-of-the-art pruning techniques applied to tasks and models similar to ours, and compared them with our method as requested by R4 (Fzkp).
</content>
</thread_object>

</thread>
<thread>
creation date:
1733167047139

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
As the discussion period draws to a close, we would like to express our sincere appreciation to all reviewers. Your thoughtful comments have helped us better understand and position our work. The review process has motivated major breakthroughs in our theoretical analysis of derivative flow, led us to fundamentally rethink our contribution of introducing overconnectivity/sparsity as an architectural choice rather than an engineering trick, and opened exciting avenues for future work.
Given that today (Dec 2) is the final day of requesting input from authors, we welcome any additional questions or requests for clarification
</content>
</thread_object>

</thread>
<thread>
creation date:
1733255695836

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
As the discussion period concludes, we'd like to highlight the consistency of our findings across a wide range of scenarios. LCNs have demonstrated consistent behavior across:
* Multiple architectures (MLPs, Transformers, Mixers)
* Various tasks (classification, language modeling)
* Different training paradigms (from-scratch training, fine-tuning, pre-training)
* Different model scales and layer configurations

These results strongly suggests that the benefits of LCNs are fundamental to the architectural design rather than specific to particular configurations. However, we understand that we provide limited experimental proof for larger top-performing models. We are actively working on two significant extensions:
1. A large-scale (1B parameter) LLM pre-training and fine-tuning experiment to extend our preliminary BERT findings
2. The ViT-ImageNet experiment with 4096 batch size to match standard training configurations

With the reviewers' and Area Chairs' permission, we would include these results (when available) in an appendix of the final paper, as they would provide additional validation of our findings at larger scales.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents an approach for reducing the compute required for inference in deep neural networks. The authors claim that when one adds long-distance residual connections to a deep neural network (i.e. connections from lower layers straight to the final layer), then a lot of the computation gets put on earlier layers and the later layers can be dropped from inference without any reduction in accuracy. To support this claim they present empirical studies on classification tasks for a few different network architectures showing that they can maintain accuracy even after pruning away later layers from these long-range connection networks.

The strengths of this paper are that it is well-written and novel, and presents a potentially useful technique for improved efficiency at inference. The weaknesses are that the purported links to biology are tenuous at best, and the authors’ provide relatively little evidence that this technique is better than other techniques for improved efficiency, e.g. either pruning or conditional computation (both of which can achieve similar improvements in inference cost).

Based on these considerations, the AC feels that this paper, though potentially an important contribution, requires more work before it is ready for publication at a major ML conference. In particular, though the authors said in rebuttal that they didn’t compare to other pruning techniques because they are lossy, it seems pretty reasonable to ask how much is the comparative gain in efficiency and *how* bad pruning techniques are in comparison to this one? That way, readers can judge whether they would be willing to take a small hit in accuracy for even better efficiency, if that was indeed the trade-off. As noted by the reviewers, the architectures and tasks used could also be expanded. And, there doesn’t seem to be any comparison to other conditional computation techniques (see e.g. https://arxiv.org/abs/2404.02258), and in the AC’s assessment, this is a key metric of performance to judge this technique. Finally, the AC would encourage the authors to consider either toning down or removing the claimed links to biology as the results here don’t really have any clear connection to how pruning works in the brain (which is not generally a process of removing entire layers regions from the network at inference, but rather, selective removal of synapses across the entire hierarchy).