MIND over Body: Adaptive Thinking using Dynamic Computation

Authors: Mrinal Mathur, Barak A. Pearlmutter, Sergey M. Plis
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Interpretability, Fixed points, Dynamic routing, Dynamic input processing, Deep Learning Framework

Abstract:
While the human brain efficiently handles various computations with a limited number of neurons, traditional deep learning networks require a significant increase in parameters to improve performance.
  Yet, these parameters are used inefficiently as the networks employ the same amount of computation for inputs of the same size, regardless of the input's complexity.
  We address this inefficiency by introducing self-introspection capabilities to the network, enabling it to adjust the number of used parameters based on the internal representation of the task and adapt the computation time based on the task complexity.
  This enables the network to adaptively reuse parameters across tasks, dynamically adjusting the computational effort to match the complexity of the input.
  We demonstrate the effectiveness of this method on language modeling and computer vision tasks.
  Notably, our model achieves 96.62\% accuracy on ImageNet with just a three-layer network, surpassing much larger ResNet-50 and EfficientNet. When applied to a transformer architecture, the approach achieves 95.8\%/88.7\% F1 scores on the SQuAD v1.1/v2.0 datasets at negligible parameter cost.
  These results showcase the potential for dynamic and reflective computation, contributing to the creation of intelligent systems that efficiently manage resources based on input data complexity.

Decision: accept (oral)
Reviewer scores: 6, 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1730474121921

thread_type:
official_review

<thread_object>
creator: reviewer_p7ez

<content>
summary:
The authors propose a framework for designing architectures that are able to adaptively control the number of computations they perform in order to produce an output given an input. Their proposal has two main components. First, separate the model into two parts: a prediction network and a control network. At each step, the latter decides which layers to use to refine it’s estimate of the output. To make computations even more granular, they use a Deep Equilibrium networks to implement each layer as a fixed-point iteration computation. Thus the control network can no only decide which layers to apply, but for how long. The authors proceed to show the effectiveness of their approach on several qualitatively different datasets.

strengths:
1. The approach is well motivated and the problem of adapting the computations used by a model is an interesting one.
2. The authors lay out the approach in good detail, explaining how it diverges and improves from previous work.
3. They conduct extensive experiments to support their goal of improving upon previous approaches.

weaknesses:
1. Some details on model architecture and metrics are missing.
2. Parts of the language used to describe the model are misleading and fall into unnecessary anthropomorphising.

questions:
1. It is not clear from Figure 1 or maybe not detailed enough in the text how the controller network determines the computation time of each layer. In other words, how are the parameters determined? Or are they always fixed and it is just a choice between 1 pass or multiple (until covergence)?
2. The complexity metric is described as “thorough” but it is not clear what makes it such. Or even what complexity means in this case.
3. CompCost is said to depend on the number of layers and iterations, but is this a sum or some other function? Not clear from the text.
4. I have an issue with the anthropomorphising the authors lean into.
    1.  There is no mention about any “body” in the text so the title is misleading. 
    2. The model doesn’t think, it processes. There is no “self-awareness”, at most it self-regulates. 
    3. And why call it “MIND”? It doesn’t even match the first letter of the name they themselves assign.
    4. The title could be “Adaptive Processing through Dynamic Computation Control” or something, which conveys a better feel about what the authors are doing.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730527608885

thread_type:
official_review

<thread_object>
creator: reviewer_57h2

<content>
summary:
The paper introduces an approach targetted at computational efficiency in deep learning by adapting amount of computation to complexity of each input. Inspired by how human brain is considered to allocate resources dynamically. Two core components, (i) introspection network and (ii) prediction network.  Introspection network analyses intermediate activations from prediction network & figures out which layers require additional compute via fixed-point iterations )FPI) as well as what can proceed with standard forward pass.  Prediction net performs FPI until convergence or threshold of iterations reached. Leverages phantom gradients method for backprop through FPI; gradients approximated without unrolling / jabobian calc to limit compute-memory needs during training.

The paper is well written and easy to read + get the core idea across.

strengths:
Computational efficiency; optimal use of resources allocating more for compelx inputs. Clever use of intermediate activations to assess input complexity. Should be able to work with existing architectures making engineering it for downstream real-world use cases simpler. Backprop with phantom gradients done in an interesting way. Mixed with use of statistical methods ~ should allow for generalisation. Considered overfitting issues within architectural design.

weaknesses:
The idea is subtle and complex + introspection network needs more compute cost. Approach may not capture actual gradient landscape given the number of adjustments made & have to be considered. Strategy for arrving at thresholds / stopping criteria around convergence is not clear as such. Gradient flow calc is non-trivial given the overall architecture.

questions:
Given the size of the appendix can you put in a full section to indicate the weaknesses (vs hinting at them in the future works/conclusion section)?
Is the trade-off in terms of gain for the complexity worth while?; how can this be known earlier before selecting this approach?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705185017

thread_type:
official_review

<thread_object>
creator: reviewer_fhgv

<content>
summary:
The paper proposes a Model INtrospection for a Dynamically adaptive model (MIND) which dynamically adjusts computation depending on the complexity of the input. It consists of two networks: the introspection network and the prediction network. The introspection network takes as input the activations from the different layers of the prediction network, and outputs a binary mask over the layers, determining the layers which require more computation through fixed point iterations. The authors demonstrate the effectiveness of the MIND model for vision tasks using a three layer CNN as the prediction network, outperforming much larger models like ResNet-50, and EfficientNet B7 on ImageNet and CIFAR-100 datasets. The authors also propose MIND-Transformer, with fixed point iterations in self attention and feedforward networks, demonstrating its superior performance on language modeling tasks, despite using fewer parameters than RoBERTa-base. The authors further demonstrate that MIND’s dynamic allocation of computational depth depending on the input complexity is more effective, both in terms of accuracy and efficiency (fewer parameters and FLOPs) over static compression techniques like pruning and quantization.

strengths:
1. The authors propose a model, MIND which dynamically adjusts computation via fixed point iterations in its prediction network using an introspection network depending on the input complexity.
2. MIND model with a three layer CNN as the prediction network, outperforms much larger models like ResNet-50, and EfficientNet B7 on ImageNet and CIFAR-100 classification datasets.
3. The authors also demonstrate that MIND using LSTMs and Transformers in the prediction network achieves superior performance on language modelling tasks using fewer parameters. 
4. MIND’s dynamic allocation of computational depth results in higher accuracy using fewer parameters and FLOPs over static compression techniques like pruning and quantization.

weaknesses:
1. It is not clear how the input complexity metric is incorporated into the introspection network's mechanism, and how this can be more generally quantifiable

2. The MIND model when used with prediction networks with many layers (as in the case of LLMs) will significantly increase the inference time as more layers with fixed point iterations are used.

questions:
1. What is m’ in line 147?
2. What is the rationale behind Equation 5 for MIND-Transformer?
3. How is $w_l$ in Equation 7 computed? Also, what is the need for a separate $m_{i,l}$ term in Eq 7?

4. Just to clarify in the toy random dot motion task the model needs to classify in one of the four possible directions and the direction of the shifted image denotes the ground truth?

5. Which dataset are the results for in Table 5?

6. What are the parameters of the introspection network, like how many layers are there in the MLP and what are the sizes of the hidden dimensions? 

7. From $p_{i,l}$ in Equation 8, how are the binary layer selection variables $m_{i,l}$ obtained?

8. Can the authors share more details about the different MIND variants, like how are fewer FPI layers decided, what is a simpler inspection network and how are the decisions of the introspection network fixed after training?

Minor - Line 127 typo, should be “prediction”
</content>
</thread_object>

</thread>
<thread>
creation date:
1730787765037

thread_type:
official_review

<thread_object>
creator: reviewer_5aig

<content>
summary:
1) This paper introduces a new method to dynamically allocate network compute based on the difficulty of the inputs allowing early exits at inference time. The proposed method comprises of 2 networks - a) prediction network that outputs activations at each layer for a given input b) an introspection network that taken in the activations and decide which layers to pick for more intensive computation (using fixed point iterations) and which layers to leave as is.

2) Authors also describe a training procedure to jointly optimize the introspection and prediction networks. 

3) Experiments show that a much smaller model (in terms of param count) achieves better performance than considered baselines on language modeling and vision tasks.

strengths:
The proposed method is generic and has been applied across modalities with sufficient ablations to prove that the proposed method work.

weaknesses:
1) The paper doesn't contrast and compare to more recent early exit methods proposed  for language modeling tasks : 
a) Confident Adaptive Language Modeling (https://arxiv.org/abs/2207.07061) 
b) LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding (https://arxiv.org/abs/2404.16710)

questions:
1) Can you provide experiments comparing the proposed method to more recent early exit methods like CALM and LayerSkip ?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731994510561

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary by the Authors

comment:
We are grateful to the reviewers for their insightful analysis that highlighted several key strengths of our work: the clever use of intermediate activations for complexity assessment (R3), the achievement of superior performance with significantly fewer parameters across both vision and language tasks (R1, R2), and the practical advantage of compatibility with existing architectures (R3). We particularly appreciate the recognition of MIND's ability to outperform larger models like ResNet-50 and EfficientNet B7 (R2), while maintaining engineering simplicity for real-world applications (R3).

Building on these acknowledged strengths, we have thoroughly addressed all concerns raised by the reviewers, significantly improving the manuscript's clarity and technical depth. Our responses below detail the extensive additional experiments, clarifications, and improvements made in response to each reviewer's feedback, further strengthening MIND's contribution to adaptive computation research.

## Reviewer 5AiG
 we conducted additional experiments comparing MIND with recent early-exit methods (CALM, LayerSkip), demonstrating our unique advantages in parameter efficiency while clarifying the distinct operational domains.

## Reviewer fHgv
we substantially expanded our technical analysis, providing detailed mathematical formulations of our input complexity metric and proving its domain-agnostic nature. We added comprehensive timing analyses across model scales, supported by new experimental data.

## Reviewer 57h2
we thoroughly addressed the convergence criteria concerns by adding empirical evidence of computational efficiency, including new ablation studies. We also added a dedicated limitations section, demonstrating scientific rigor and transparency.

## Reviewer P7eZ
we refined our model architecture description with precise technical details and metrics, while maintaining the demonstrated strong empirical results that the reviewer praised.


These comprehensive revisions, combined with the strong experimental results and architectural innovations already noted by the reviewers, demonstrate MIND's significant contribution to efficient, adaptive computation across domains. We believe these improvements address all concerns while reinforcing the original strengths identified in the reviews.
</content>
</thread_object>

</thread>
<thread>
creation date:
1747861156309

thread_type:
public_comment

<thread_object>
creator: ~ali_hojjat1

<content>
title:
Question

comment:
Dear Authors,

Thank you for sharing your work, and congratulations on a well executed and insightful paper. It is a thoughtful and valuable contribution to the field.

I had a quick question regarding the evaluation. If I’m interpreting the numbers correctly, the accuracy on the standard ImageNet validation set in Table 2 appears to be approximately 10% higher (around 88%) than on the MultiLabel version (around 78%). I observed a similar pattern for the baselines as well. This stood out to me, as I would typically expect the MultiLabel dataset, with its improved multi label annotations, to yield higher accuracy, given that more predictions can potentially match the correct labels. This expectation also appears to be supported by the results reported in the dataset’s paper, which show performance gains when using relabeled annotations:

| Architecture      | Vanilla | ReLabel       |
|------------------|---------|---------------|
| ResNet18          | 71.7    | 72.5 (+0.8)   |
| ResNet50          | 77.5    | 78.9 (+1.4)   |
| ResNet101         | 78.1    | 80.7 (+2.6)   |
| EfficientNet B0   | 77.4    | 78.0 (+0.6)   |
| EfficientNet B1   | 79.2    | 80.3 (+1.1)   |
| EfficientNet B2   | 80.3    | 81.0 (+0.7)   |
| EfficientNet B3   | 81.7    | 82.5 (+0.8)   |
| ReXNet (x1.0)     | 77.9    | 78.4 (+0.5)   |

I was wondering whether this discrepancy might be related to the evaluation setup or to the way the labels are handled. I would be very interested to hear your thoughts. Thank you once again for the excellent work and for sharing your results.

[1]: Re Labeling ImageNet: From Single to Multi Labels, From Global to Localized Labels (Yun et al., 2021)
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes Model INtrospection for a Dynamically adaptive model (MIND), a method to dynamically allocate network compute based on the difficulty of the inputs. The proposed method consists of two networks - a) prediction network and b) an introspection network that decides which layers to run for more intensive computation (using fixed point iterations). The authors demonstrate the effectiveness of the MIND model for vision tasks using a three layer CNN as the prediction network, outperforming much larger models like ResNet-50, and EfficientNet B7 on ImageNet and CIFAR-100 datasets.
The authors further demonstrate that MIND’s dynamic allocation of computational depth depending on the input complexity is more effective, both in terms of accuracy and efficiency (fewer parameters and FLOPs) over static compression techniques like pruning and quantization.

Strengths: The approach is well motivated and the problem of adapting the computations used by a model is an interesting one. The method is presented clearly and well contrasted with prior work. Extensive experiments show clear improvements compared to previous approaches.
Weaknesses: Most of the raised weaknesses were addressed during the discussion. There remain some concerns with respect to the complexity of the method. 

Overall this paper presents a solid contribution to an important problem and should definitely be accepted, possibly even as an oral presentation.