Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes Neural Attentive Circuits (NAC) which learns module parameterization and the sparse connectivity between modules trained jointly End-to-end. The architecture is based on the Perciever-IO arhictecture which serves as a baseline for their ablation experiments. They introduce a SKMDPA which is a sparse attention mechanism utilizing with the Concrete distribution (Gumbel Softmax Distribution) and ModFC and ModFFN which conditions their compution on an input code. These modules control the attention connectivity and their computation based on the output of the Circuit Generator which can either be input-conditional or unconditional. They demonstrate the input-uncondition version of their work on a few-shot image classification tasks on the Cifar and CUB datasets and achieve better results than their baseline Perciever network.  They show with ablation on tiny imagenet that simply adding their modules and learnable sparse connectivity does not significantly improve in distirbution results, but greatly improve out of distribution results. They further demonstrate adding different graph-connectivity priors to the learned connectivity also significantly improves performance. They experimented with the sample-conditioned circuit design generation trained on the Natural Language for Visual Reasoning for Real using the CLIP encoder to preprocess both text and images. The circuit generator is conditioned only on the input text and only the image is given to the network executor. They visualize a TSNE of the connectivity graphs output by the circuit generator and show that their is a strong grouping in graph connectivity based on type of question given.
strengths_and_weaknesses: Strengths:
- Proposes a novel architecture for jointly learning the sparse connectivity between modules and the model parameters.
- Ran ablation which demonstrated their improvements with learned non sample-conditioned sparse connectivity can improve out of distribution accuracy on tiny-imagenet and further graph priors improve in and out of distribution accuracy.
- Demonstrated improvements over Perciever-IO on few-shot adaptation.
- Demonstrated with a visualization that the architecture can learn different paths for different types of problems on the Natural Language for Visual Reasoning for Real datset.
- Demonstrated that their method can be used to sparsify the architecture and achieve the same performance as Perciever-IO with 80% of circuits dropped. They further showed that this leads to a real inference speedup.

Weaknesses:
- Does not demonstrate the architecture as a General purpose architecture after introducing it as such. The results with the text modality use clip embeddings.
- Weak empirical results. No strong baselines for the non sample dependent results. Very similar to architecture search and sparsification, but no reference. No comparisons on the same tasks as the original Perciever-IO paper.
- The sparsification results seem strong, but still there aren't comparisons to other sparsification methods or Neural Architecture search methods.
questions: Can you compare your model in the same setting as the Perciever-IO paper? Do you have metrics for comparison on language tasks? It would make it more convincing to compare this model as a general perpose architecture if there comparisons in more modalities.

It is likely useful to compare this work to other works for sparse attention and dynamic routing to increase inference efficiency as well as NAS works which also use the concrete distribution and other hard selection mechanisms for architecture search. [1], [2], [3].

The uncondition circuit generator work is significantly similar to many works in NAS and sparsification and this work would benefit greatly from comparison.

The Conditional Circuit Generator is quite novel especially if applicable to multimodal data, but the performance and properties of the network could be explored in much more detail.

Could you elaborate about the ModFNN properties?
"unlike in transformers, each copy is conditioned by a unique learnable code vector and therefore performs a different computation. Consequently, though the computation performed in each module is different, the total number of parameters does not noticeably increase with the number of modules"
I'm not sure I understand. Are they unique and increases parameters or are they shared? Some of it is elaborated on in section 2.2, but it still not entirely clear if they are unique to each layer or module.

Can accuracy and performance be discussed in section 4.2 with baselines? It would allow stronger conclusions be ot made about the results.

For "We observe that the Neural Attentive Circuit (NAC) is much more robust than Perceiver IO to sparsification at inference time" do you have results with PercieverIO sparsification baselines?

For Figure 5b, do you have a baseline for the compute cost and inference speed of Perciever IO?

May be useful to compare to [5]. The results with the meta-network inference such as in section 6.3 is somewhat similar to the work with an Unconditional Circuit Generator.

Unrelated to the paper quality, but could this architecture be effective for continual learning? Some methods in continual learning also try to dynamically change the network to partition off tasks to prevent catastrophic forgetting.

[1] Cai, Shaofeng, Yao Shu, and Wei Wang. "Dynamic routing networks." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021.
[2] Xie, Sirui, et al. "SNAS: stochastic neural architecture search." International Conference on Learning Representations. 2018.
[3] Correia, Gonçalo M., Vlad Niculae, and André FT Martins. "Adaptively Sparse Transformers." Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
[4] Yoon, Jaehong, et al. "Lifelong Learning with Dynamically Expandable Networks." International Conference on Learning Representations. 2018.
[5] Shaw, Albert, et al. "Meta architecture search." Advances in Neural Information Processing Systems 32 (2019).
limitations: I'm not sure the authors should claim this is a general-purpose modular neural network architecture without experiments in multiple modalities.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a novel general purpose modular neural architecture; Neural Attentive Circuits. In NAC, information is routed through between a set of modules that perform conditioned computations. Information routing and computation conditioning is specified by the circuit generator and information is processed by the circuit executor. The paper proposes different instantiations for the circuit generator depending on the use case. NAC is compared to PerceiverIO, a general purpose architecture used as a basis for developing NAC, on image classification, few-shot adaptation and OOD generalization. Although NAC does not reach PIO’s performance level on the ImageNet validation set, it performs better on OOD generalization and Few-shot adaptation. Furthermore, pruning several modules significantly decreases the model’s computational cost without a significant loss in accuracy. Ablations of the model highlight the importance of design choices such the conditioned computations, modularity and sparsity priors. Finally, the usefulness of data conditioning in the circuit generator is demonstrated on a visual reasoning task where similar instructions generated similar circuit connectivity patterns.
strengths_and_weaknesses: Strengths

- The paper proposes a novel method for modularizing neural networks.
- This method is has a flexible and efficient architecture design; module connectivity and modulation can be reduce the computational cost in the model and could potentially be incorporated in any model to replace MLPs.
- The model’s signature initialization can be used to embed specific priors in the model.
- A large percentage of modules can be pruned without significantly decreasing performance.
- Module connectivity provides insight on how the model solves tasks.
- The architecture design improves learning in low data regimes, OOD robustness and few-shot adaptation on two benchmarks.

Weaknesses

- While the architecture is claimed to be general purpose (applicable to arbitrary data types), its usefulness is demonstrated only on visual data.
- As the authors mentioned in the limitations, the method is not applied to large scale data compared to Perceiver and PerceiverIO. Furthermore, performance on the ImageNet validation set is significantly lower compared to PIO.
- If I understood section 2 correctly, the state $\theta^{i}_u$ of a module is only used to compute the query in layer $l+1$. If this is the case, the state does not represent a state of the model (as much as the weights $W$ and $W_c$). It would be less confusing to give $\theta^{i}_u$ a different name.
- Looking at the hyper-parameters in table 3, the number of layers and number of layers that do not share weights both equal 8 which means the module weights $W$ are not shared across layers. If this is the case, the sharing of signatures and codes across layers is an unnecessary constraint on the model (since modules in different layers perform different computations). Furthermore, this means the modules in different layers are completely different and the notion of a state at different layers does not make sense for the model.
- Considering the model’s similarity with PerceiverIO, the input is a MxC array of data that is transformed into a NxD latent array using the processor. PIO performs transformer-based attention on the N vectors before performing transforming them with an MLP. In NAC, SKMDPA is performed over U modules. It’s unclear how vectors of the latent array are channeled through modules. Can several vectors be processed by the same module ? And can several modules process the same vector ? It would be better to include the latent array dimension N in the description of the model and to clarify in detail the changes made to PIO to arrive at NAC.
- What is the architecture of the circuit generator ?
- How are CLIP text embeddings fed to the circuit generator ? How are all embeddings fed to PIO in the comparison experiment ? If both PIO and NAC achieve 64% accuracy on NLVR2 then NAC does not improve over PIO in this task.
questions: The paper lacks in clarity. It would be beneficial to explain in more detail the model design and experiments following the issues mentioned in weaknesses.

Since signatures and codes are shared across layers, the connectivity and modulation are similar in across layers. It would be interesting to vary dynamic signatures and codes while maintaining the weights shared across layers. Was model design explored previously ? If so, how did it perform compared to the current design ?
limitations: The limitations are addressed in the supplementary material and no negative societal impact was discussed in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a novel neural architecture (NAC) which learns to jointly predict the optimal (sparse) connectivity (via signatures) and parameterization (via codes) of its general purpose modules (ModFC/FFN). The architecture appears parameter- and computation-efficient, compares favorably against Perceiver IO in terms of few-shot adaptation (CUB/CIFAR) and OOD generalization (Tiny-ImageNet), and meaningfully generates similar connectivity patterns when conditioned on similar input samples (NLVR2).
strengths_and_weaknesses: Strengths
+ (Originality, Major) The paper’s novel architectural features (ModFC, sparse graph priors) compared to Perceiver IO, and performance improvements (linear scaling w.r.t. input size, adaptive computation with large speed boost) compared to Neural Interpreters, are impressive.
+ (Clarity) The writing is clear and easy to follow. The figures are well-made and very helpful.

Weaknesses
- (Quality) The empirical validation of the proposed architecture is insufficient in my opinion. To claim that NAC is a stronger general purpose architecture, the scope of the comparisons (now only few-shot adaptation and OOD generalization in CV datasets) and the number of baselines (now only Perceiver IO) both need to be reasonably improved. E.g. Perceiver IO was evaluated on 6 different modalities (Table 5 in their paper) and against non-general purpose baselines within each modality.
- (Significance)
Due to the insufficient empirical validation, it’s hard to conclude that the proposed architecture is significant and will inspire or lead to future breakthroughs.

questions: - All experiments seem to use some form of preprocessing (convolutional or CLIP) which in a way renders the general purpose claim of the paper questionable. Does the proposed architecture not work with raw inputs (i.e. byte arrays) like Perceiver IO?
- In Table 1, is (Perceiver IO + ModFC) equivalent to (NAC - sparse graph prior) (i.e. without minimizing Eq 9)? Shouldn’t a dense connectivity perform better under ID validation due to less constraints? Mechanistically, how does the spare connectivity in NACs help achieve better OOD generalization?

limitations: The authors have adequately addressed the limitations of their work.
review_title: null

Meta-review: metareview: The reviewers unanimously agreed that the work presented here was novel and improved the state of the art in models that perform conditional computation on a wide variety of different kinds of data. The reviewers were originally concerned that the submission focused evaluation only on visual data, but the authors provided additional data during the response period that made clear the generality of the proposed technique. As such, I recommend acceptance. I look forward to visiting the authors' poster in New Orleans! Authors: this enthusiastic acceptance is predicated on the expectation that you update the paper with these new results.
confidence: Certain
Decision: Accept