Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Humans are good at generalizing previously acquired knowledge to find solutions to novel problems. In this way, they can learn a complex task by combining knowledge from different experiences instead of learning from scratch. This study suggests that neural networks can also afford this compositional generality by having task representations in a geometrical structure that allows a linear classifier to be used for generalization.

strengths_and_weaknesses: Strength,
This work provides clear evidence that the representational structures (parallelism scores) are closely related to the ability for efficient cognitive control and generalization while the representations and cognitive control have been examined largely in separate domains. The effects of training facilitating fast learning can be found not only in human participants but also in the neural networks, which is interesting because this may not be easily explained by reinforcement learning but supports the idea that learning entails building abstract task structure representations.

Potential weaknesses,
It is not clear whether the neural representations were based on the data acquired at the time of the presentation of the conditioned cue, at the time of decision making (maybe ash the time of the second stimuli), or the mean activity across a mini block. If it is the former, this is before the stimuli presentations and decision making, which is not true in the ANN architecture (the ANN were made with the inputs of not only conditions but also stimuli). If it includes decision making, it may not be the rule but the activity associated with the decision making might be represented.

In the previous study (Bernardi 2020, et al.), the rule cannot be separated from the stimuli while the representational structure cannot be explained by the similarity between stimuli. That is, the structure was built based on associated rules for each of the visually independent stimuli. However, in this experiment, can some pairs of parallel representations also be explained by the (dis)similarity of input vectors? (e.g. the same pair of stimuli followed by the different action cues). How would the same condition be represented if they were paired with different stimuli pairs?
questions: Does the ANN learn different conditions in separate blocks for pretraining? How can the ANN learn a new condition while not modulating the weights learned from the pretraining of previous conditions (a.k.a. catastrophic forgetting)? Does the order of pretraining conditions matter in the accurate generalization? I think this part is the key to this paper but I can't find how this part was made.
limitations: Humans intuitively know which features of sensory stimuli can be combined and which features cannot be combined (e.g. red vertical stimulus can be shown but red blue stimuli cannot be presented). Therefore, they can generalize the representation of 'vertical' to 'blue' stimuli even only when they were trained with 'red' or when the colors of training stimuli are covered with their shape. However, the proposed ANN would not yet be able to overcome these kinds of biases. On the other hand, humans might have more experiences to answer to 'both' and 'either' than their negation, this would make it harder to respond to 'neither' or 'not both' conditions, while the ANN might not have  (or might not able to capture) these biases.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents an analysis of fMRI task data which seeks to uncover "abstract representations" in the brain. This same analysis is then applied to a simple artificial neural network.

The analysis is built around a "parallelism score" which measures displacement vectors between brain activations. Given three modalities, (sensory, logical, and motor), each modality is held fixed and the other two are varied. The parallelism score for the fixed modality is an average measure of how aligned the displacement vectors are, averaged across combinations of the other two modalities.

The authors find that the highest motor parallelism is found in the motor areas, the highest visual parallelism in the visual areas, and that the logical parallelism is distributed throughout the brain, but lies mostly in the temporal and frontal areas.

This same analysis is applied to an artificial neural network. It is found that pre-training the NN results in better parallelism scores and better zero-shot performance.
strengths_and_weaknesses: ## Strengths
- Application of the parallelism score to neural networks is novel as far as I can tell
- Identifying the cognitive mechanism behind compositionality is an important problem
- Analysis of this particular fMRI dataset is novel and shows the promise of this sort of analysis

## Weaknesses
- My main concerns are about the significance of the machine learning results. The paper claims that it introduces a new pre-training scheme. But the model architecture being considered is a two layer feed forward network and the pre-training is similar to schemes that have already been proposed, e.g., gSCAN [1]. The simplicity of the architecture and setting make it hard to see whether the results about pre-training would still hold in general. 
- The particular task setting also makes the comparison between human behavior and ANNs hard to interpret. The paper claims that their trained ANN shows a human-like response pattern to tasks. That is, the lowest level of the ANN is activated for sensory modalities, the middle layer for logic modalities, and the last layer for motor modalities. But this cannot be used to support the claim that the ANN has converged on a human-like hierarchy of representations, because there is no analogous ordered layering in the brain. Rather, the human brain results show that the motor areas respond to the motor modalities, the logic areas to the logic modalities, and the sensory areas to the sensory modalities. 
- Some clarity in description would be helpful. In section 3.3, I am not sure what the model inputs and outputs are supposed to be.
- The stories about the brain and about the ANN don't seem to interact much. This paper presents some findings about the human brain and some findings about pre-training ANNs, but beyond section 3.7, the two stories do not seem very related.
- The paper talks about identifying abstractions, but the only abstractions that can be discovered using the parallelism score are ones in which the representations have a linear relationship. 

## Bibliography
[1] Ruis, Laura et al. “A Benchmark for Systematic Generalization in Grounded Language Understanding.” ArXiv abs/2003.05161 (2020)

questions: - line 105: What is mean by "coding axis"? In the text that follows, I gather that it is a displacement vector associated with a given modality. Is this correct?
- line 120: In this case, what is meant by "neural activation space"? The space of all fMRI scans?
- line 124: What is the activation vector? An vector of fMRI voxel inensity?
- Figure 3 caption: This is the first place where the activation vector is mentioned. I would move this into section 2.3.
- Figure 3 caption: "The cosine angle between two linear decoders" --> Is this the ANN mentioned in line 135? That seems like a non-linear decoder. What does it mean to take the cosine angle between decoders? Does this refer to the classification boundary? Or the hidden state representation?
- line 135: What is the input to the model? Is it the intensity for all voxels?
- Section 2.3: The PS will be high if the coding directions are parallel, but it will also be high if the coding directions are identical. How do we know that the coding directions do not all lie in the same low dimensional space, i.e., as in Figure 2B. Importantly, it would be good to know, of the pairs $v_i$ and $v_j$ that have high cosine similarity, what is the $l_2$ distance?
- Section 3.3: What is the input to these ANNs? A text description of the task?
- Section 3.4: What does it mean to measure the PS of an ANN? What is the activation vector for the ANN?
- line 217: Is there only one training example per context?
- line 220: "unlike humans (see Fig. 1b), ANNs with no prior knowledge cannot compositionally generalize." I understand the claim being made, but it should be noted that an adult human is not exactly without "prior knowledge"
- Figure 6: In 6B Brain systems are on the y-axis and rule domains are shown in groups. In 6C rule domains are shown on the x-axis and layers are shown in groups. I suggest putting layers on the y-axis in 6C to make the figures easier to compare.

## minor things
- line 171: missing Figure 7
- Section 3.5: missing Figure 9

limitations: Limitations are adequately addressed
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Building off prior work proposing the role of “parallel abstract representations”—linearly additive vector representations of task dimensions—in compositional generalization, the authors set out to evaluate this framework through an analysis of existing behavioral and fMRI data as well as through ANN experimentation. The authors find that human task generalizability patterns are consistent with this factorized task representation framework and that parallel abstract representations are present for sensory, logic, and motor rule types in cortical systems that correspond to previous decoding work. They then propose a pretraining paradigm for ANNs on either task primitives or on simpler compositions of task primitives, and find that this improves zero-shot generalization to a more complex permuted rules task as well as sample efficiency, while resulting in representations which mirror some aspects of the neural encoding geometry and cortical hierarchy.
strengths_and_weaknesses: This paper addresses the interesting problem of compositional generalization, and does well to benchmark against human data. The extension of the “parallel abstract representations” framework to a new higher dimensional task and the resulting fMRI analysis is interesting, namely that these types of linearly additive vector representations of task abstractions appear to emerge in an expected and interpretable way.

The move to computational models and assessment of representational geometry in that regime is well motivated, and the results are very interesting, but in many respects, the details are drastically under-defined in the main text. In some cases, the reader is left struggling to understand critical aspects of what the authors actually did, until moving to the supplementary materials. For example, the precise exact input structure to the ANN is left to speculation until finding supplementary figure 8. This type of information needs to be expanded considerably in the main text.

questions: The authors note (once again only in the appendix) that pretraining was stopped once pretraining tasks achieved 99%. To confirm, >99% accuracy on each task, e.g. until the worst task achieves 99% accuracy? How many training samples / iterations did this take under each pretraining paradigm?

For the Vanilla-ANN comparison, how is that generated? Is it just randomly initialized? From what distribution? (Okay I found this in appendix; add more details to main text.)

Aside from pretraining familiarizing the networks with task context rules, the networks also become more familiar with and learn information about the sensory stimulus input units. Do you think that this confound ends up being relevant for the observed increased sample efficiency, both for the primitives pretraining and especially for the simple task pretraining, which I assume was probably pretrained for more iterations to get to 99% due to the increased task complexity?

It would be helpful to see a better breakdown, in general, for both brain and ANN experiments, about which specific rules were more consistently encoded, e.g. how did PS scores differ across rules within each rule domain? Did certain dimensions drive mean PS scores more than others?

Overall, this is an interesting study with great potential for publication, but the authors really need to be more explicit and detailed in their description of the ANN pretraining experiments, especially in the main text, so as to alleviate reader concerns on potential confounds.

limitations: The authors have addressed some limitations of this work. I agree that future work should attempt to replicate this using other task paradigms.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work investigates compositional generalization in humans (using fMRI data) and ANNs. The working hypothesis of the paper is that parallel abstract representations - consistent variable representations across different contexts - support compositional generalization. The Parallelism Score, a measure from neuroimaging research, is used to measure the consistency of representations in humans and ANNs. In their experiments, the authors use the C-PRO paradigm to construct a 64-context compositional task for humans and ANNs. The paradigm enables systematically varying conditions to test generalization abilities. In their experiments, they expose humans and ANNs to limited contexts and test them in novel contexts. First they confirm that humans perform well on these tasks, but drop slightly in novel contexts. They observe parallel abstract representations in fMRI data, supporting their hypothesis that this enables generalization.  Based on their finding that for humans, increased exposure to a rule improves performance in novel contexts, they design pretraining tasks to improve generalization in ANNs. This includes 1-rule and 2-rule simplifications of the 3-rule tasks of C-PRO, enabling the ANN to learn prior representations for sub-rule variables. Without pretraining, ANNs struggle in novel contexts. Pretraining significantly improves performance in novel contexts, enables zero-shot performance, and more sample efficient learning. Pretraining enables compositional generalization in ANNs similar to humans and similarly increases parallel abstract representations. The authors also report similarities in the hierarchical Parallel Scores between human fMRI data and  ANNs. 
strengths_and_weaknesses: I think this is an excellent paper. The authors apply ideas from neuroscience research to the growing field of study that is compositional generalization in ANNs. My only caveat to confidently recommending acceptance is that I am not qualified to assess the validity of the neuroscientific claims made in the paper. But it seems that the authors have followed a solid methodology throughout and do not overstate any claims, so I would feel comfortable still recommending acceptance. Furthermore, the findings of the paper related to ANNs are quite interesting and novel on their own.

**Strengths**

1. The ideas in this paper are novel and valuable in the study of compositional generalization in ANNs. I am not aware of previous work applying the idea of parallel abstract representations to ANNs*, and this seems useful for future research. Furthermore, the methodology followed by the authors to perform analogous experiments in neuroimaging and ANNs is unique and well thought out. A human-centric benchmark for compositional generalization at such a scale (64 contexts) is certainly valuable.

2. For the most part, the claims made by the authors are supported and well motivated throughout the paper. Their pretraining method boosts generalization significantly and should be further developed in different deep learning contexts. I would have expected the PS measures on ANN hidden layers to be higher than those reported in Figure 5 A, but the zero-shot performance and sample efficiency makes a strong case that these pretraining methods are effective.

3. The paper is excellently written and well structured. The ideas are presented clearly, without overstating any claims.

\*  The closest idea to this that I have read of is the concept of “localism” from the paper by Hupkes et al. titled Compositionality Decomposed: How do Neural Networks Generalise? (2020). It’s not the same idea, but is similar in that it expects compositional models to build consistent variable representations independent of context.

**Weaknesses**

1. There is one experiment where the results are not as convincing as one would wish. In Figure 5A the PS measure increases only slightly after pretraining. This shows that pretraining does in fact encourage parallel abstract representations, but it does not necessarily show that this is absolutely crucial to compositional generalization. The PS measures increase slightly, but the generalization ability increases significantly, so perhaps some other factor enabled by pretraining is enabling generalization. Since the working hypothesis of the paper is the link between parallel abstract representations and compositional generalization, it might be worth discussing this result more.

2. The paper performs its ANN experiments with a simple 2-layer feedforward neural network. I understand that this isolates the effect of pretraining for the experiments, but in the process it ignores the neural network architectures that are currently state-of-the-art in most domains i.e. Transformers, CNNs, and LSTMs. Most existing studies on compositional generalization in ANNs have focused on these more advanced architectures and if this paper had done so it would have made a stronger case for the proposed methods and methodologies. As such, the findings of this paper are not directly comparable to leading edge research on the generalization capabilities of ANNs.


questions: 1. What is your response to weakness (1)? Do you think the relatively small increase in Figure 5A undermines your hypothesis that parallel abstract representations are important for compositional generalization?

2. Please elaborate on how you arrived at your pretraining method from your finding that in humans "increased exposure to specific rules improved performance on subsequent novel contexts". I understand the finding, and the pretraining method, but to me the link isn't obvious. The pretraining method does not simply expose the ANN to more examples of contexts in which the rule appears (that would simply be showing the ANN more 3-rule examples containing that rule). The pretraining method exposes the ANN to simplified contexts (1-rule or 2-rule examples), where it can focus on learning representations. How do you motivate the link?
limitations: The authors dedicate an entire section to discussing the limitations of their work. This is valuable and insightful.
review_title: null

Meta-review: metareview: The manuscript presents a story about compositionality that ties together neuroscience and models; with a focus on how compositionality enables generalization in a task performed by humans undergoing fMRI. Reviewers were largely happy with the manuscript and authors thoroughly addressed the questions that reviewers had.

The manuscript is suggestive. The experiment is in many ways limited, and it's not clear what conclusions one can draw at present about the design of models or of the brain. But, it is likely a significant audience at NeurIPS will be interested in this topic and it may spark followup work. This looks like the beginnings of an interesting line of work expanding the paradigm of mapping between brains and artificial neural networks.
confidence: Certain
Decision: Accept