Venue: NeurIPS 2021 Workshop SVRHM

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The submission studies how category-orthogonal information guides object recognition in a small recurrent network (RNN). The authors hypothesize that category-orthogonal information (e.g., location, size) is not reduced during processing, but is instead conditioned on in order to process category-relevant information (e.g., shape). Two experiments (decodability and an ablation study) confirm this hypothesis.


Strengths:
- The question is meaningful and carefully investigated with a correlational study and a causal manipulation.

Weaknesses:
- The broader impact of the study is not made clear because the small RNN is not demonstrated to be representative of biological or modern machine learning systems.
  - It is mentioned in the paper that biological systems use recurrent connections, but the particular architectural studied in the paper is not motivated further; the system & problem setting is also far from object categorization systems used in current practice (ie. deep neural networks, even though MNIST and its variant is claimed to be "visually challenging").
- The experiments are missing some important explanations that are required to interpret the results:
  - I like the approach of using "linear readouts" to detect for "decodability" (thus availability) of the category and auxiliary information. However, the process for training the readout gadgets was insufficiently described: I am assuming they are linear classifiers but I don't believe this was described in the text. Moreover, how are train/test splits constructed for these gadgets? How is the target value produced? 
  - The causal experiment is interesting and important to establish the functional importance of the auxiliary information, but the manipulations and baseline were not clear from the text, and so the experiment was less impactful than it could have been.

Minor:
- Some terminology is not sufficiently defined / explained:
  - "lateral connections"
  - "gain modulation"
review_title: An interesting study on context-dependent processing in an RNN network; however, the relevance for biological and machine learning systems is not made clear.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ## Summary
The authors aim to understand the computational role recurrence and feedback in visual object categorization. They note that while multiple studies have demonstrated the benefits of recurrence in cases such as cluttered environments, we know little about the mechanisms involved. They then propose the hypothesis that recurrence amplifies information about category-auxiliary variables (e.g. object location) and uses it to modulate activity across the model. In this way, a model can selectively attend to regions in the image that contain the object of interest. Finally, they confirm this hypothesis through linear readouts of model activations to predict category-auxiliary variables.

I really enjoyed this paper. It had a clear well-motivated hypothesis and used straight-forward methods to test it. As a result, the conclusions are convincing and valuable for understanding the computational role of recurrent processing in vision.

## Positives
- Due to the simplicity of the architecture, recurrent layers, dataset, and linear decoders, it's hard to poke holes in any of the analyses or conclusions. The results also show a very significant and consistent effect of time on the decodability of all category-auxiliary variables considered.
- The dataset was nicely constructed and seems to be a great benchmark for classification tasks in cluttered environments.
- I really like the idea of adding recurrence back to the input as a way of visualizing its effect. It makes it clear that the category-auxiliary variables are likely being used to amplify the region of interest and remove other clutter.

## Negatives
- My biggest concern is the lack of any purely feed-forward benchmark. The argument is that recurrence is beneficial for extracting and using category-auxiliary varialbes (which I believe), but can a feedforward model do the same? Granted, at timestep t=1 the model has only performed feed-forward computations, but it was still trained with recurrence and might rely on future timesteps to better extract category-auxiliary varialbes. The representations in a model that is only ever trained with feed-forward connections might be very differently. Furthermore, if the depth of a feedforward model is made equal to that of the unrolled RNN, does recurrence still provide an advantage in terms of task performance and decodability of category-auxiliary variables?
- Category-auxiliary variables should be defined earlier on. I was initially unsure as to the meaning, and I think that naming them explicitly in the abstract (location, scale, orientation) would resolve this issue.
- The main text doesn't mention what transformations are used for lateral or feedback recurrence ($C_{lat}$ and $C_{td}^m$). The appendix makes it clear that these are convolutional, fully-connected, or transposed convolutional layers, but I think it would be best to quickly mention this in the main text for clarity.
- The reasons for emphasizing recurrence in classification tasks in particular wasn't obvious to me. In fact, the proposed category-auxiliary variables and their use in modulating network activity seems relevant for spacial attention more generally. In other words, instead of arguing that recurrent processing enables the extraction and use of category-orthogonal variables in categorization, a more general argument could have been made for the role of recurrent processing in tasks that require spatial attention.
review_title: Insights into the computational role of recurrence and feedback in vision

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ## Summary:

In the present paper, the authors test the hypothesis that the role of recurrent connectivity in object-categorization is to convey category-orthogonal information such as location or orientation which aids in recognizing a given object, particularly in challenging conditions such as cluttered environments. The authors aim to test this hypothesis by training a simple recurrent convolutional neural network on an object-categorization task with cluttered images and evaluating the performance as well as the presence of category-orthogonal information in the network across layers and time steps. They show that category can be read out from the recurrent network as well as category-orthogonal information across layers and increasingly across time steps. Further they show that this category-orthogonal infromation is conveyed by feedback and that feedback information flow is functionally relevant for the object-categorization task. 

## Pros:

- the hypothesis and results are concisely communicated and the logic of the paper is easy to follow
- the perturbation analyses nicely demonstrate that the auxiliary-variable information is not only a side-effect of the categorization training of the network but that it is directly related to task perfromance and clutter reduction
- visualization of the methods and results is clear and supports readability of the paper

## Cons:

- to me it was not entirely clear where the formalization of the recurrent flow is coming from. Is that idea based on ideas from neuroscience ? Or is that a theory that the authors came up with by themselves ? It would be nice to read more about why the authors think that this is a good quantification of recurrent information flow

## Overall summary:

In sum, I really enjoyed reading the paper. It deals with an intriguing question, uses a well defined method and clear tests to investigate the question and the results clearly align with the conclusions by the authors. I definitely recommend accepting the paper.
review_title: Review for "Category-orthogonal object features guide information processing in recurrent neural networks trained for object categorization"

Decision: Accept (Poster)