Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The problem of generalization to out-of-distribution (OOD) samples is key to most problems in AI. The authors address this challenge in the context of IQ-like tasks by introducing a method called FINE, for Functional indirection neural estimator. The authors are inspired by the idea of indirection, trying to connect two different representations and using one to learn or interpret the other. The authors show that the proposed architecture does well in those IQ tasks when considering images that are different from those in the training set upon applying the same rules as those in the training set. 
strengths_and_weaknesses: Strengths

The question of generalization to OOD is fundamental to intelligence problems. 

The idea of indirection is very interesting and the authors propose a very nice implementation of this idea. 

Figure 4 is interesting in showing clusters in weight space. It would be interesting to show whether other models reveal the same property or whether this is specific to the proposed architecture.

Weaknesses

The term OOD is often used in a very loose manner. This study is a good example. To really define OOD, one should define D (i.e., the distribution). What I understand the authors are doing is selecting some “classes” in Omniglot and testing on other classes. Or selecting some “classes” in CIFAR and testing on other classes. This seems pretty standard. But the question is how much OOD is really being tested here. Imagine that your training class is letter “i” and your test class is letter “l”. Those letters are really similar. Sure, they are different “classes”. The issue is how similar the training set is to the test set. Can the authors train with Omniglot classes and test with CIFAR classes or vice versa? That would be impressive OOD! 

The word generalization here only applies to testing with somewhat different images. The hard challenge in IQ tests is to generalize to novel rules.  

questions: The paper starts with the assertion that humans can generalize to OOD. I am curious about the evidence for this. Sure, everybody says this sort of thing. But what kind of real quantitative evidence do the authors have for this statement?

On line 85, the authors assert that the models are required to recognize the objects and figure out the relation between them. What is the evidence for any of this? Sure, from introspection, humans may reason about the task in this way, but this does not mean that this is what the models are *required* to do. This is an example of anthropomorphizing.
(In contrast, lines 92/93 are trivially true. Yes, models must use the training data, there is no magic! )

The conclusions are similarly based on introspection only. The authors state that FINE reliably figures out the hidden relational pattern in each IQ task and is able to solve new tasks but the authors do not show any of this. The authors show that the model works better than a few other models when considering somewhat different images between the training set and the test set, which is pretty cool in and of itself. But there is nothing less and nothing more than that here. Other than somewhat different images, where does the paper show "new tasks" or "discovering relational pattern"?  




limitations: The authors spell out some limitations mostly to highlight other interesting problems that could be addressed. The key challenges mentioned above are not discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a mechanism for functional indirection (FINE) in neural networks to achieve OOD generalization in abstract reasoning tasks. FINE proposes to dynamically select weights for a neural network backbone for a particular data input-output pair and use those weights to make prediction for an input that share similar hidden rule. The weights are selected from pre-defined key-value memory which is comprised of the weights that spans the space of possible functions. FINE is like a constrained form of hypernetworks where weights of the main (backbone) network are determined by another network. In this paper, this role is played by function composer $\phi$ which finds optimal weights and their arrangement for the main network using a limited basis of weights in the memory.

The paper further introduces a new abstract reasoning dataset based on Omniglot and CIFAR100 for evaluation.
strengths_and_weaknesses: **Strength:**
- The proposed methodology of achieving functional indirection is novel and interesting.
- The paper is well written and easy to follow.
- The method is shown to achieve good performance on OOD generalization across unseen categories of the datasets used for training.


**Weaknesses:**
My main concerns revolve around the evaluation of method.
- The paper goal is to solve OOD abstract reasoning and analogy making. However, the method is not evaluated on the known abstract reasoning benchmark of PGMs. Instead authors propose similar but less complex tasks for evaluation.
-  The newly introduced datasets test generalization only for the unseen characters of the training dataset but not for unseen rules that could be extrapolated or interpolated. See me related comment in the questions section.
- The proposed method, FINE, implicitly selects the weights based on the transformation (hidden rule) of the input-output pair. This is highly similar to [1] where a mixture of expert (networks) explicitly compete to explain image transformations on MNIST and Omniglot. [1] also showed huge benefits of mechanism-specific function selection for OOD generalization. I believe a comparison or even explanation on the similarities among the method would be good to have in the paper.
- Since this framework focuses on the generalization of image transformation mechanisms across unseen classes (of Omniglot and CIFAR100), I would encourage authors to test FINE model trained with Omniglot transformations on MNIST data with transformation, which was shown in [1]. 

[1] https://arxiv.org/abs/1712.00961
[2] https://arxiv.org/abs/1807.04225
questions: - Why create new set of IQ tasks when a similar benchmark of PGMs studying different forms of rule based OOD generalization exists already? It fulfils the criterion of providing hints of the hidden rule with few images and then using that to infer the predictions.
- Why are tasks restricted to identifying image transformations only? Would the method not be able to infer complex relational reasoning tasks as demonstrated in PGMs?
- How are the basis of network weights restricted such that it only spans to a limited set of functions that can be used by the backbone?
- Wouldn’t the limited basis of network weights restrict generalization to only observed rules and their combinations thereof? How would you scale to unseen rules?
- Some of the references in the related work section are missing for e.g. Neural Interpreters [3] uses attention mechanism to recompose functional modules for each input-output pair and test their method on abstract reasoning tasks. Similarly, Switch transformers[4] switch modules based on relevance to inputs.
- The model and the evaluation share similarities with meta-learning framework where a few input-output pairs are provided to adapt the network weights (e.g. in MAML) and the resulting model is used to make inference on the unknown inputs coming from the same classes. Analogous to the FINE datasets, the classes would be the hidden rule that input-output share. Curious to know what authors think about this and would it be possible to make a small experiment with Omniglot?
- Why is similarity metric chosen to identify correct output $y’$ instead of using normal cross-entropy?
- Do the newly introduced datasets test generalization only for unseen characters or also unseen rules? For e.g. the model could be trained for detecting character rotation from 0 to 90 degrees and tested on the characters rotated from 90 -270 degrees.

[3] https://arxiv.org/abs/2110.06399
[4] https://arxiv.org/abs/2101.03961
limitations: The main limitations seem to be around experimental evaluation. I would be happy to increase my score if authors address those concerns.


There doesn't seem to be any obvious negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper addresses the out-of-distribution generalization of deep learning models in IQ visual tasks involving extracting geometric transformation between a pair of images and applying the extracted transformation to a new image. It presents a memory-augmented neural architecture and on-the-fly model parameter retrieval from the memory to achieve OOD generalization in functional spaces.

strengths_and_weaknesses: Strengths:
+ Unlike previous work that performs indirection and analogy-making in the data spaces, this paper proposes a mechanism for OOD in functional spaces.

+ The weights of backbone can be determined on-the-fly using (input, output) query to retrieve from a memory.

Weaknesses:
- The proposed method may need a large memory to store trainable weights for very deep backbones.

- The authors tested the proposed method only in functional spaces related to geometrical transformation. It's not clear whether the proposed method would work
for other transformations.
questions: It's not clear whether the proposed method would work for other transformations.
limitations: Yes, the authors have addressed the limitations of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces FINE, a method for achieving out of distribution generalization through analogy making and indirection in the space of functions. FINE makes an analogy from a given input and output example to infer the function that ties them then uses indirection to approximate the function by composing a set of functions saved in memory. The paper introduces a visual reasoning dataset for evaluating out of distribution generalization. The dataset is based on standard vision benchmarks, CIFAR100 and Omniglot. Each sample consists of an input-output pair used as a cue for the transformation, an input image and 4 choices. FINE performs better than several comparable models over all functions and function combinations in terms of accuracy and sample efficiency. Ablation experiments highlight the importance of the number of layers and the memory size.
strengths_and_weaknesses: Strengths:
- The paper takes inspiration from features of human intelligence and combines them in a novel way.
- It proposes a formal framework that uses this technique to mitigate the lack of OOD generalization in neural networks. It shows that the proposed method can abstract image-level transformations in the latent space and that the abstraction allows the model to generalize transformations to novel images.
- The paper proposes a novel use case in visual reasoning that highlights the new framework’s advantage. The method is compared to several relevant baselines.
- The paper is clearly written overall. The code is provided for reproducing the results.
- The authors discuss the certain limitations of their approach.

Weaknesses:
- To learn the proposed task, the model has to identify and build the transformation. Thus, the OOD samples should be different functions, not only different input images (for example, can the model learn rotations of angles between 0 and 90 degrees and extrapolate to angles of 90-180 degrees ?)
- To provide a fair comparison to baselines, all models need to be adapted to this framework. The training setup and hyperparameters used for training these models are not explained in the main paper or the supplementary material. This information is crucial for explaining their performance.
- The paper doesn’t discuss other applications for this framework. OOD generalization is an active research topic in ML and image classification is among the main applications (adversarial examples and domain shifts and noise corruptions are OOD examples). How can this framework benefit OOD generalization in this task?
- Although the use of NICE layers is intuitively motivated, it is not a necessary building block for FINE. Its use is motivated by the reversibility of certain transformations, which is a design choice in the dataset. Since most baselines (excluding hypernetwork) are not equipped with NICE layers, it would be a fair comparison to provide results on all models without NICE layers.
- The choice of using 2 NICE layers instead of one layer is not fully explained.
- The $y_t$ vectors used in the analogy step are output from $\gamma_t(y)$. The motivation and significance of this choice are not explained.
- In the first 2 paragraphs of subsection 3.1, H1 considers a unique output for each input while H2 considers the possibility of several outputs based on the transformation. The second hypothesis is not different from H1 if the target transformation is supplied with the input. This is the choice made indirectly in the dataset by providing an input-output pair as a hint for the transformation. It would be more concise to write this section without using hypotheses. It’s simpler to explain FINE’s advantage when the task requires learning many functions with a single backbone.


questions: These questions are directly related to the weaknesses mentioned above.
- The paper should clarify that the OOD generalization claim concerns the input images that undergo the transformations, not the transformations that are learned by the model. Evaluations on unseen transformations would be a valuable addition to the paper. 
- The training setup for baseline models should be described in the supplementary material. Ideally, all models should be capable of inferring the transformation.
- As mentioned above, It would be a fair comparison to provide results on all models without NICE layers.
- Can this framework be adapted and evaluated on OOD generalization in image classification or other tasks in ML ?
- The first 2 paragraphs of section 3.1 could reformulated for clarification. As mentioned above, it would be more concise to write this section without using hypotheses. It’s simpler to explain FINE’s advantage when the task requires learning many functions with a single backbone.
 

limitations: The authors address certain limitations of the paper but address no negative social impact.
review_title: null

Meta-review: metareview: This paper tackles OOD generalisation through a mechanism for analogy-making in functional spaces rather than the data space. It involves construction of a functional framework that maps inputs to outputs---by abstracting the transformation between inputs and outputs through a separate (hyper)network which provides the weights of the mappings. It further contributes a benchmark for evaluating OOD on IQ tasks.

The reviewers agree that the paper tackles an interesting and relevant problem with the perspective of functional indirection, and the IQ task does appear challenging.

The primary outstanding issue with the work appears to be with what class of models they  compare against---there exists work in meta-learning (e.g. CAVIA, MAML) that ought to be discussed. If they are not appropriate it is crucial that this is explained, because from a functional perspective, they are very similar. Indeed as Reviewer kR5Y points out there are clearly relevant pieces of work that out to figure as comparisons here.
Simply speculating that 'it is not clear...can deal with unseen transformations' will not do; this ought to be established in order for the paper to stand strongly on its own.

And while I'm somewhat inclined to buy the proposition that PGM etc can have 'shortcuts' but I would perhaps have still expected evaluations on these in addition to the IQ task setup to mitigate claimed deficiencies in benchmarks like RAVEN or PGM.

The authors also provided additional experiments over more extreme OOD settings as well as tempered some imprecise statements to address reviewer concerns, which was good.

On balance, though it appears as if the paper has more merits than issues, and most of the issues raised could be addressed with some work. I would strongly urge the authors to actually make the edits for comparison and incorporate the additional experiments over existing benchmarks from the rebuttal into the manuscript, as requested by the reviewers.
confidence: Less certain
Decision: Accept