Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Paper tackles the out-of-support extrapolation problem where the input at test is out of the support of the training distribution. A bilinear model is trained to predict the output $h(x)$ using $\bar{h}(x’, \Delta x)$ where $x’$ is some chosen anchor point and \Delta x is the distance of $x$ to the anchor point. Given an unseen test point, the model is able to extrapolate (in certain situations) by using an anchor point seen in training and the distance from it. 

strength_and_weaknesses: Out-of-support extrapolation is an interesting but challenging task and the paper provides a novel way of  looking at the problem via a reformulated out-of-combination task. 

My major concerns:

1. While the goal of the work is to explore a “general setting does not assume apriori access to distribution shift or model class”, I believe that their assumption of bilinear transduction is essentially assuming apriori knowledge about the type of distribution shift or model class. While the assumption is weaker than translation invariance/equivariance, it is still strong and is able to extrapolate only in very specific cases. 

2. Many of the experiments (periodic functions, push/reach and other robotic manipulations) only require translation invariance/equivariance for out-of-support extrapolation but such models are not compared with. For point-cloud tasks, out-of-support extrapolation requires rotation/translation/scale symmetries but it is hard to judge if these symmetries are learnt automatically from the method as the input to the model is not the point cloud itself, but low dimensional PCA features and the mean of all the points. Adding these symmetries for point-cloud tasks has been explored a lot (e.g., [1]) and should be compared with. 

3. I think the approach will have problems with high-dimensional inputs because modeling of each dimension independently is not accurate for most high-dimensional inputs (like images) unless one uses PCA. If PCA or other such techniques is required for the approach, I think this should be discussed along with their approach. 



Other questions/comments:

1. The method is related to translation symmetry due to the $\Delta x$ term and is able to learn translation invariance/equivariance. But it is not clear to me how much more general it is without using additional data augmentations or PCA. For example, can one expect extrapolation to other symmetries (rotation, scaling) without data augmentations and PCA? 
2. Most of the experiments consist of symmetry due to which extrapolation is possible. What are other types of out-of-support cases that can be solved with the current approach but do not fit into translation/rotation/scale invariance/equivariance?
3. Please add the difference between train and test distributions for each experiment in the main text. 

4. In Meta-world and Adroit benchmarks, please compare with a translation equivariant model. 
5. For point cloud classification, please compare with methods that use the point clouds as inputs directly and not the low dimensional features. 
6. For Slider experiments, it would be interesting to see the ground truth slider torque dynamics for different object masses to see the difference between train and test, and understand how the method extrapolates. 



References:

[1] Thomas, Nathaniel, et al. "Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds." *arXiv preprint arXiv:1802.08219* (2018).

**Post-rebuttal**
The authors have adequately addressed my concerns and I am increasing my score. As in my original review, I think the paper provides a novel way of looking at the challenging out-of-sample problem, is well-formulated and easy to read. After rebuttal, authors have convinced me that the approach empirically extrapolates in many different settings without just using equivariance. 
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to read. The reformulation of an out-of-support problem to a potentially easier out-of-combination problem is novel and interesting. 


summary_of_the_review: While the goal of the work is to explore a “general setting does not assume apriori access to distribution shift or model class”, I believe their assumption of bilinear transduction is still strong and is able to extrapolate only in very specific cases. Most of the experiments essentially require symmetries to solve the out-of-support task (for example translation/rotation/scale invariance/equivariance) but such invariant/equivariant methods are not compared with.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the issue about the mismatch between $\mathcal{D}\_{test}$ and $\mathcal{D}\_{train}$ (i.e., out-of-distribution (OOD)). Different from a common OOD situation that the ratio between the density function of $\mathcal{D}\_{test}$ and $\mathcal{D}\_{train}$ is bounded, the authors focus on out-of-support where the density ratios are not bounded. To solve such a problem, the authors propose a transductive re-parameterization by introducing an anchor point $x'$ for a query point $x$. By leveraging the techniques from low-rank matrix completion, the authors provide a guarantee for OOS extrapolation using such transductive re-parameterization under certain conditions.
strength_and_weaknesses: Strength: The topic is interesting. The result seems novel and convincing. The presentation is relatively clear.

Weaknesses: Some places may need more explanation and justification. I have some detailed questions as follows.

1. The theorem in Section 3.2 is derived by leveraging the techniques in matrix completion. I wonder what are the key differences and similarities between this paper's result and the matrix completion result.

2. If I understood correctly, the proposed transductive re-parameterization needs $(x-x')$ and $x'$, so the training is on every pair of $x'$ and $x$. Will this cause some complexity issues during training when the number of training samples is large (as the number of pairs is almost the square of the training samples)?
clarity,_quality,_novelty_and_reproducibility: This paper is novel and clearly written. The overall quality of this paper is high.
summary_of_the_review: I like this paper in general, due to the novelty of the result and the clearness of representation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper first discusses out-of-combination scenario, which is a special case of out-of-support scenario. Then the paper propose to use anchor points to convert OOS case to OOC case for improving generalization.
strength_and_weaknesses: Strength:
1. Detailed analysis of different o.o.d scenarios.
2. Under certain constraints, the proposed method of using anchor points works in improving generalization performance.

Weaknesses:
1. It looks like to me the proposed method only works for a very limited scenario, in which o.o.s can be easily converted to o.o.c. For example, in the tasks chosen by authors, the data is almost stationary with respect to the anchor point, meaning the data distribution now is only on delta x. I wonder if any real-world o.o.d problems are like this. For example, in image classification task, the test images can be o.o.d because of unseen lightning conditions, unseen camera angles, unseen background and etc. I wonder if the proposed method can be extended to any more complex datasets where the stationary property does not hold true.
2. The discussion of o.o.c condition is a bit loose. For example the theorem on page 5 (what is an informal theorem anyway) requires the condition to be as depicted in figure 3. I think assumptions in theorem should be defined more rigorously than referring to a figure.
clarity,_quality,_novelty_and_reproducibility: The paper is in general clearly written. Discussion in o.o.c can be improved with more rigorous discussion. In general quality is good. Not reproducible as code is not provided.
summary_of_the_review: The paper has some novelty in recognizing a special case of o.o.s generalization and propose a method to tackle this case. However it is not clear to me how much percentage of real-world o.o.d problems belong to this special case. The proposed method seems to require strong pre-condition ( stationary data distribution w.r.t anchor points) to work
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies the out-of-support (OOS) extrapolation problem, whose goal is to make prediction on a sample that is outside the support of the training distribution. The authors tackle the problem by reducing the OOS problem into an out-of-combination problem, which is a special case of the OOS problem, using a bilinear transduction scheme. The authors provides generalization bound on the proposed bilinear transduction scheme, as well as experimental results on multiple synthetic problems which proves the efficacy of the proposed method. Further experimental validation on robot control problems shows that the proposed method can be effective in certain real-world settings.

The paper initially received split reviews. All reviewers agreed that the studied OOS extrapolation problem is interesting and challenging, and the proposed solution to the problem is novel. The reviewers also agreed that the method could be effective when certain assumptions are satisfied. 

However, there was a large disagreement among the reviewers on whether the assumptions the paper is based on is reasonable or not (bilinear transduction, stationary property) for real-world applications. Some of the reviewers argue that the proposed assumption may be too restrictive and does not apply to all OOS problems, as we can always find counterexamples where the proposed assumption does not hold (e.g. test images with unseen camera positions and lighting conditions for image classification tasks). On the other hand, other reviewers did not consider it as a critical weakness as long as the authors clearly describe the types of problems for which the proposed extrapolation scheme works. After an in-depth internal discussion on the topic, there was consensus among the reviewers to accept the paper, as the merits outweigh the limitations.

However, since there exist a clear limitation, I suggest the authors clearly discuss the limitation of the proposed method, and describe scenarios that can benefit from it, reflecting the suggestions from the reviewers.
justification_for_why_not_higher_score: There exists a clear limitation of the work, in that it is based on a very strong assumption (regularity in the task space). Also, the paper did not fully explore what types of out-of-support extrapolation problems can benefit from the proposed method. 
justification_for_why_not_lower_score: The paper proposes an interesting idea that could lead to further research on out-of-support extrapolation problems.
Decision: Accept: poster