Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary:  This work studies offline data-driven model based optimization (MBO), where the goal is to produce optimized designs, purely using a static dataset. We proposed to view offline data driven model-based optimization through the lens of domain adaptation, and proposed a method that trains a model of the objective with an additional regularizer to promote invariance between the representations learned by the model in expectation under the training data distribution and the distribution of optimized points. 
strengths_and_weaknesses: Strengths:

1.This paper studies the policy learning problem under the distributional shift. The total goal is to learn an invariant policy (representation) which maximizes some specific objectives (specific rewards) in the varying environments. Some related work is provided to support their novelty. 

2.Since the authors assumes the variability of marginal distributions (e.g., covariate shifts), they proposed the IOM method with MMD divergence between training and target features. Some theoretical results are provided to show the performance guarantee of the IOM method (a lower bound). 

3.Extensive experiments are provided to verify the effectiveness of the proposed method. Some ablation studies are provided with details.

Weaknesses:

1.I feel confused on the novelty of the proposed method. From my viewpoint, this paper presents a policy learning method with distributional shift. However, offline policy learning methods with distributional shift is already a popular topic. Moreover, it is not rare to use some DA tricks to improve the transferability of the learned policy across different environments. 

2.Related work should distinguish this paper from previous policy learning methods. Their motivations are settings are very similar. 
questions: Please refer to "Weakness".
limitations: They have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper develops an offline model-based optimization method based on learning invariant representations, a popular approach in domain adaptation. The authors frame the invariant representation learning as a bi-level optimization problem and propose a practical algorithm for solving it. The authors also develop an effective offline hyperparameter selection procedure that's empirically shown to lead to the smallest sub-optimality gap. Experiment results show that the proposed offline MBO method outperforms previous approaches. 
strengths_and_weaknesses: ### Strength
1. The proposed offline model-based optimization method based on learning invariant representation is well justified and empirical results show strong performance.
2. The paper is well-written in general and most parts are easy to follow. 
3. The paper discussed an offline tuning strategy that's critical for real-life applications.

### Weakness
1. The clarity of the paper could be further improved. Please see my comments in Questions section.
questions: 1. Figure 1 is confusing. Why is conservatism shown to significantly overestimate the function value in the high-data region? Could the authors provide the reasoning on why this could happen? 
2. In line 92, the authors state " IOM not only does not require a generative model" but in line 62-63 "we instantiate IOM using the $\chi^2$-discrepancy measure via a least-squares generative adversarial network". These two statements are conflicting with each other. Could the authors clarify?
limitations: The authors addressed the limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the offline data-driven model-based optimization problem, where the dataset is static and doesn't allow for collecting new data. The main challenge is to prevent the optimizer from finding an out-of-distribution design. To achieve this, the authors cast this problem as a domain adaption problem and focus on representation learning. To deal with the out-of-distribution solutions, the authors learn an invariant representation which ensures the representation of out-of-distribution designs is uninformative for distinguishing good solutions from bad ones. As a result, the function value of out-of-distribution designs will stay close to the mean of the function values of training data. Consequently, it prevents the optimizer from selecting those designs. The authors also provided analysis, showing that the design found by their methods can compete favorably against the distributions of designs in the training dataset in the worst case. Empirically, the proposed method performs can attain improved designs over the best designs in the training data.
strengths_and_weaknesses: Strength:

1. The entire paper is well-structured and easy to follow. In particular, the method of learning an invariant representation to combat the out-of-distribution designs is very well-motivated. In constrat to methods that modify the constraints, the proposed method is compatible with nearly all the off-the-shelf optimizers. 

2. The authors also provided many intuitive explanations for their method, which were very helpful for understanding the paper. For example, why will the invariance lead to "average" values for those out-of-distribution designs? 

3. Though not significant, the authors provide a worst-case analysis for the performance of the method.

Weakness:

4. From my perspective, I feel that the method might be an overkill to the problem. Incorporating representation learning introduces extra but unnecessary complexity to the problem: as a result of the invariant representation, the function value of out-of-distribution designs will be penalized to be the "average" function values of the training data. So, why not directly learn an auxiliary function to penalize the function value of those out-of-distribution designs? This should be an easier task than learning an invariant representation.

5. The theoretical analysis is kind of straightforward from the existing literature. For example, similar results have already been shown in offline reinforcement learning (see Theorem 4.4), e.g., Yu et al., 2020. 

Minor:

6. For Assumption 4.1, "there exists a representation \phi ..." seems redundant to min_{\phi, g}. I would  remove \min_{\phi, g}.

7. I didn't fully get "a novel connection with domain adaptation" (Line 51 - 52). Can you please highlight the "novel connection"?


Overall, I think this is an interesting work, and it might be interesting for people working on offline reinforcement learning. This paper provides some promise for attacking distribution-shift from the representation learning perspective. 


Yu, Tianhe, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. "Mopo: Model-based offline policy optimization." Advances in Neural Information Processing Systems 33 (2020): 14129-14142.
questions: See the strengths and weaknesses part.
limitations: See the strengths and weaknesses part.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a model-based optimization method, which handles the distribution shift challenges through the perspective of domain adaptation. The key idea is to regularize the learned model by treating the training data as the “source domain” and the optimized designs as the “target domain”. The invariance optimizer forces the distribution shift between source and target to be minimal at the representation level, which ensures the out-of-distribution points will not be erroneously overestimated. Overall, the idea of this work is novel to me and the intuition behind is convincing to some extent. Although I didn’t check all details, the proposed bi-level optimization problem and the corresponding practical algorithm appear to be technically sound. 
strengths_and_weaknesses: Strength points:
1. This paper contributes some new ideas. The key idea is formulating offline model-based optimization as domain adaptation, which may have a broad impact.
2. This paper is clearly written and well organized.
3. The practical implementation of IOM algorithm and the offline tuning is detailed.

Weaknesses:
Please refer to Limitations.
questions: 1. The distribution shift in MBO is not clearly defined or illustrated in this paper. In domain adaptation, the source and target domains have different data distributions but share the same feature space. In this paper, the target optimized designs are obtained by optimizing the learned model with respect to the inputs via gradient ascent, similar to adversarial samples. My main concern is whether the adversarial samples equivalent to the case in domain adaptation?  
2. In section 6.1-Evaluation protocol, how do you test or deploy the produced designs?

limitations: 1. The bi-level optimization objective is not convincing to me. Why do authors formalize IOM training as a bi-level optimization problem with respect to the optimal distribution and the learned model parameters simultaneously? The standard approach of learning a naïve model, which is then optimized via gradient ascent with a invariant regularization term also makes sense to me. Explanations on the necessity of bi-level optimization are welcome.
2. The author claimed that invariance can effectively handle distribution and conservatism is not necessary. Does it mean IOM is consistently better than conservatism? Any insightful explanations? I suppose that the conclusion needs more examples to support.
3. The offline tuning strategy needs multiple runs of IOM with a set of hyperparameters, which may be time-consuming. The computational efficiency should be considered. 
4. In section 6.2, more details on oracle turning should be added. Besides, for suboptimality gap in Figure 3, the comparisons with COMs are not fair, since it only considers two naïve case for COMs, more tuning strategies for COMs needs to be evaluated.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper considers the problem of model-based black-box optimization given a static dataset. The main concern is not to report a design that is influenced by model overestimations. The authors propose to formulate this problem as domain adaptation and propose an algorithm IOM that formulates the problem as bi-level optimization and enforces invariance at the representation level by using a specifically designed regularizer. Experimental results show the comparison against other methods for model-based optimization.   

strengths_and_weaknesses: I had a hard time reading this paper and this is the primary reason for not being able to grasp the main contributions. Overall, I think that the text can be significantly improved. Even the sections that were supposed to convey intuition were confusing to me. For example, the “Intuition (Line 156)” section did not convey intuition and confused me even more. I did not understand sentences such as:
* Such an invariance can be enforced when $\mu_{OPT}$ is sufficiently far away from the training distribution.
* .. if $\mu_{OPT}$ is too far away from the training distribution. 
* This would pull the optimizer closer to the training distribution. 

The abstract should do a better job of highlighting the main contributions better. IOM is used in the abstract, but it is unclear what IOM stands for. 

Not sure what the main message of Figure 1 is. I think it does not convey intuition well (the last sentence is unclear). 
Things should also be more formal. $\mu_{OPT}$ and $J(\mu)$ have the same definition in Eq. 1. It would also be helpful if the distributional shift that the paper talks about is mathematically formalized. 

I was also unable to comprehend the theoretical result. If the circle is positive and the star is minimized, then mu_{OPT} is better than mu_{data}. I’m not sure what the utility of such a result is. Again, why do we care about mu_{data} when optimizing, and not only consider the best point/value given the noiseless setting. 

Overall, I feel that the entire method is overly complicated. I am not convinced that we need to solve the bi-level optimization problem (which is known to be notoriously hard). While I understand why one might be interested in obtaining “soft” allocation (i.e., distribution over solutions), I do not understand why we need to look at the entire training data distribution and not the best point/value within the given training data. 

The first experiment is also unclear to me. Do you compare it against the best point in the training dataset? How can CMA-ES and REINFORCE be conventional Bayesian Optimization methods?
I’d suggest looking at more relevant arXiv:2002.09038 and similar model-based (BO) robust works. Another question is how fitting a GP to the training data, defining confidence bounds, and outputting the design that corresponds to the max lower confidence bound performs in this experiment?

Some minor comments:
– 41: Typo “invariance”
– 127: Is this Dirac $\delta$? 

questions: Please see the previous section.
limitations: The authors only stated one limitation that is again very vague in the conclusion.
review_title: null

Meta-review: metareview: Reviewers are all positively inclined for the paper. Most of them have stated that the paper
proposes some novel and relevant ideas with theoretical support and empirical evidences
of theirs soundness. As such, we think that the paper can be accepted to the conference, and
expect some revisions that would improve clarity of the work. 

confidence: Certain
Decision: Accept