Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Good points
----
- The objective of the paper is sound: fight distributional shift in systems whose predictions
might have life-changing consequences (e.g data bias toxicity prediction models, etc.).
- The paper is well-written and easy to follow.

Bad points
----
- I don't see just how this model is "parametric". In statistics, "parametric" the adversarial
distribution is modeled as a gaussian, etc. with sought-for parameters (mean, covariance, etc.).
In the absence of that, I would have expected "parametric" to mean parametrizing the adversarial
distribution as the (softmax) output of a neural network. Neither of the above is the case in 
this paper. So, what are the "parameters" in the proposed DRO adversary ? All I can see is that
the authors do  a full search over all distributions, subject to a KL constraint (see sections
2 and 3.2).
There is nothing "parametric" about this.
- The authors say "In particular, direct gradient descent on the uncertainty set suffers from
instability due to the large variance of the gradients (Greensmith et al., 2004), and
hyper-parameter selection is not straightforward." I'm not sure about this claim (which
is one of the main premises of the manuscript. What do the authors make of this paper
for example Faury et al. (AAAI 2020) "Distributionally Robust Counterfactual Risk Minimization" ?
The authors of that paper demonstrate how to efficiently formulate and solve KL-based DRO
problems. That paper also contains both theoretical and practical insights.
- The technical contribution of the paper is negligible (if any).
- The arguments in the paper very heuristic.
- Since the paper is supposed to be empirical (see previous points), I would have expected
experiments on real datasets.


Errors
---
- Change "solve the inner-max efficient" to "solve the inner maximization problem efficiently"
- Change "$x, y ~$ " to "$(x,y) ~ $" all through the manuscript
- Eqn (5): why not take $p$ and $q_{\psi_0}$ to equal the empirical distribution (as is usually
done) in DRO ?
- In eqn defining $q_{\psi_0}$, replace $\arg\max_{q_\psi}$ with $\arg\max_\psi$
review_title: Modeling the Second Player in Distributionally Robust Optimization

final_score: rating: 7: Good paper, accept
review_content: The paper proposes to define the uncertainty set in the DRO problem as a family of parametric generative models, which is to allow more flexibility in the choice of the uncertainty set architecture. To realize this idea, the paper first proposes a new relaxation of the DRO game's inner maximization problem (with KL constraints) so as to improve the training stability. It then develops a principled approach to select the hyper-parameters of the proposed method.

Strengths:
+ The paper is well-written.
+ The proposed method is novel and important for the DRO community.
+ Experiments with real-world problems are conducted to evaluate the effectiveness of the proposed method. I particularly like the experimental analysis the authors conducted to understand the behavior of their proposed method. 

Weaknesses:
- The experiments are only on NLP tasks.

I have few questions to the authors:
1) How good the adversary model needs to be for the proposed method to perform well? In the experiments, an auto-regressive transformer model based on the GPT-2 language model is employed. What is the accuracy of this model on the train dataset of the DRO problem? Will the proposed method performance be too sensitive to the accuracy of the adversary model?
2) In the experiment (last paragraph of Section 5.1), the temperature \tau and the normalizing window k are fixed whilst the adversary learning rate \lambda is searched by grid-search. So how \tau and k are selected in practice? What is the performance of the proposed method when \tau and k vary? 

review_title: The paper proposes a novel and important DRO method, and good experiments are conducted to evaluate the efficacy of the proposed method.

final_score: rating: 6: Marginally above acceptance threshold
review_content: TL;DR: The paper makes an interesting contribution from a practical point of view, but two important theoretical concerns need to be addressed in the rebuttal for acceptance.

The paper proposes the use of ideas taken from the literature on distributionally robust optimization within a parametric framework. More precisely, the main idea is to consider only a (parameterised) subset of the traditional KLD-uncertainty sets. As this avoids the need for elegant analytic solutions (at the expense of a more brute force computation), it has the flavour of a more ‘black box’ approach towards the deployment of DRO.  Overall, I really enjoyed the way this paper was written. Purpose and use of the contributions are clear throughout, and the reader is drawn in.  I also liked the contribution and believe that the paper demonstrated its ideas to be useful. There are however two points of major concern from a more theoretical side. In my mind, these are rather substantial, and I will list them below. To recommend that the paper be accepted, these points will have to be addressed in a future version of the paper:

(1) How do you ensure that the KLD between $q_{\psi}$ and $p$ is finite? p clearly is the empirical measure (as is emphasised e.g. just above eq. (5)), but $q_{\psi}$ will be continuous. This means that the KLD between the two distributions is not defined/infinity for any value of \psi (Mismatch of support problem). These kind of problems are the precise reasons why other quasi-distances (like the Wasserstein distances) have become increasingly interesting for ML. As far as I can tell, this problem is not elaborated upon anywhere in the paper. 

(2) It is totally unclear to me why it should be viable to suddenly flip the direction of the KLD. The KLD is not symmetric and in general will not even have the same minimum. In fact, generally speaking the only time the minimum will be the same in either direction is when the KLD’s global minimum is such that $q_{\psi} = q_{\tau, \theta}$ (i.e. we can drop the KLD term for the loss in (7) completely, so that it simply equals $C$). Given the definition of $q_{\tau, \theta}$, it is unreasonable to assume that this global minimum is attained. This makes the flipping of the KLD’s direction questionable at best. Calling the outcome an ‘approximation’ is then grossly inaccurate.  (See e.g. the visualisations here: https://wiseodd.github.io/techblog/2016/12/21/forward-reverse-kl/)

Lastly, since the chief concern of the paper is the construction of new uncertainty sets, I would have liked to see two additional recent references discussed which have produced uncertainty sets purely based on moments (https://arxiv.org/abs/2007.04458, ICML 2020) as well as on general IPMs (https://arxiv.org/abs/2006.04349, NeurIPs 2020). Both these types of uncertainty sets do *not* suffer from the mismatch of support problem, and—like the famous f-divergence based uncertainty sets—have elegant dual forms. 


POST-DISCUSSION: The authors promised to clarify the two issues I pointed out in ways that are satisfactory for a paper whose main concern is practicality (as opposed to theoretical rigour). I will thus raise my score to a weak accept.
review_title: Two key issues left unaddressed

final_score: rating: 7: Good paper, accept
review_content: This paper considers distributionally robust optimization (DRO) and uses the neural generative models to characterize the uncertainty sets. To tackle the optimization challenges, several implementation tricks are incorporated to solve the minimax problem. The proposed robust method is validated on NLP tasks. 

This paper is well-written and of a good structure. Although the main idea is simple, the authors make several modifications to the algorithm to make it tractable and with performance guaranteed heuristically. To summarize, the main contribution of this paper is a new algorithm that combines standard techniques, such as Lagrangian relaxation and KL reverse, into the DRO problem with KL uncertainty sets. And this algorithm was shown to perform well under synthetic and real-data NLP tasks. Since there is no novel techniques proposed in this paper and there is no performance guarantee for the proposed framework, overall, I think this is a borderline paper due to its limitations in theoretical development and technical novelty. 

Moreover, if the main focus of this paper is on developing a new computational framework that can lead to more robust results, then the authors should compare with more benchmark methods, while I only see the comparison with ERM, Topic-CVaR, etc. For example, I am wondering is it applicable to compare with Wasserstein DRO or Huber's classical work of Total variation based DRO, or some other DRO works in the literature, so that it will be more convincing on the performance of the proposed method. 

A minor typo in the paper: in section 6, there is a duplicated "produce" in the sentence: "In such cases where good quality generative models are unavailable, or such model cannot produce produce densities efficiently".

review_title: Recommendation to Accept

Meta-review: # Paper Summary

This paper considers the problem of distributionally robust optimization (DRO), in which one is attempting to minimize a loss on the worst of all distributions that are some distance (here, measured in terms of KL divergence) from the training set. The main novelty here is that this adversarial distribution is represented as a model, with parameters that are learned jointly with the primary model.

This is an intuitive idea, but as the authors explain, attempting to implement it leads to a number of complications. One of these is that it is challenging to constrain the adversarial distribution model to be a certain KL divergence away from the training set. To address this, they write down the Lagrangian, but do not actually optimize over the Lagrange multiplier resulting from this constraint: instead, they keep it at a fixed constant value (a hyperparameter). A second, and potentially more worrisome, issue is that it is difficult to optimize the KL divergence as written--instead, they swap the two parameters, which is of course incorrect but they claim leads to much nicer convergence behavior.

They also propose a stopping condition, which terminates optimization once the robust validation loss (i.e. the validation loss w.r.t. the worst permissible distribution) stops decreasing. Normally, this would require a search for the worst such distribution at every iteration, which would be prohibitively expensive, so they propose instead only checking the distributions that have been found by the adversary during the course of optimization.

They close with a set of experiments that is nicely designed to narrow in on and explore particular details of their approach (e.g. they have an experiment that validates their stopping criterion), and have a realistic experiment on two NLP datasets.

# Pros

1. Reviewers agreed that it was very well-written, well-organized, and comprehensive
1. Good discussion of background material. The paper is very accessible
1. Intuitive idea, although the details of the approach become somewhat complex
1. Aside from the "realistic" experiment, each is designed to explore a particular facet of their approach

# Cons

1. Some reviewers were concerned that the baselines were insufficient. In response the authors added the new Hu et al. baseline (NonParam), which seemed to be satisfactory
1. While the approach is more general, one reviewer noted that the experiments only consider NLP problems. This is a minor negative point, in my view
1. One reviewer was concerned that the results were "too good", and encouraged the authors to double-check their results. My belief is that, at least on the non-"realistic" experiments (which were mostly intended to drill down into specific attributes of their approach, rather than demonstrate its overall performance), this is because the problem was constructed to perform especially poorly with a non-DRO approach
1. One reviewer was unsatisfied with the idea of swapping the parameters to the KL divergence (I share this concern). The authors clarified, both in the response and in the paper, that swapping the parameters is indeed incorrect, and may in fact be a very bad approximation to the true quantity of interest, but that the performance difference was so dramatic that it couldn't be undone. This seemed to partially satisfy the reviewer

# Conclusion

All four reviewers ultimately recommended acceptance. The major concerns were (i) that the baselines weren't good enough (which the authors addressed by adding a new baseline), and (ii) that swapping the parameters to the KL divergence results in a very poor approximation to the original KL divergence (which the authors now explicitly acknowledge in the paper, with an explanation for why they feel it is necessary). Overall, this is a nice idea, and while bringing it into practice may require more hand-waving than would be ideal (which is the main reason I suggested a poster acceptance instead of a spotlight or oral), it seems to work well experimentally, and the experiments are overall very careful and well thought-out. Additionally, the writing quality is excellent, as is the organization and presentation of background material.
Decision: Accept (Poster)