Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an algorithm to compute a DRO objective where the uncertainty set is defined with deviations in moments of the representation. The algorithm allows one to compute the minimax DRO loss using a primal-dual formulation and reasoning about the stationary points.
strength_and_weaknesses: The derivation in the paper is theoretically interesting.

My main concern is this:

While the paper guarantees $\theta$ is learned well with $\phi$ fixed, it is unclear to me why I can learn a representation with the given objective. The reason is that changing $\phi$ changes the uncertainty set for the setup DRO problem! Without the ability to use a fixed uncertainty set from some known restriction, what is the value of the proposed method? The experiments seem to support this concern as the authors say "All the methods become vulnerable for large pnoise possibly because of the backbone neural network model."


Weaknesses/Questions:
1. Both corrollary 2 and theorem 3.2, along with the stated uncertainty set are from ((Li et al., 2022). Why do the authors claim " We propose a distributionally robust probabilistic supervised learning method"? The main contribution seems to be the algorithm.
2. What can go wrong when let $\beta=0$?
3. Theoretically, it seems like when learning $\phi$, it seems possible to end up in a minima where $\phi=0$. Can the authors explain why this won't happen when the objective is optimized?
4. Is mixing log-likelihood and brier score used in an application? Can the authors point to a case?
5. The experiments in table 1 do not seem to favor the proposed method much; softmax is better or similar.
clarity,_quality,_novelty_and_reproducibility: The paper is cleanly written by the novelty seems to be the algorithm to compute the DRO alone, which I believe is a useful thing.
summary_of_the_review: While I think the ability to compute minimax loss is great, I am unclear about some details and what the value of the paper is.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present a method for learning a model that outputs probabilities wherein the output probability is the output of an optimization over an uncertainty set rather than, say the result of a softmax. They show how to learn a model using this approach and compare the method to simple alternatives on several datasets.
strength_and_weaknesses: See below.
clarity,_quality,_novelty_and_reproducibility: See below.
summary_of_the_review: Overall, I found the technical aspects of this paper clear and well-written, and the methods and evaluations reasonable. With that said, I am still very unclear on the motivation for this method, especially given that if failed to outperform simple softmax on any of the evaluation problems. My concerns are below:

Major:

1. I did not follow the motivation for this method and strongly recommend that authors work to make the intuition clearer in the introduction.    Further, the experiments did nothing to make this motivation clearer. If anything, I left the paper wondering why I would use this method when a simple softmax appears to perform nearly identically. I recommend design their experiments to highlight and provide intuition for the differences between existing approaches and the proposed method and, if no differences exist, explain why a negative result is interesting. If there are no differences and no value in a negative result then it is hard for me to see the significance of the method.

Minor:

2. Define "proper" in the intro.

3. Contributions: "Extensive" seems strong here. Instead of saying "we performed evaluations", describe what your evaluations show and how that contributes to our knowledge.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers distributionally robust probabilistic supervised learning. The ambiguity set is constructed to include distributions that share the same marginal with the empirical distribution and are no more than ε away from the empirical in terms of first-order feature moment divergence. The strong duality is shown such that the primal DRO problem is equivalent to a regularized ERM problem. The authors characterize the solutions to the proposed method and present an efficient algorithm for specific losses. Moreover, neural network representations are incorporated and the DRO problem can serve as a differential layer to enable end-to-end differentiable learning. 


strength_and_weaknesses: strength:
- A DRO version of probabilistic supervised learning is proposed under first-order moment ambiguity sets. 
- The DRO problem can be incorporated into neural networks and perform end-to-end differentiable learning


Weaknesses
- The theoretical result is weak, and the technical contribution is not clear -- in all theorems and Propositions, the reference (Li et al., 2022) is cited, but there is no clear justification for the differences with (Li et al., 2022) 
- The empirical study on real-world data is not enough, especially the baseline methods compared are very limited, only neural network models with the softmax and the spherical softmax function.

clarity,_quality,_novelty_and_reproducibility: This paper is well-organized and easy to follow. I found that some references might be related to this work but seems missing in the paper:

Shafieezadeh Abadeh, S., Mohajerin Esfahani, P. M., & Kuhn, D. (2015). Distributionally robust logistic regression. Advances in Neural Information Processing Systems, 28.
Lee, C., & Mehrotra, S. (2015). A distributionally-robust approach for finding support vector machines. Available from Optimization Online.
Chen, R., Hao, B., & Paschalidis, I. (2021). Distributionally Robust Multiclass Classification and Applications in Deep CNN Image Classifiers. arXiv preprint arXiv:2109.12772.
Amos, B., & Kolter, J. Z. (2017, July). Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning (pp. 136-145). PMLR.
Agrawal, A., Amos, B., Barratt, S., Boyd, S., Diamond, S., & Kolter, J. Z. (2019). Differentiable convex optimization layers. Advances in neural information processing systems, 32.


summary_of_the_review: Overall this paper is well-written and considers a problem that can be of broad interest to the machine-learning community. The results are standard -- such as the duality with regularized ERM and the risk bounds. However, the theoretical results look weak and most importantly, the authors didn't explain their technical contribution clearly, as compared with the work (Li et al., 2022). Moreover, the result in Corollary 2 is only limited to the case epsilon=0, which is the standard ERM, not the DRO problem. 
Also considering that the numerical results are also limited and do not compare with many baseline methods, I think this is more like a borderline paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper is concerned with probabilistic predictions: Instead of predicting a class, the prediction is a distribution over the label space. The paper proceeds by using a probabilistic loss functional, and consider an adversarial training approach. The proposed method is a minimax problem, with perturbations in a moment set that is specified using the feature function. The paper then provides the dual formulation and integrates this formulation into a gradient descent algorithm (Algorithm 1).
strength_and_weaknesses: The paper lacks clarity and mathematical rigor:
1. Before Proposition 1 in the main text, the authors claim that “the ambiguity set ($\mathcal{A}(\mathbb{P}^{\mathrm{emp}}$) is a compact convex set.” However, no proof is provided.
2. In the proof of Proposition 1, page 13, line 6, the authors invoke Sion’s minimax theorem. However, the authors did not verify that the loss function is lower-semicontinuous in $\mathbb{P}_{Y|X}$ and upper-semicontinuous in $\mathbb{Q}$.
3. Also, in the proof of Proposition 1, in the middle of page 14, the authors claimed that “Again by strong duality, we can rearrange…”. However, no justification for strong duality has been provided. 
Further, I feel that the use of $I_{\mathcal{C}}(\cdot)$ is unnecessary. In fact, the indicator function is non-smooth, non-convex, which complicates the whole process of applying duality results.
4. The notations of $\min_{\mathbb{P}}$ is really confusing in the statement of Proposition 1. The expectation is already taken with respect to $P_x^{emp}$, so what is the $\min_{\mathbb{P}}$ in equation (3) concerning? Shouldn't $\mathbb{P}$ be $\mathbb{P}_{Y|X}$?
5. The results of this paper seem to follow from Li et al. (2022) [Proposition 1, Corollary 2 and Theorem 3], however, up to today (Oct 23), I could not find the manuscript of Li et al. (2022) on arXiv or on any public domain (openreview). At this point, it is difficult for me to judge the novelty of the manuscript.
6. Using the setup of this paper, any distribution can be conveniently represented as a matrix because the space $\mathcal{Y}$ is discrete and the space $\mathcal{X}$ can be reduced to a finite set supported on the training data. From this perspective, the problem becomes uninteresting because, in the end, we are only perturbing the elements of a matrix. The problem only becomes interesting, in my opinion, when $\mathcal{Y}$ is continuous or we allow perturbations of the marginal distribution on $X$.
7. The numerical experiments can not demonstrate that the proposed method is superior than existing approaches. The experiments are also conducted with $\varepsilon = 0$, which implies no moment robustness.
8. The authors should make clear how a probabilistic prediction is made at test time, when only $x$ is given and there is no information on $y$.
clarity,_quality,_novelty_and_reproducibility: - Unclear mathematical exposition 
- Limited novelty when compared with Li et al (2022)
summary_of_the_review: - The paper lacks mathematical rigor, which severely hinders my understanding and appreciation of the results. 
- There is a significant overlap with Li et al (2022) which requires clarification.
- The experiment does not demonstrate the benefit of the proposed method
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a method for predicting distributions in the DRO setting. The reviewers had several concerns, with a particular focus on lack of clear motivation, and empirical results which do not demonstrate an advantage for the method. 
justification_for_why_not_higher_score: The paper did not receive support from any of the reviewers. The method is not properly motivated and shows no empirical advantage over simple baselines.
Decision: Reject