Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a distributionally robust logistic regression with mixed-type features. 
Optimization method is considered. Some properties related to complexity analysis were given. 
A number of experiments were performed on classical datasets.  

strengths_and_weaknesses: Strengths: 
1.	It considers the expansion of recent distributionally robust logistic regression (LR) with continuous features to that with mixed features. 
2.	The complexity analysis is given. In particular, it proves the the non-existence of regularized version for the formulation derived from Wasserstein distance. 

Weaknesses: 
1.	It appears to be an incremental derivation from existing distributionally robust logistic regression (LR) with continuous features. 
2.	In terms of experimental results, the proposed formulation does not appear much better than the classical LR in terms of accuracy. Actually, it is not clear there is any significant advantages when compared to LR (Please see next comment). Without significant improvement, the new formulation is more like a conceptual exercise. 

3.	The comparative evaluation in lines 282-287 is not so convincing. Currently, only the number of best performance is counted; 

“The table shows that for the unregularized model, both of our distributionally robust logistic
regression achieve the lowest classification error in 78% of the instances, whereas the classical logistic
regression achieves the lowest classification error in 61% of the instances. For the regularized model,
the results remain at 78% (our model) vs. 61% (classical logistic regression). Overall, the non-robust
models failed to achieve the lowest classification error in three of the instances, whereas the robust
287 models missed the lowest classification error in only one of the instances.”

This is analogous to ‘bean counting”. 
However, the variance is not considered.
It would be desirable to show variance  information. Maybe some statistical test can be used for judge if there is any significant difference between the LR with the DROs, or between the regularized r-LR with the regularized r-DROs. 


questions: The presentation is  clear. The discussions are pretty clear. 
The strengths and weaknesses can be found in above comments. 
No questions are raised here. 


limitations: Yes
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper considers the mixed feature distributionally robust logistic regression problem, which extends the setting in [1] to the mixed feature (i.e., the feature space can be divided into the numerical (real number) and categorical (integer) parts.  The main contribution of this paper lies in deriving an equivalent dual reformulation of this new model and further developing tractable methods to address it. More specifically, by fully exploiting the problem-specific structure arising from the log-loss, the authors fully utilize its exponential conic representation and provide a Column-and-Constraint Solution Scheme to solve it. Finite steps convergence guarantee has also been established. Extensive experiments have been shown to demonstrate the effectiveness of its model performance.  

[1]: Shafieezadeh-Abadeh, Soroosh et al. “Distributionally Robust Logistic Regression.” NIPS (2015).

strengths_and_weaknesses: **Strengths**: 

The mixed features considered in the paper are less explored in distributionally robust optimization. This paper takes a first step towards filling this gap. The paper is easy to follow and well-organized. Comprehensive experiments have been given to illustrate the extension.  Notably, Theorem 3 seems very interesting to me. They point out the absence of reformulation as a regularized problem, while this property holds for many DRO problems.

**Weakness**:

There are three major concerns: 

1. The novelty of both modeling and techniques is moderate. The proposed model is a natural extension based on [1]. The dual reformulation is also relatively standard if we adopted the method provided in [1].  

2. All methodologies developed in this paper highly depend on the log-loss and specific cost function (see theorem 2 for details), which are rather restrictive. In my opinion, the impact of this paper on broad ML tasks is limited. 

3. On the computational side, although the paper gives a polynomial-time solvable method to address the resulting dual reformulation, it still relies on certain off-the-shelf solvers, which will potentially suffer from the scalability issue on medium or large-scale datasets. More importantly, for the experiment results, the proposed mixed feature model does not gain substantial improvements over other baselines (i.e., vanilla logistic regression or distributionally robust LR) in terms of test accuracy (i..e, see tables 1 and 2),  but instead cost too much computational effort. 

For the above reasons, this paper is strictly below the neurips acceptance level. 

questions: No 
limitations: The computational paradigm and theoretical results (i.e., polynomial solvable）is restricted to the logistic regression case.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the distributionally robust logistic regression problem with mixed (numerical and categorical) features. The problem is proved to be solvable by a polynomial-time solution scheme. In particular, this paper develops a column-and-constraint approach to solve the problem. This paper also shows that the distributionally robust logistic regression problem is not equivalent to regularized logistic regression as in previous works with numerical features only. When categorical features are present, this paper proves that the problem is not equivalent to regularized problems for any regularizer.
strengths_and_weaknesses: Strengths

1. This paper studies distributionally robust logistic regression with mixed features, while previous works mainly focus on numerical features only or simply treat the categorical feature as numerical ones. 
2. This paper proposes a column-and-constraint approach to solve the problem efficiently.
3. This paper shows that the problem with mixed features is not equivalent to regularized problems, as is commonly the case for distributionally robust logistic regression with numerical features.
4. Numerical results on multiple real datasets are provided.
 
Weaknesses
1. The main advantage and motivation of not dealing with the categorical features directly are only shown in the example of a worst-case distribution in Figure 1, which might be insufficient and the results could be stronger if the author could have certain theoretical comparisons. 

questions: I have a question regarding the example shown in Figure 1. By treating the categorical features as continuous ones, the authors claimed that the worst-case distribution places non-zero probability on an extreme scenario 15, 312, I am wondering would such scenarios be avoided when we restrict the range of the continuous features, like within [-1,1]. Also, it would be better if the author could comment on some theoretical comparisons between the performance of the Mixed-feature model method and the continuous-feature model in sec 2.2.
Typos:
1. In algorithm 2, delta \in \{0,1,2,...\}, a typo in the first comma.
limitations: It seems that the authors do not mention limitations of the proposed method.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: To overcome the non-robustness in classical logistic regression, the authors consider the distributionally robust logistic regression by introducing an ambiguity set defined by the popular Wasserstein distance.
In contrast to the previous literature, the proposed method focus on mixed features. Despite the optimization problem itself being exponential-size for generic losses, the authors show that it can be solved in polynomial time using the column-and-constraint generation scheme.
Theoretically, they provide finite-sample and asymptotic performance guarantees for the proposed method. 
Experiments on synthetic and real-world datasets show the accuracy and efficiency of the proposed method in both unregularized and regularized versions.
strengths_and_weaknesses: In general, the paper is well-written and easy to follow, the motivation to consider categorical variables is clearly stated, and the theoretical results also seem to be correct.
As far as I have read, the work provides novel discussions and results which are not captured by the cited literature. 
However, I have some concerns about the numerical experiments and also some suggested corrections/additions, which are presented below.
questions: **Major Comments**

* My major concerns are about the experiments in Section 4. 
1. Sections 4.2--4.3: I do not find the proposed DRO (resp. r-DRO) performs much better than LR (resp. r-LR), while DRO (resp. r-DRO) might require a longer runtime than LR (resp. r-LR). The authors should provide comparison results on both performance and time to show the accuracy-time tradeoff.
2. As the authors discussed in Section 1, the motivation to consider the robust version of LR is that classical LR is prone to overfitting, especially in face of outliers or distribution shifts. However, this kind of setting has not been considered in experiments. The authors should provide simulations or real data analysis on these specific settings to show the advantages of DRO (resp. r-DRO) over classical LR (resp. r-LR).
3. Sections 4.2--4.3: Only classical logistic regression (with/without regularization) is compared with the proposed method. I suggest the authors provide more comparisons with existing distributionally robust regression methods (e.g., "Distributionally
robust logistic regression" and "Robust Wasserstein profile inference and
applications to machine learning." ) w.r.t. both computational time and classification error.
4. The authors take the medians over 20 random training set-test set splits as the final results in Tables 1--2. What if the results are taken as means (rather than medians)? Does the proposed method still outperform the competitors?
5. For some of the UCI datasets in Table 1 (e.g., lenses with $N=24$ and balloons with $N=16$), the sample size $N$ is too small to provide a convincing classification error on the test set.

* Wasserstein distance estimation usually suffers from the curse-of-dimensionality, see "Projection-based techniques for high-dimensional optimal transport problems". In particular, the convergence rate of the empirical Wasserstein distance may be arbitrarily slow as the size of dimensions increases. I wonder how does the number of variables affect the performance of the proposed method? Some discussion and empirical results are needed.

**Minor Comments**

1. The definition of $\mathbb{B}$ seems to be inconsistent in lines 16 and line 73.
2. In problem (4), the notations need to be defined and explained in detail.
3. In Section 3, please add some references for the column-and-constraint generation scheme.
4. In Section 4.1, the comparison between the left panel and the middle panel of Fig. 2 is unfair for their scales are not consistent. Please provide the log-scale runtime versus increasing $m$.
limitations:
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper addresses distributionally robust logistic regression with mixed features (i.e., continuous and categorical). Formally, the logistic regression criterion is minimized for any distribution in a Wasserstein neighborhood of the empirical distribution, see Eq. (1). The categorical nature of the features considered yields a more complex equivalent problem than in previous works [28], see Theorem 1. The authors then show the importance of considering categorical features as non continuous (Section 2.2), and that their problem cannot be rephrased as a regularized logistic regression (Theorem 3). The equivalent problem derived in Thm 1 is however NP-hard to solve for a generic loss function (Theorem 2), but the authors devise an algorithm to solve it with the log loss (Theorem 4). Experiments are also provided (Section 4).
strengths_and_weaknesses: Strengths:
- the paper is clear and well written
- the authors provide a sound and thorough study of the problem

Weaknesses:
- I am not extremely familiar with the literature, but this work might be considered to be closely related to the reference [28], that already studies distributionally robust logistic regression, but with continuous features only. Though, I feel that the categorical features yields problems of a significantly different nature. Theorem 3 is a perfect illustration of this fact.
- I am not 100% convinced by the proof reasoning which shows that categorical features cannot be treated as continuous ones, see my questions below.

Overall I like the paper and tend towards acceptance. My only concern regards its significance, with respect to [28] in particular, although I feel that the problems derived are sufficiently different in nature to justify a novel publication. I am open to discussion with the authors and other reviewers familiar with the literature on this point.
questions: Major:
- regarding the justification for not considering categorical values as continuous: why considering the absolute value and not the square difference (possibly scaled by 1/4 if necessary)? does it make any difference? how come that $z=15,312$ is considered, this distribution cannot be in the ball centered at $\hat{P}_N$, or am I missing something here?
- Instead of saying *generic loss*, I would specify that a linear loss is used in Theorem 2
- I find Theorem 3 very interesting, and would give a brief description of the proof, as it holds for **any** loss, what I believe is a strong result
- what happens if the discrepancy between measures to create $\mathcal{B}_\epsilon$ is not the Wasserstein?

Minor:
- l.91: $\sum_j z_j$ should be equal to 1 or less or equal than 1?
- l.92: given the proposed definition, $\mathbb{C}(2)$ should be {$(0,0), (0, 1), (1, 0)$}, no?
- display 97-98: have the dimensions aligned with the $\beta$, i.e., put $\beta_0$ in the end, or have $\mathbb{R}^{1 + n + k}$, could help
- the notation $d$ for the distance in the Wasserstein definition (2) might be confusing with $d\xi$
- I found formulation (6) from the appendix more readable and understandable than Eq. (4), although I understand the desire for a convex formulation. Maybe adding a remark to explain how to make the softplus constraints convex can be a solution
- Fig. 1: add "robust" in the legend for the methods concerned (left), $(z, y) = $ on the $x$ axis no?
limitations: This is a theoretical work with no foreseeable negative societal impact.
review_title: null

Meta-review: metareview: The focus of the submission is distributionally robust logistic regression when the discrepancy used in the ambiguity set is the Wasserstein distance and the features are mixed (i.e., they can contain both numerical and categorical variables). After showing that the resulting optimization problem (1) with the log-loss function can be reformulated as a finite-dimensional exponential conic program (Theorem 1), they (i) prove that (1) can be solved in polynomial time (Theorem 2), (ii) show that it does not admit a regularized logistic regression form (Theorem 3) as it is the case for purely numerical features, (iii) propose a column-and-constraint solver (Theorem 4-5). The practical efficiency of the proposed method is illustrated on 14 UCI benchmarks.

Logistic regression (LR) is among the most popular tools in machine learning and statistics. Handling mixed features for LR in the distributionally robust case is a relevant problem. The submission represents a solid work combining both important theoretical and empirical insights as it was evaluated by the reviewers.
confidence: Certain
Decision: Accept