Wasserstein Logistic Regression with Mixed Features
Authors: Aras Selvi
Mohammad Reza Belbasi
Martin B Haugh
Wolfram Wiesemann
Venue: NeurIPS 2022 Accept (2022)
Keywords: Distributionally Robust Optimization
Wasserstein Distance
Logistic Regression

Abstract: Recent work has leveraged the popular distributionally robust optimization paradigm to combat overfitting in classical logistic regression. While the resulting classification scheme displays a promising performance in numerical experiments, it is inherently limited to numerical features. In this paper, we show that distributionally robust logistic regression with mixed (\emph{i.e.}, numerical and categorical) features, despite amounting to an optimization problem of exponential size, admits a polynomial-time solution scheme. We subsequently develop a practically efficient cutting plane approach that solves the problem as a sequence of polynomial-time solvable exponential conic programs. Our method retains many of the desirable theoretical features of previous works, but---in contrast to the literature---it does not admit an equivalent representation as a regularized logistic regression, that is, it represents a genuinely novel variant of the logistic regression problem. We show that our method outperforms both the unregularized and the regularized logistic regression on categorical as well as mixed-feature benchmark instances.

Wasserstein Logistic Regression with Mixed Features
Aras Selvi Mohammad Reza Belbasi Martin B. Haugh Wolfram Wiesemann Imperial College Business School, Imperial College London, United Kingdom {a. selvv119 belbbasi,,, m. haugh ww):iimeeiial. acc.uk uk
Abstract
Recent work has leveraged the popular distributionally robust optimization paradigm to combat overfitting classical logistic regression. While the resulting classification scheme displays a promising performance in numerical experiments, it is inherently limited numerical features. In this paper, we show that distri- butionally robust logistic regression with mixed (i.e., numerical and categorical) features, despite amounting to an optimization problem of exponential size, ad- mits polynomial some-mmmll-mmeeeeeeee time solution scheme. We subsequently develop practically efficient column-ana wowtunn-nnddcooo auddccostrriin approach that solves the problem as a sequence of polynomial-time solvable exponential conic programs. Our model retains many of the desirable theoretical features of previous works, but-in contrast the regularized logistic
literature-De it does not admit an equivalent representation regression, that is, it represents a genuinely novel variant of logistic regression. We show that our method outperforms both the unregularized and the regularized logistic regression on categorical as well as mixed-feature benchmark instances.
1
Introduction
Consider a data set (ar,, yy y with feature vectors x and associated binary labels y (-1,1). Classical logistic regression assumes that the labels depend probabilistically on the features via
Prob(y x)) = [1 + exp(-y- 180 3. =:)]]]
where the parameters (9o, 33 E IR I + are estimated from the empirical risk minimization problem
minimize (Bo.g)
Σ (o(
subject to (Bo. 3) E 1+
with the log-loss function (a( y) == log (1 + exp (-y [80 + 33
+3xx)))) x1)) Its compelling performance across many domains, the availability of mature and computationally efficient algorithms well as its interpretability have all contributed to the widespread adoption logistic regression 6 18 27).
Similar to other machine learning models, logistic regression is prone to overfitting, especially when number of training samples is small relative to the number of considered features. Moreover, logistic regression can be sensitive erroneous feature and label values as well as distribution shifts under which the training and test sets stem from different distributions. In recent years, distributionally robust (DR) optimization [3] has been proposed to simultaneously address these challenges. To this end, the DR optimization paradigm models a machine learning task as a zero-sum game between the decision maker, who seeks to obtain the most accurate model (e.g.. the parameter vector ( in a logistic regression), and a fictitious adversary who observes the decision maker's 's model and subsequently selects the worst data-generating distribution in the vicinity of the empirical distribution formed from the available training data. In this context, the similarity of distributions is commonly
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

measured in terms of moment bounds 19 39 6 d-divergences such as the Kullback- divergence
T9 22 or the Wasserstein distance [5 25 has been observed that in many cases, the resulting
DR machine learning models are equivalent to regularized versions of the same models, in which case distributionally robustness provides a new perspective on regularizers that are often selected ad hoc [2] 41]. Most importantly, DR machine learning models often admit computationally efficient reformulations as finite-dimensional convex optimization problems that can be solved in polynomial time with off-the-shelf solvers.
In this work we study DR variant of the logistic regression where the distance between the empirical distribution and the unknown true gett-gennrating distribution is measured by the popular Wasserstein (also known as cuntorovicc-Rubbnstte has been first studied by (II,, who show that the resulting DR logistic regression problem admits an equivalent reformulation as a polynomial-size convex optimization problem if all features are numerical. Applying similar techniques to the mixed-feature logistic regression problem, which appears to be most prevalent in practice, would result in an exponential-size convex optimization problem whose naïve solution with the methods of [31] does not scale to interesting problem sizes (ef. Section 4 Our contributions may be summarized as follows.
or earth mover's) distance [15] This problem
(i) On the theoretical side, we show that the complexity of DR mixed-feature regression crucially relies on the selected loss function. In particular, for the log-loss function employed in logistic regression, the problem- despite its natural representation as an exponential size convex optimization problem problem-admiee admits polynomial-tie time solution scheme. We also show that in stark contrast to earlier variants of the problem, our mixed-feature regression does not admit equivalent representation evidence that the DR logistic regression problem with mixed features is fundamentally different to the DR problem with only numerical features.
regularized problem. This provides compelling
(ii) On the computational side, we propose aollumn-nnd-oostiin scheme that solves the DR mixed-feature logistic regression as sequence of polynomial solvable exponential conic programs. We show that the key step of our procedure, the identification of the most violated constraint, can be implemented efficiently for broad range of metrics, despite its natural representation
combinatorial optimization problem. Indeed, identifying the most violated constraint by brute force is out of the question, while the standard approach [K] of solving the most violated constraint problem would have an unacceptably high runtime.
(iii) On the numerical side, we show that our oolumn-and-consstrinn scheme drastically reduces computation times over a naïve monolithic implementation of the regression problem. We also show that our model performs favorably on standard categorical and mixed-feature benchmark instances when compared against classical and regularized logistic regression.
The literature on DR machine learning under the Wasserstein distance is vast and rapidly growing. Recent works have considered, among others, the use of Wasserstein DR models in multi-label learning [14), generative adversarial networks 2| and the generation of adversarial examples [40], density estimation [38 and learning Gaussian mixture models (20) graph-based semi-supervised learning [35] supervised dimensionality reduction [3] and reinforcement learning [I]. DR has also
been found help alleviate problems with overfitting [24 21), label uncertainty (21] and distribution
shifts (37 36.. We refer to (21) for a recent review of the literature. Our work is most closely related
to [31] and we compare our findings with the results of that work in Section 2
We proceed as follows. Section 2 defines and analyzes the mixed-feature DR logistic regression problem, which is solved in Section 3 via cellun-andd-oonee
generation. We report numerical
results in Section 4 Auxiliary material and all proofs are relegated to the appendix.
Notation. We define B {0. 1} and [N] = {1,...,,M NY for N E N. The set of all probability distributions supported on a set E is denoted by PE) while the Dirac distribution placing unit probability mass on æ E RR is denoted by 8n E (TR"). The indicator function [[ attains the value 1 (0) whenever the expression and to distinguish them from the d used to denote distances.
(not) satisfied. Finally, we use Roman d to denote differentials

---

2 Mixed-Feature DR Logistic Regression
Section 2.1 derives an exponential-size convex programming formulation of the DR logistic regression problem that serves as the basis of our analysis. Section 2.2 compares our formulation with a naive model that treats categorical features as continuous ones. Section 2.3 finally, shows that our formulation, despite its exponential size, can be solved in polynomial time due to the benign structure of the log-loss function. also establishes that in stark contrast to the literature, our formulation does not reduce to regularized non-robust logistic regression, that is, it constitutes a genuinely novel variant of the logistic regression problem.
Our formulation enjoys strong finite-sample and asymptotic performance guarantees from the litera- ture. Moreover, despite its exponential size, our model always accommodates worst-case distributions that exhibit desirable sparsity pattern. We relegate these results to Appendix A.
2.1 Formulation as an Exponential-Size Convex Program
From on, consider mixed-feature data set {E" (z', 2² v Hie[N] with numerical features xx (xf (.i.-.... xt, E R2, m categorical features 2 (z)...., 2m C(k) x C(km) and a binary label y' E {1], Here, C(s) = {z E IB*-1 2jc(--1] 2j S 1) for E NN [1] represents the one-hot encoding of a categorical feature with possible values; in particular, C(2) = B encodes a binary feature. We denote by C = C(k)) x... x C(km) and E = R2 x C (-1, +1} the support of the categorical features well as the data respectively, and we k ky t be the number of slopes used for the categorical features.
If we had access to the true data- teataagenraainggggg distribution pa E Po(E), we would solve the (non-robust) logistic regression problem
minimize
Epo [la(2, 2,
subject to B = (Bo, GN. , Bc) E 1g+++*
where Epo denotes the expectation under ppo, and where the log-loss function (/3 now takes the form
(a(x, (x z, y) = log (1 + exp - (Bo + 3N T x + Be 2)))
to account for the presence of categorical features. Since the distribution pa is unknown in practice, the empirical risk minimization problem replaces Ipd with the empirical distribution PN == N Σi 6g- that places equal probability mass on all observations { ]xe[N]- Standard arguments show that when these observations are i.i.d., the empirical risk minimization problem recovers the logistic regression under pp as N - co. In practice, however, data tends to be scarce, and the empirical risk minimization problem exhibits an 'optimism bias' that is also known as overfitting [6] 18 27
the error maximization effect of optimization [10 23 or the optimizer's 's curse [34).
DR logistic regression combats the aforementioned overfitting phenomenon by solving the semi- infinite optimization problem
minimize
sup (2N)
EQ (1,(2,22,1)
subject to B = ( Gx Bc) E R 1+
where the ambiguity sei B. (Px)) contains all distributions Q in (soon be defined) vicinity of the empirical distribution FN. and where the expectation is taken with respect to Q. Problem | can be interpreted as a zero-sum game between the decision maker, who chooses a logistic regression model parameterized by (3, and fictitious adversary that observes (3 and subsequently chooses the "worst" distribution (in terms of the incurred log-loss) from B. (P ). Contrary to the classical non-robust logistic regression, problem ① guarantees overestimate the log-loss incurred by B under the unknown true distribution pd as long as P0 contained in B. (Px). On the other hand, problem (I) recovers the empirical risk minimization problem when the ambiguity set B. (Pxx) approaches a singleton set that only contains the empirical distribution Py. Note that problem ① is convex as the convexity of the logistic regression objective Teg ( 2, 2,y]] is preserved under the supremum operator. That said, the problem typically constitutes semi-infinic program as it comprises finitely
3

---

many decision variables but-if the embedded supremum in the objective is brought to the constraints via an epigraph reformulation-inniinni -infinitely many constraints whenever the ambiguity set harbors infinitely many distributions. As such, it is not obvious how the problem can be solved efficiently.
In this paper, we choose as ambiguity set the Wasserstein ball B, (Px) = {Q E Po(E)
W(Q, <) of radius >> 0 that is centered at the empirical distribution PNN. Definition 1 (Wasserstein Distance). The type-1 Wasserstein Cantorovich-Rubiisee mover's) distance between two distributions E Po(E) and Q E Po(E)) is defined as
or earth
U d(E, II(E, dE') II 3) Q(d)). II dE') PP(55)
W)Q,P) = inf
(2)
IIEP)(E?)
where { (x. y. E E and g' (x', z', v// E, while d(g. E') the ground metric on
The Wasserstein distance can be interpreted as the minimum cost of moving Q to P when d(ć. g') (') is the cost of moving a unit mass from & g'. The Wasserstein radius thus imposes budget on the transportation cost that the adversary can spend on perturbing the empirical distribution PN.
We next define the ground metric d that we use throughout the paper.
Definition (Ground Metric), We measure the distance between two data-points E = (æ, 2, v) E and 5 = (x:' z', E with z (Z1,..-a Zm)) and z' (2(,--.. z'm)
d(E,E') = 1|= æ'll delzz z') + a 12y # v/}. where ||-1] is any rational norm on R", >> O and the metric de C satisfies
(3a)
1[ + =11) 1/v
dc(z,z')
for some O.
(3b)
kE(m)
Intuitively, the ground metric d measures the distance of the numerical features æ and w' by the norm distance æ æ'Il, whereas the distance of the categorical features and z' is measured by (a power of) the number of discrepancies between z and z'. Likewise, any discrepancy between the labels and y' is accounted for by constant which allows for scaling between the features and the labels.
We first extend the convex optimization model of (31] for the DR continuous-feature logistic regres- sion to exponential conic model that accommodates for categorical features.
Theorem (Exponential Conic Representation). The DR logistic regression problem 1 admits the equivalent reformalation
minimize B.A.s.u.. at
de +
Σ 81
N
NEiN]
subject to
"x,x + + <
(u 1, 8; Ade(z, (ee(zzz)) x²)) Kexp
Vi [V]. C
(o 1, 1y' BN x² y Be z Bu - Adc(2,21)) Kexp
T
uT - $
(Mi,,, 1, -8; Ade(z, z²) ) >ni) E KexD
Vi [N], Vz Ve
(Vi 1, Ax xx' + + Bc + 1² 30
)lc(2,z)) dx)) E Koxp
||xx]]+ _
B (Bo, BN, Gc) E Rl+ X20, 0, SERN (u "i U1,, ) E R² iE [N] and E
(4)
as a dimeesdimensional exponential conic program, where Keep denotes the exponential cone
Kexp == cl (((a,b,c) a 2 exp(c/b), O, b > 03) R
The s;'s 's and À that appear in (4) are dual variables that arise from dual of the inner (i.e,, sup) problem in d1). For each and é [N]. (u o "i "T,=) a vector auxiliary variables that we use to model softplus constraints that arise from our intermediate analysis of problem (I) (Appendix E).

---

1.25
Empirical Features
estimated cacificient
Continuous- Sortiruoux-Feotures Features
1.00
0.75
0.50
Logasic Regresson
0.25
100
200
300
400
500
(16312-11
size of the dataset
support
Figure 1: Left: Estimates of for the standard logistic regression, our mixed-feature well the cattimuous-fetture model as a function of the size N of the data set. All results are reported as averages over 2,000 statistically independent runs. Right: Comparison of the empirical distribution as well as the worst-case distributions for 8 = 1 under the two DR models for N = 250 samples. The probabilities are plotted on log-scale make small values visible.
Problem 4 is finite-dimensional and convex. Moreover, since exponential conic programs admit self-concordant barrier function, they can be solved in polynomial time relative to their input size (28 Due the presence of categorical features in regression problem, however, problem (4) comprises exponentially many variables and constraints, and a naive solution of 4 would therefore require exponential amount of time. The exponential size of problem (4) renders our formulation fundamentally different from the model of [31], and it significantly complicates the solution.
2.2 Comparison with Contimuous-Feature DR Logistic Regression
Our formulation 44 of the mixed-feature DR logistic regression (1) accounts for categorical features E the expense of an exponential number of variables and constraints. It therefore tempting to treat the categorical features as continuous ones and directly apply the continoous-featur-oon reformulation of the DR logistic regression (I) proposed by [31]. In the following, we argue that such a reformulation would hedge against nonsensical worst-case distributions that in turn lead to overly conservative regression models. To see this, consider a stylized setting where the data set (z*, y² (e(N) comprises single binary feature z' E { 1 +1} that impacts the label y f-1,+1} via the logistic model
Prob(y =) = [1 + exp(-1) 3²)]
with 3 = (While we use 2' E (0, 1} the other sections of the paper, the analysis and results of this subsection and Appendix B assumed 2 E {-1,+1) as this made comparisons with [3]] casier.) We attempt to recover this logistic model in two ways:
1. fexturfeature model. We employ our DR logistic regression II with single categorical feature, using p = and x = in the ground metric (cf. Definition 2}.
2. Continuous-featue model. We employ variant of problem I that treats the categorical
feature 2; continuous one. We choose as ground metric
1 -11== 2'1 I[y + v/l.
C(E,E')
1 We therefore use the feature distances [ * 2'] in the mixed-featur case and [ - z'| in the continuous-featume case. Note they both attain the value 0 (1) for equal (different) feature values.
Figure (left) compares the mean values (dashed lines) as well as the 15% and 85% quantiles (shaded regions) of / for the standard logistic regression, our mixed-feature as well as the continuous-feature model as a function of the size N of the data set. For both DR models, we employed the same
Wasserstein radius xx 1 1/vN, which is motivated by the finite sample guarantee presented in the model excessively shrinks
next section (cf. Theorem 6). The figure shows that the continuous-feant its estimates of 3, thus leading overly conservative results. In fact, the true parameter value 3 = 1
5

---

consistently lies outside the confidence region of the continuous-featur model, independent of the sample size N. This is due to the fact that the cantinuous-feature model accounts for nonsensical worst-case distributions under which the categorical feature can take values outside its domain {-1,+1), see Figure (right). In fact, the continuous-feature model places non-zero probability on an extreme scenario under which the binary feature 2 E {-1, +1} attains the value 15, 312. (In Appendix B we explain why this value of 15, 312 arises.) Our hand, restricts the worst-case distribution to the domain of the categorical feature and thus hedges against realistic distributions only.
model, on the other
We note that in practice, the radius e of the Wasserstein ball will be chosen via cross-validation (ef. Section 4. in which case our mixed-feature model reliably outperforms the classical logistic regression on standard benchmark instances. While one may argue that the impact of nonsensical worst-case distributions in the continuous-feature model alleviated by the vross-validated radii, Figure [ offers strong theoretical and practical reasons against the use of a continuous-featus for categorical or maxxed-featur problems: The aartinuous-feature model hedges against a clumsily perturbed worst-case distribution that introduces pronounced bias in the estimation. We also note that another possible approach would be to use the continuous-feature model but to restrict the support of binary features to [-1, I]. Unfortunately, for the log-loss function no tractable reformulation with support constraints appears to be known; see [32] Moreover, Section 4 will show that the mixed-feature model, despite its exponential size, can be solved quickly and reliably with our novel column-ame and-constrain approach from Section
model
2.3 Complexity Analysis
We first show that despite its exponential size, the DR logistic regression problem (T] admits polynomial-time solution. Key to this perhaps surprising finding is the shape of the loss function: while problem (1 is strongly NP-hard (and thus unlikely admit a polynomial-time solution scheme) for generic loss functions, it can be solved in polynomial time for the log-loss function Lg employed in logistic regression.
Theorem 2 (Complexity of the DR Logistic Regression
For generic loss functions 1g, problem 1 is strongly NP-hard even n Ond N
(ii) For the loss function ( (x, z, y) = log (1 + exp [-y (go + BN Tx 3c 2))) and the ground metric of Definition 2 problem I can be solved to d-accuracy in polynomial time.
Recall that an optimization problem is solved to 5-accuracy if 6-suboptimal solution is identified that satisfies all constraints modulo a violation of at most 6. The consideration of 8-accurate solutions is standard in the numerical solution of nonlinear programs where an optimal solution may be irrational.
by now well-known result shows that when m 0 (no categorical features), the DR logistic regression problem ① reduces to classical logistic regression with additional regularization term | ||+ in the objective function when the output label weight x in Definition 31 32). We next show that this reduction to classical regularized logistic regression longer holds our problem setting when categorical features are present.
2 approaches oo
Theorem 3 (Absence of Reformulation
Regularized Problem). Even when the output label weight k approaches oo in the DR logistic regression reformulation
II] problem IT)
does not admit an equivalent
minimize subject to
EE L 2, w)] + R(3)
B (Bo, Bx. Be) E RR 1+
as a classical regularized logistic regression for any regularizer R. II+++ IR.
Note that Theorem 3 does not only preclude the existence of specific regularizer, but it excludes the existence of any regularizer, no matter how complex its dependence on 3 might be. We are not aware of any prior results of this form in the literature. Some insight for the result Theorem may be found via the special case consider in its proof in Appendix E and in Remark that follows it.

---

Algorithm Column-and-Coostrriin Generation Scheme for Problem
Set LBg = -0 and UBp = +o0
Choose (possibly empty) subsets W+,W C [ while LB, f UB, do
Let 0 be the optimal value and (B, A, 8, u= x=) be minimizer of problem 4 where:
the first constraint set is only enforced for all (i, 2) E W
the second constraint set is only enforced for all (i, x) E W
only include those (2 o(22)) for which (i, 2) W
only include those (Mr v(,z) for which (8, z) WT
Identify most violated index (it ( of the first constraint set in (4) and add it to W Identify a most violated index (i 2 of the second constraint set in (4) and add to W
Let 1 and t be the violations of (i x) and (i
respectively
Update LB, = 8 and UB, = min(UB, 4" + log(1 max( (- }}}
Update / = -++1
end while
Algorithm 2 Identification of Most Violated Constraints in the Reduced Problem (4)
for j E {I....,,y) } do
Find a feature value 2 that minimizes y' Bc. 2j across all 2j E C(k,) \ {zj) end for Let π [m] - [m]] be an ordering such that
,',,)x(()) (2 21))) S v' Be (2%(0) #(0)))
AA S S j sm
Set W
for ā E (0, 1...., . do
Update w = Ww {z), where 2, = if m() S and z, 2} otherwise
end for Determine the most violated constraint from the candidate set W
3 Column-and-Constrain Solution Scheme
Our DR logistic regression problem 44) comprises exponentially many variables and constraints, which renders solution as monolithic exponential conic program challenging. Instead, Algo- rithm emmplos volumm-and-consse solution of relaxations problem (4) that omit most of its variables and constraints and (ii) adding those variables and constraints that promise to maximally tighten the relaxations.
generation scheme, e.g., [43), that alternates between (i) the
Theorem 4. Algorithm 7 solves problem 44) in finitely many iterations. Moreover LB, and UB, constitute monotone sequences of lower and upper bounds on the optimal value of problem {
key step in Algorithm is the identification most violated indices (i, 2) E [N] x the first and second constraint set in the reduced DR regression problem (4) for a fixed solution (B, A, 8, -= v²). For the first constraint set in ④ the identification of such indices requires for each data point E [N] the solution of the combinatorial problem
ut + at
(ut 1, -s Ade(z, ()) E Koxp,
maximize
min at
(vt -1 Br xx' - B'Bc z y' - 8; -de((2 E Kexsp
subject
E
(5)
where an optimal value greater than 1 corresponds to a violated constraint; an analogous problem can be defined for the second constraint set in (4). Despite its combinatorial nature, the above problem can be solved efficiently by means of Algorithm
Theorem 5. Algorithm 2 identifies a most violated constraint for given data-point E [N] in time Ol + n + m²)
7

---

Latlay
Urm Hog seccendt
Unoe 100 secomnda
time leep secordda)
2.3
Lutking Panks
20
40
60
80
100
2.5
10
33
20
60 B0
100
number of instances (M)
number of oibinary features (16
number of Instances (N)
Figure 2: Runtime comparison between our column-and-constrailn scheme and a naive solution of problem 4 monolithic exponential conic program. Left: Runtimes for 10 binary features as a function of the number N of data points. Middle: Runtimes for = 50 as a function of m. Right: Runtimes
aoluumn-nnd-oostraale scheme only for varying combinations of and N. In all graphs, shaded regions correspond to 10%-90% confidence regions and bold lines report median values over 50 statistically independent runs. Note the log-scale the plots.
Algorithm first determines for each categorical feature zj. J 1,..., m,, the feature value 27 E C(ky) \ [2] that, ceteris paribus, contributes to a maximal constraint violation in time (kk). The subsequent step sorts the different features j ...... in descending order of their contribution to constraint violations in time (m l)). The last step of Algorithm 2 finally, constructs the most violated constraint z across those which 8 ....... m categorical feature values deviate from z and subsequently picks the most violated constraint of these in time 0nn=m²). We note that the overall complexity of O(k + + m²) can be reduced O(k n m ml m) by a clever use of data structures in the final step; for ease of exposition, we omit the details. Finally, since Algorithm@ji applied each data point [N]. the overall complexity increases by factor of N. We provide additional intuition behind Algorithm immeddately hefore the proof of Theorem 5 in Appendix E.
4 Numerical Results
Section 4.1 first compares the runtimes of our cowtuun--nd--oottrrann scheme from Section 3 with those of solving the DR logistic regression problem (4) naively monolithic exponential conic program. We subsequently compare the classification performance problem (L) with those of classical unregularized and regularized logistic regression on standard benchmark instances with categorical features (Section 4.2) and mixed features (Appendix C).
All algorithms were implemented in Julia [S] (MIT license) and executed on Intel Xeon 2.66GHz processors with SGB memory in single-core mode. We MOSEK 9.3 [26] (commercial) solve all exponential conic programs through JuMP 12 (MPL2 License). (We note the open source solvers ppopt and CVXOpt could be used instead of MOSEK.) All source codes and detailed results
available on GitHub battps://gitee http://ttttau...ooeeeeeeeeee wwwo//ellii... http://eevaaaassaeeeeeeeee wwwaaaassssasssssennniiRll
4.1 Runtime Comparison with Monolithic Formulation
We first compare the computation times of our those of solving the DR logistic regression problem (4) naïvely as monolithic exponential conic program. To this end, we randomly generate synthetic logistic regression instances with varying numbers of data points and m of binary features. While F ( (left) shows that both approaches scale similarly in the number N of data points, Figure 2 (middle) reveals that the solution of the monolithic formulation scales exponentially in the number m of binary features.
scheme from Section 3 with
contrast, our column-and auuum---nd-oostraiinn scheme scales gracefully in both the number N of data points and the numher m of binary features (cf. Figure 2 right). A similar behavior can be observed with more general, non-binary categorical features; omit the results due to space constraints.
4.2 Performance on Cntroorical-eatuure Instances
We next compare the classification performance (in terms of the out-of-sample classification error) of our unregularized ('DRO') and Lasso-regularized ("r-DRO') DR logistic regression problem (1] with those of classical unregularized ('LR') as well Lasso-regularizedd ('r-LRR'), mass transportation- regularized (MT) 31 32), and robust Wasserstein profile inferenne-requllarized (PI) [7] logistic

---

Data Set
N
k
m
LR
DRO (k 1)
DRO (s m))
r-LR
T-DRO (K 1)
r-DRO (K m)
MT 1)
MT m)
Pl (a: 0.05)
breast-cancer
277
42
29.50%
29.15%
2
29.05%
29.16%
28.82%
29.44%
29.40%
29.33%
spect
267
22
22
18.81%
17.72%
17.51%
17.79%
18.38%
15.83%tt
18.49%
18.74%
20.60%
monks-3
554
11
2.07%
2.08%
2.10%
2.14%
2.19%
2.13%
2.15%
2.15%
37.87%
tic-tac-toe
958
18
1.92%
1.69%
1.67%f
1.68%
1.68%
1.6%%
1.72%
1.81%
30,02%
kr-vs-kp
3,196
37
36
2.64%
2.66%
2.64%
2.69%
2.60%
2.00%
2.77%
2.74%
12.85%
balance-scale-
625
16
0.85%
0.82%
0.72%
0.73%
0.71%
0.64%
0.72%
0.69%
30.10%
hayes-rothe
160
11
17.00%
16.31
16.09%
18.66%
18.19%
17.78%
17.50%
16.38%
37.59%
hymphography
148
42
18
21.17%
17.83%
16.72%
17.38%
17.52%
17.03%
19.41%
17.83%
20.00%
care
1.728
15
4.73%
4.73%
4.73%
4.99%
4,76%
4,87%
4.73%
4.77%
15.37%
splices
3,189
229
6.80%
5.92%+*
6.66%
6,22%
5.99%
6.02%
6.57%
6.87%
11.08%
house-votes-84
435
32
6.60%
4.43%
5.37%
4.54%
4.41%
5.16%
4.78%
5.54%
4.26%
6.590
152
5.94%
5.93%
5.92%
5.90%
5.90%
5,90%
6.04%
6.03%
20.45%
nimacy-tumors
339
25
17
13.93%
13.66%
13.76%
14.51%
14.06%
14.19%
13.87%
13.85%
19.18%
audiology+
226
92
69
14.18%
14.03%
14.40%
3.11%
2.80%
3.91%
12.64%
7.49%
15.49%
Table 1: Classification errors of unregularized as well as Lasso-, mass transportation-, and profile inferenceereegulaiizee variants of the classical logistic regression and our DR regression on UCI benchmark instances with categorical features only. The smallest error within each model group (unregularized vs. regularized) is highlighted in italics, whereas the smallest error overall (across all groups) is printed bold. The dagger (t) and double dagger ($) symbols next the best model denote statistically significant improvements over LR and the second best model, respectively.
regression on the 14 most popular UCI data that only contain categorical features having more than 30 rows [1T] (varying licenses). Instances with multiple output labels are indicated with star; we convert them into instances with binary output labels by distinguishing between the majority class vs. all other classes. The instances vary in the number N of data points, the number m of categorical features as well as, accordingly, the number of slopes considered in the one-hot encoding of the categorical features (cf. Section 2. All results are reported as means over 100 random training set-test set splits (80%:20%) The radius (0, 10
1} of the Wasserstein ball 10 ..... 2 are selected via 5-fold cross-
10
1)
as well as the Lasso penalty validation. We consider two variants our DR logistic regression that employ different output label m) in the ground metric (cf. Definition The results reported in Table []
{0, 10
weight (x = 1 vs. (Run-times and are provided in Appendix D where we also describe how statistical significance was assessed.) The table shows that for the unregularized model, the classical logistic regression achieves the lowest classification error in 21% of the instances, whereas our DR logistic regression achieve the lowest classification eπror in 36% and 79% of the instances for double-counting of ties means these percentages do not sum to 100%.) For the regularized model, the results change to 14% (classical logistic regression) vs. 57% (each of our models). Within the numerical-featur Wasserstein DRO benchmarks, PI achieves the lowest classification error 14% of the instances, whereas MT achieves in 42% and 57% of the instances for 1, m, respectively. Globally, one of our methods is the winning approach in 12 out of 14 datasets (in the remaining they are the second best approach without statistically significant inferiority). In 9 of the datasets our methods win strictly (without a tie), and in 7 of these datasets the improvements are statistically significant over all other approaches.
1, m. respectively. (The
Conclusions
We proposed a new DR mixed-feature logistic regression model where the proximity between empirical distribution and the unknown true data-generating distribution is measured by the popular Wasserstein distance. Despite its exponential-size formulation, prove that the underlying optimization problem can be solved in polynomial time, and we develop a practically efficient column-anddconsttriin generation scheme for its solution. The promising performance of our model demonstrated in numerical experiments on standard benchmark instances. We note that our column- and-constraint scheme readily extends other DR mixed-feature machine learning models such as linear regression, support vector machines and decision trees. Applying our algorithm in those settings is promising direction for future research. A further interesting direction is the development of tailored solution schemes rather than using off-the-shelf solvers for solving our DR mixed-feature logistic regression problem.

---

Acknowledgments and Disclosure of Funding
Funding from the EPSRC grant EP/W003317/1 is gratefully acknowledged. The first author ac- knowledges support from The Alan Turing Institute. The authors are grateful for the comments and suggestions of the anonymous reviewers, which have helped significantly improve the manuscript. This work has been produced without the involvement of any commercial entities that may cause conflict of interest or competing interests.
References
[1] M. A. Abdullah, H. Ren, H. Bou-Ammar, V. Milenkovic, R. Luo, M. Zhang, and Wang. Wasserstein robust reinforcement learning. http://arxiv. org/ herg/abs/1900 1907. 13196
[2] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, 2017.
[3] A. Ben-Tale L. El Ghaoui, and A. Nemirovski. Robust Optimization. Princeton University Press, 2009.
[4] D. Bertsimas and D. den Hertog. Robust and Adaptive Optimization. Dynamic Ideas, 2022.
[5] Bezanson, A. Edelman, S. Karpinski, and V. B. Shah. Julia: A fresh approach numerical
computing. /attp:://doi.orgg 11370111101/111177
[6] C. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
[7] Jose Blanchet, Yang Kang, and Karthyek Murthy, Robust Wasserstein profile inference and applications to machine learning. Journal of Applied Probability, 56(3):830-857, 857, 2019. doi: 10.1017/jp:.219.499
[8] S. Chakrabarti, H. Yiming, T. Li, S. Feizi, and X. Wu. Quantum Wasserstein generative adversarial networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
[9] E. Delage and Y. Ye. Distributionally robust optimization under moment uncertainty with application data-driven problems. Operations Research, 5863):59-661,, 2010.
[10] V. DeMiguel and F. Nogales. Portfolio selection with robust estimation. Operations Research, 57(3):560 577, 2009.
[11] D. Dua and C. Graff. UCI machine learning repository. http://archive. iics www.uci.... edd/mll
[12] I. Dunning, J. Huchette, and M. Lubin. JuMP: A modeling language for mathematical optimiza- tion. SIAM Review, 59(2):295-33) 320, 2017.
[13] R. Flamary, M. Cuturi, N. Courty, and A. Rakotomamonjy. Wasserstein discriminant analysis. Machine Learning, 107(12):1923-144 2018.
[14] C. Frogner, Zhang. Mobahi, M. Araya, and A. Poggio. Learning with a Wasserstein loss. In Advances in Neural Information Processing Systems, volume 28, 2015.
[15) R. Gao and A. Kleywegt. Distributionally robust stochastic optimization with Wasserstein distance.
02199
[16] M. R. Garey and D. S. Johnson. Computers and Intractability: A Guide to the Theory of NP-C Completeness. W. H. Freeman, 1979.
[17] M. Grötschel, L. Lovász, and A. Schrijver. Geometric Algorithms and Combinatorial Optimiza- tion. Springer, 1988.
[18] T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical learning: Data mining, Inference, and Prediction. Springer, 2009.
[19] Hu and I.. J. Hong. Kullback- divergence constrained distriuttioin robust optimiza- tion. http://wwwwwpopaaaaaacaooo/ooooa
10

---

[20] S. Kolouri, G. K. Rohde, and H. Hoffmann. Sliced Wasserstein distance for learning Gaussian mixture models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[21] D. Kuhn, P. Mohajerin Esfahani, V.A. Nguyen, and S. Shafiezzadeh-Abudeh. Wasserstein distributionally robust optimization: Theory and applications in machine learning. INFORMS TutORials in Operations Research, pages 130-169, 2019.
[22] H. Lam. Recovering best statistical guarantees via the empirical divergence- distribution- ally robust optimization. Operations Research, ((4):1090-1100 110,, 2019.
[23] R. O. Michaud. The Markowitz optimization enigma: Is 'pptimized'optima??? Financial Analysts Journal, 45(1):31-42, 1989.
[24] P. Mohajerin Esfahani and D. Kuhn. Data-driven distributionally robust optimization using the wasserstein metric: performance guarantees and tractable reformulations. Math. Program., 171: 115-166, 166, 2018.
[25] P. Mohajerin Esfahani and D. Kuhn. Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations. Matheraatiaal Programming, (1112))1-55, 2018.
[26] MOSEK ApS. The MOSEK optimization toolbox for MATLAB manual. Version 9.0. http:
/ /docs. /docs.mosek. w.mm...0. tww.o/o/ooobx//deex.....oooo html 2019.
[27] K. P. Murphy. Probabilistic Machine Learning: An Introduction. MIT Press, 2022.
[28] Y. Nesterov. Lectures Convex Optimization. Springer, 2nd edition, 2018.
[29] Steven L Salzherg. On comparing classifiers: Pitfalls to avoid and recommended approach. Data mining and knowledge discovery, 133):317-328, 1997.
[30] S. A. Serrano. Algorithms for unsymmetric cone optimization and an implementation for problems with the exponential cone. Stanford University, 2015.
[31] S. Shaficezadeh-Abadee Mohajerin Esfahani, and D. Kuhn. Distributionally robust logistic regression. In Advances in Neural Information Processing Systems, volume 28, 2015.
[32] S. Shafieezadeh--badeeh. D. Kuhn, and P. Mohajerin Esfahani. Regularization via mass trans- portation. Journal of Machine Learning Research, 20 201103):1-68, 2019.
[33] A. Shapiro. On duality theory of conic linear problems. In Semi-Infinite Programming, pages 135-165. Kluwer Academic Publishers, 2001.
[34] J. E. Smith and R. L. Winkler. The optimizer's ' curse: Skepticism and postdecision surprise in decision analysis. Management Science, 52(3):311332,, 2006.
[35] Solomon, R. Rustamov, L.. Guibas, and A. Butscher. Wasserstein propagation for semi- supervised learning. In Proceedings of the 31st International Conference on Machine Learning. 2014.
[36] Tobias Sutter, Andreas Krause, and Daniel Kuhn. Robust generalization despite distribution shift via minimum discriminating information. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 19754-29761 Curran Associates, Inc., 2021.
[37] Bahar Taskesen, Man-Chung Yue, Jose Blanchet, Daniel Kuhn, and Viet Anh Nguyen. Sequen- tial domain adaptation by synthesizing distributionally robust experts. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10162-10172. PMLR, 18-24 Jul 2021.
[38] Weed and Q. Berthet. Estimation of smooth densities in Wasserstein distance, In Proceedings of the 32nd Conference on Learning Theory, 2019.

---

[39] W. Wiesemann, D. Kuhn, and M. Sim. Distributionally robust convex optimization. Operations Research, 62(6):1358 1376, 2014.
[40] E. Wong, F. Schmidt, and Kolter. Wasserstein adversarial examples via projected Sinkhorn iterations. In Proceedings of the 36th International Conference on Machine Learning, 2019.
[41] H. Xu, C. Caramanis, and S. Mannor. Robustness and regularization of support vector machines. Journal of Machine Learning Research,
488--1100, 2009.
[42] M. Yue, D. Kuhn, and W. Wiesemann. On linear optimization over Wasserstein halls. Mathe- matical Programmingg 2021.
[43] Bo Zeng and Long Zhao. Solving two-stage robust optimization problems using a column-and- constraint generation method. Operations Research Letters, 41(5):457-46,, 2013.
[44] Jingyi Zhang, Ping Ma, Wenxuan Zhong, and Cheng Meng. Projection-based techniques for high-dimensional optimal transport problems. WIREs Computational Statistics, 15888/
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The abstract explains (i) the motivation for using Wasserstein DR optimization in machine learning, (ii) recent successes of this approach in logistic regression with numerical features and (iii) the lack of work on the mixed- feature setting; these explanations are further elaborated upon in Section [ This section also provides high-level summary of our contributions, which are line with the developments in the remainder of the paper.
(b) Did you describe the limitations of your work? [Yes] Our numerical results have been conducted in an objective and reproducible way (choosing the most popular UCI data sets), and our experiments include data sets where the standard logistic regression outperforms our approach.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Our work extends well-known machine learning method (DR logistic regression) to new but well-known setting (presence of categorical features). In light of the ethics review guidelines, we therefore believe that our work does not have any direct implications society beyond the ones that are already known.
(d) Have you read the ethics review guidelines and ensured that your paper conformss to them? [Yes] We have carefully reviewed the ethics guideline on the NeurIPS 2022 conference webpage. We believe that our paper conforms to these guidelines.
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes] All of our lemmas and theorems state the full set of assumptions, with exception of the blanket assumptions that are assumed to hold throughout the paper and that are clearly marked as such.
(b) Did you include complete proofs of all theoretical results? [Yes] All proofs are included in the appendix of the paper.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] All code, data and instructions for our experimental results are published on GitHub. To maintain anonymity during the review process, we did not provide link until the camera-ready version.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] The most important choices are described in the numerical section, while some of the details are relegated to the appendix.
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [Yes] The runtime results provide confidence regions across multiple random runs, and we report error bars for all other experiments in the appendix. (d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] Information about the employed CPU is provided in the numerical results section; details about the nigh-performance cluster that we used are relegated to the appendix. Note that all experiments are run in single-core mode.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We are citing UCI for the data sets, MOSEK for the solution of the optimization problems as well as Julia (and JuMP) for the implementation of the algorithms.
(b) Did you mention the license of the assets? [Yes] We mention the licenses of UCL, MOSEK and Julia in the numerical results section.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] All code, data and instructions for our experimental results are published on GitHub. To maintain anonymity during the review process, we did not provide link until the camera-ready version.
13

---

(d) Did you discuss whether and how consent was obtained from people whose data you't rre using/curating? [N/A] We are using the most popular UCI machine learning data sets, for which to our best knowledge consent has already been given.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] We are using the most popular UCI machine learning data sets, for which to our best knowledge this issue has already been resolved.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] Our research does not have any crowdsourcing or human subject components.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] Our research does not have any crowdsourcing or human subject components.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] Our research does not have any crowdsoure- ing human subject components.
14


Reviews:
[{"review_id": "yty4aBSyHb", "reviewer": "Reviewer_7VSn", "summary": "This paper considers the mixed feature distributionally robust logistic regression problem, which extends the setting in [1] to the mixed feature (i.e., the feature space can be divided into the numerical (real number) and categorical (integer) parts.  The main contribution of this paper lies in deriving an equivalent dual reformulation of this new model and further developing tractable methods to address it. More specifically, by fully exploiting the problem-specific structure arising from the log-loss, the authors fully utilize its exponential conic representation and provide a Column-and-Constraint Solution Scheme to solve it. Finite steps convergence guarantee has also been established. Extensive experiments have been shown to demonstrate the effectiveness of its model performance.  \n\n[1]: Shafieezadeh-Abadeh, Soroosh et al. “Distributionally Robust Logistic Regression.” NIPS (2015).\n", "questions": "No ", "limitations": "The computational paradigm and theoretical results (i.e., polynomial solvable）is restricted to the logistic regression case. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**: \n\nThe mixed features considered in the paper are less explored in distributionally robust optimization. This paper takes a first step towards filling this gap. The paper is easy to follow and well-organized. Comprehensive experiments have been given to illustrate the extension.  Notably, Theorem 3 seems very interesting to me. They point out the absence of reformulation as a regularized problem, while this property holds for many DRO problems.\n\n**Weakness**:\n\nThere are three major concerns: \n\n1. The novelty of both modeling and techniques is moderate. The proposed model is a natural extension based on [1]. The dual reformulation is also relatively standard if we adopted the method provided in [1].  \n\n2. All methodologies developed in this paper highly depend on the log-loss and specific cost function (see theorem 2 for details), which are rather restrictive. In my opinion, the impact of this paper on broad ML tasks is limited. \n\n3. On the computational side, although the paper gives a polynomial-time solvable method to address the resulting dual reformulation, it still relies on certain off-the-shelf solvers, which will potentially suffer from the scalability issue on medium or large-scale datasets. More importantly, for the experiment results, the proposed mixed feature model does not gain substantial improvements over other baselines (i.e., vanilla logistic regression or distributionally robust LR) in terms of test accuracy (i..e, see tables 1 and 2),  but instead cost too much computational effort. \n\nFor the above reasons, this paper is strictly below the neurips acceptance level. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "wBh4ptSzkjx", "reviewer": "Reviewer_53YQ", "summary": "To overcome the non-robustness in classical logistic regression, the authors consider the distributionally robust logistic regression by introducing an ambiguity set defined by the popular Wasserstein distance.\nIn contrast to the previous literature, the proposed method focus on mixed features. Despite the optimization problem itself being exponential-size for generic losses, the authors show that it can be solved in polynomial time using the column-and-constraint generation scheme.\nTheoretically, they provide finite-sample and asymptotic performance guarantees for the proposed method. \nExperiments on synthetic and real-world datasets show the accuracy and efficiency of the proposed method in both unregularized and regularized versions.", "questions": "**Major Comments**\n\n* My major concerns are about the experiments in Section 4. \n1. Sections 4.2--4.3: I do not find the proposed DRO (resp. r-DRO) performs much better than LR (resp. r-LR), while DRO (resp. r-DRO) might require a longer runtime than LR (resp. r-LR). The authors should provide comparison results on both performance and time to show the accuracy-time tradeoff.\n2. As the authors discussed in Section 1, the motivation to consider the robust version of LR is that classical LR is prone to overfitting, especially in face of outliers or distribution shifts. However, this kind of setting has not been considered in experiments. The authors should provide simulations or real data analysis on these specific settings to show the advantages of DRO (resp. r-DRO) over classical LR (resp. r-LR).\n3. Sections 4.2--4.3: Only classical logistic regression (with/without regularization) is compared with the proposed method. I suggest the authors provide more comparisons with existing distributionally robust regression methods (e.g., \"Distributionally\nrobust logistic regression\" and \"Robust Wasserstein profile inference and\napplications to machine learning.\" ) w.r.t. both computational time and classification error.\n4. The authors take the medians over 20 random training set-test set splits as the final results in Tables 1--2. What if the results are taken as means (rather than medians)? Does the proposed method still outperform the competitors?\n5. For some of the UCI datasets in Table 1 (e.g., lenses with $N=24$ and balloons with $N=16$), the sample size $N$ is too small to provide a convincing classification error on the test set.\n\n* Wasserstein distance estimation usually suffers from the curse-of-dimensionality, see \"Projection-based techniques for high-dimensional optimal transport problems\". In particular, the convergence rate of the empirical Wasserstein distance may be arbitrarily slow as the size of dimensions increases. I wonder how does the number of variables affect the performance of the proposed method? Some discussion and empirical results are needed.\n\n**Minor Comments**\n\n1. The definition of $\\mathbb{B}$ seems to be inconsistent in lines 16 and line 73.\n2. In problem (4), the notations need to be defined and explained in detail.\n3. In Section 3, please add some references for the column-and-constraint generation scheme.\n4. In Section 4.1, the comparison between the left panel and the middle panel of Fig. 2 is unfair for their scales are not consistent. Please provide the log-scale runtime versus increasing $m$.", "limitations": "  ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In general, the paper is well-written and easy to follow, the motivation to consider categorical variables is clearly stated, and the theoretical results also seem to be correct.\nAs far as I have read, the work provides novel discussions and results which are not captured by the cited literature. \nHowever, I have some concerns about the numerical experiments and also some suggested corrections/additions, which are presented below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "oXUNFLadop1", "reviewer": "Reviewer_ZDos", "summary": "This paper studies the distributionally robust logistic regression problem with mixed (numerical and categorical) features. The problem is proved to be solvable by a polynomial-time solution scheme. In particular, this paper develops a column-and-constraint approach to solve the problem. This paper also shows that the distributionally robust logistic regression problem is not equivalent to regularized logistic regression as in previous works with numerical features only. When categorical features are present, this paper proves that the problem is not equivalent to regularized problems for any regularizer.", "questions": "I have a question regarding the example shown in Figure 1. By treating the categorical features as continuous ones, the authors claimed that the worst-case distribution places non-zero probability on an extreme scenario 15, 312, I am wondering would such scenarios be avoided when we restrict the range of the continuous features, like within [-1,1]. Also, it would be better if the author could comment on some theoretical comparisons between the performance of the Mixed-feature model method and the continuous-feature model in sec 2.2.\nTypos:\n1. In algorithm 2, delta \\in \\{0,1,2,...\\}, a typo in the first comma.", "limitations": "It seems that the authors do not mention limitations of the proposed method.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. This paper studies distributionally robust logistic regression with mixed features, while previous works mainly focus on numerical features only or simply treat the categorical feature as numerical ones. \n2. This paper proposes a column-and-constraint approach to solve the problem efficiently.\n3. This paper shows that the problem with mixed features is not equivalent to regularized problems, as is commonly the case for distributionally robust logistic regression with numerical features.\n4. Numerical results on multiple real datasets are provided.\n \nWeaknesses\n1. The main advantage and motivation of not dealing with the categorical features directly are only shown in the example of a worst-case distribution in Figure 1, which might be insufficient and the results could be stronger if the author could have certain theoretical comparisons. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "oPb6MRFSWrP", "reviewer": "Reviewer_Qih9", "summary": "This paper proposes a distributionally robust logistic regression with mixed-type features. \nOptimization method is considered. Some properties related to complexity analysis were given. \nA number of experiments were performed on classical datasets.  \n", "questions": "The presentation is  clear. The discussions are pretty clear. \nThe strengths and weaknesses can be found in above comments. \nNo questions are raised here. \n\n", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1.\tIt considers the expansion of recent distributionally robust logistic regression (LR) with continuous features to that with mixed features. \n2.\tThe complexity analysis is given. In particular, it proves the the non-existence of regularized version for the formulation derived from Wasserstein distance. \n\nWeaknesses: \n1.\tIt appears to be an incremental derivation from existing distributionally robust logistic regression (LR) with continuous features. \n2.\tIn terms of experimental results, the proposed formulation does not appear much better than the classical LR in terms of accuracy. Actually, it is not clear there is any significant advantages when compared to LR (Please see next comment). Without significant improvement, the new formulation is more like a conceptual exercise. \n\n3.\tThe comparative evaluation in lines 282-287 is not so convincing. Currently, only the number of best performance is counted; \n\n“The table shows that for the unregularized model, both of our distributionally robust logistic\nregression achieve the lowest classification error in 78% of the instances, whereas the classical logistic\nregression achieves the lowest classification error in 61% of the instances. For the regularized model,\nthe results remain at 78% (our model) vs. 61% (classical logistic regression). Overall, the non-robust\nmodels failed to achieve the lowest classification error in three of the instances, whereas the robust\n287 models missed the lowest classification error in only one of the instances.”\n\nThis is analogous to ‘bean counting”. \nHowever, the variance is not considered.\nIt would be desirable to show variance  information. Maybe some statistical test can be used for judge if there is any significant difference between the LR with the DROs, or between the regularized r-LR with the regularized r-DROs. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BzljoklYltS", "reviewer": "Reviewer_tKKP", "summary": "This paper addresses distributionally robust logistic regression with mixed features (i.e., continuous and categorical). Formally, the logistic regression criterion is minimized for any distribution in a Wasserstein neighborhood of the empirical distribution, see Eq. (1). The categorical nature of the features considered yields a more complex equivalent problem than in previous works [28], see Theorem 1. The authors then show the importance of considering categorical features as non continuous (Section 2.2), and that their problem cannot be rephrased as a regularized logistic regression (Theorem 3). The equivalent problem derived in Thm 1 is however NP-hard to solve for a generic loss function (Theorem 2), but the authors devise an algorithm to solve it with the log loss (Theorem 4). Experiments are also provided (Section 4).", "questions": "Major:\n- regarding the justification for not considering categorical values as continuous: why considering the absolute value and not the square difference (possibly scaled by 1/4 if necessary)? does it make any difference? how come that $z=15,312$ is considered, this distribution cannot be in the ball centered at $\\hat{P}_N$, or am I missing something here?\n- Instead of saying *generic loss*, I would specify that a linear loss is used in Theorem 2\n- I find Theorem 3 very interesting, and would give a brief description of the proof, as it holds for **any** loss, what I believe is a strong result\n- what happens if the discrepancy between measures to create $\\mathcal{B}_\\epsilon$ is not the Wasserstein?\n\nMinor:\n- l.91: $\\sum_j z_j$ should be equal to 1 or less or equal than 1?\n- l.92: given the proposed definition, $\\mathbb{C}(2)$ should be {$(0,0), (0, 1), (1, 0)$}, no?\n- display 97-98: have the dimensions aligned with the $\\beta$, i.e., put $\\beta_0$ in the end, or have $\\mathbb{R}^{1 + n + k}$, could help\n- the notation $d$ for the distance in the Wasserstein definition (2) might be confusing with $d\\xi$\n- I found formulation (6) from the appendix more readable and understandable than Eq. (4), although I understand the desire for a convex formulation. Maybe adding a remark to explain how to make the softplus constraints convex can be a solution\n- Fig. 1: add \"robust\" in the legend for the methods concerned (left), $(z, y) = $ on the $x$ axis no?", "limitations": "This is a theoretical work with no foreseeable negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- the paper is clear and well written\n- the authors provide a sound and thorough study of the problem\n\nWeaknesses:\n- I am not extremely familiar with the literature, but this work might be considered to be closely related to the reference [28], that already studies distributionally robust logistic regression, but with continuous features only. Though, I feel that the categorical features yields problems of a significantly different nature. Theorem 3 is a perfect illustration of this fact.\n- I am not 100% convinced by the proof reasoning which shows that categorical features cannot be treated as continuous ones, see my questions below.\n\nOverall I like the paper and tend towards acceptance. My only concern regards its significance, with respect to [28] in particular, although I feel that the problems derived are sufficiently different in nature to justify a novel publication. I am open to discussion with the authors and other reviewers familiar with the literature on this point.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept