Learning Mixed Multinomial Logits with Provable Guarantees
Authors: Yiqun Hu
David Simchi-Levi
Zhenzhen Yan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Mixed Multinomial Logits (MMNL)
Non-parametric estimation
Statistical learning
Provable algorithms
Sample complexity
Conditional gradient (Frank-Wolfe)

Abstract: 	A mixture of multinomial logits (MMNL) generalizes the single logit model, which is commonly used in predicting the probabilities of different outcomes. While extensive algorithms have been developed in the literature to learn MMNL models, theoretical results are limited. Built on the Frank-Wolfe (FW) method, we propose a new algorithm that learns both mixture weights and component-specific logit parameters with provable convergence guarantees for an arbitrary number of mixtures. Our algorithm utilizes historical choice data to generate a set of candidate choice probability vectors, each being close to the ground truth with a high probability. We further provide a sample complexity analysis to show that only a polynomial number of samples is required to secure the performance guarantee of our algorithm. Finally, we conduct simulation studies to evaluate the performance and demonstrate how to apply our algorithm to real-world applications.

Learning Mixed Multinomial Logits with Provable Guarantees
Yiqun Hu MIT huyiqun@mit. edu
David Simchi-Levi MIT dslevi@mit. edu
Zhenzhen Yan Nanyang Technological University yanzzōntu. edu. ag
Abstract
A mixture of multinomial logits (MMNL) generalizes the single logit model, which is commonly used in predicting the probabilities of different outcomes. While extensive algorithms have been developed in the literature learn MMNL models, theoretical results are limited. Built on the Frank-Wond propose new algorithm that learns both mixture weights and component-specifc logit parameters with provable convergence guarantees for an arbitrary number of mixtures. Our algorithm utilizes historical choice data to generate set candidate choice probability vectors, each being close to the ground truth with a high probability. We further provide sample complexity analysis show that only polynomial number of samples is required to secure the performance guarantee of our algorithm. Finally, we conduct simulation studies to evaluate the performance and demonstrate how apply our algorithm to real-world applications.
(FW) method, we
Introduction
Multinomial logit models (MNL) are widely used in variety of settings to predict the probabilities of different outcomes. It is termed as multinomial logistic regression in statistics and its formula
exp(u;)
for choosing the j-th option, Σ, exex((:)) is well known the softmax function for multi-class classification in machine learning. In choice modeling, it is type of discrete choice model used to analyze and predict people's ' decisions.
Mixed multinomial logit (MMNL) is a mixture model consisted of multiple individual MNL com- ponents and can be adopted when the target data is heterogeneous, For instance, Yang et al. [0018 demonstrates that using single sof tmax does not have enough capacity to model natural language and claims this as a bottleneck. To alleviate this issue, they proposed new method called Mixture of Softmaxes (MoS), which has higher expressiveness and can better incorporate the contextual information. Later, new techniques are proposed on top of this architecture to further boost the efficiency, such as the Mixtape from Yang et al. [019]. On the other hand, McFadden and Train (000) shows that MMNL models can lift the restrictive "independence of irrelevant alternatives" (IIA) assumption present in single MNL models as well as approximate any random utility-based discrete choice models to arbitrary degree accuracy under appropriate assumptions.
Despite its importance, the study on the learning theory of MMNL is limited. A common approach in the literature is assume a parametric family of the mixing distribution (cf Train 2009 and Berry et al. (1995)). But this approach suffers from the model misspecification issue. i.c., there could be systematical errors in the estimation if the assumed parametric family is different from the true one. Dnpectation-Maximiaaae for mixture models. But it does not guarantee the estimators' convergence to the true value of the parameters. Recent inspiring work by Jagabathula et al. (2020) creatively applies the Frank- Wolfe (FW) algorithms to estimate the MMNL model and establish the sub-linear convergence of the
(EM) algorithm is widely used non- parameerettic method
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

proposed method. However, the major issue with their approach is that the convergence result only applies the aggregate predictors, and cannot recover each individual mixture. In this paper, we propose a new Stochastic Subregion Frank- Woole (SSRFW) algorithm, which can generate each individual mixture within an e-ball to one of the ground truth vectors with a high probability.
We summarize the main contributions of this paper as follows. First, we provide learning algorithm for MMNL models that comes with provable convergence guarantees of the estimators. In particular, each generated mixture from our algorithm is shown to be within an e-ball to one of the ground truth with high probabiliiyy. To the best of our knowledge, this the first paper to prove these results for MMNL models with an arbitrary number of mixture types and minimal assumptions on the model parameters. Second, we further quantify the sample complexity of the proposed algorithm. We show that only a polynomial number of samples is required to achieve a high probability convergence guarantee.
1.1 Preliminaries
Consider a setting where the decision makers need choose among a set of [M] = .1....: : M} items, where each alternative has feature vector 2, E Rd, Furthermore, we assume the population consists of K different components (types), each associated with unique "preference vector" Gx E Rd and a mixture weight agg, where ΣK Og 1. other words, 3x. 's are not drawn from a parametric mixing distribution and consider MMNL in this paper as a type of latent class models, which is why we claim our method as non non-parametric. This is consistent with the definitions used other literature such as Train 2009 and Jagabathula et al. 2020].
Gy can also be used as user embeddings with high interpretability for decision makers who belong to the same type, which represent the corresponding type's taste on different aspects of the alternatives.
Under linear utility assumption, the probability of choosing j-th item for particular mixture k is
9 (3x) = exp(3 2,)
E ex xi)
and the aggregated counterpart for the entire population is g = Σ aug; (Bx), E [M].
To simplify notation, we use qu to denote the logit vector (B3a) [a (Bx). IM (Bx)] E RM, and qkj for (Bk),, j E [M]. Vk. We will also refer to these logit vectors as choice probability vectors.
1.2
Background
1.2.1
Related Work
Learning MMNL models remains an open-research problem. One common approach in the literature is to presume a parametric family of distributions on the parameters and apply parametric estimation methods such as the maximum likelihood estimation (MLE) method (c.f. Train (009 or the least square regression model to compute the parameters. A well-known work in this regime is by [Berry al. 1995). They assume the component-specifi parameters are normal distributed and proposed a two-step estimation method to learn the parameters using an aggregated market share data. A potential issue for this stream of methods is model misspecification, which leads to inaccurate predictions. In particular, if the assumed parametric family is different from the true one, there could be systematical errors in the estimation.
Non- parametric estimation is also widely recognized when learning MMNL models, with the most commonly known and used method being the Expectation- -atirttiiii-aaiiizazaationnn (EM) algorithm (c.f. Demp ster et al. (1977). TTain [2008] studied three different types of EM algorithms using historical choice data from each individual decision maker. The two main restrictions for the EM algorithm are 1) the algorithm can get stuck in local optimum (hence no guarantee on the convergence of the estimators) and 2) the number of mixture types needs to be pre-specified (additional heuristics are needed if such information is not known in advance, which may also introduce undesired noises).

---

Jagabathula et al. [2200]
recently developed an estimation method for mixture models also from a non-
parametric perspective based on the Frank-Wools (FW) algorithm (cf. [Frank and Wolfe 1956], D Jaggi
[2013), Jaggi and Sulovsky 2010). [Bach, 2013), Harchaoui et al. (2055] Lacoste-Julien and Jaggi
[2015]. which is an iterative method originally designed to solve constrained quadratic optimization.
They manage to establish a sublinear convergence rate of the proposed estimation method for MMNL models. But the convergence result only applies to the aggregate choice probabilities. In contrast, we provide the convergence guarantee for each individual choice probability (more details about their approach and the comparisons will be discussed in Section 2).
There is also another stream of work related to MMNL models from theoretical learning perspective. The key idea is to solve a system of equations in terms of all the parameters and prove that there is one and only one solution (identifiability). Due to the entanglement of the individual logit parameters and mixture weights, as well as the high non-linearity the system of equations, only 2-MNI. models (MMNL with two mixture types) have been studied so far. Chierichetti et al. (2018) considers the setting where the mixture weights are equal (i.e., each mixture represents 50% of the population)
and Tang [020) recently studied 2-MNL with unknown weights. This paper enriches the theoretical
learning literature for MMNL models by generalizing the analysis to an arbitrary number of mixture types and with minimal assumptions on the model parameters.
1.2.2 Ranking Models
Plackett-Luuc (PL) models is another class of model commonly used for learning preferences. It is closely related to MNI. models, with the main difference that it relies on ranking (ordinal) data.
Let X' = {I1,..... } be a set of m alternatives and denote L(X)) as the set of linear orders (full rankings) over X. A ranking R E C(X' is Ii, > Iig > .... > Fi,, where æi, and zt,,, are the most and least preferred alternative respectively. Ranking data are obtained by repeatedly selecting items after removing the previously selected items, according to the MNL choice model. Developed by
Plackett 1968 1975]. the ranking distribution for R = [2i, > Iip
> Fim is characterized by
112
P( B)
Pl X Up 24. B)
m
exp(B 2I,))
q=p exp( zin))
K
exp(3 zi,))
Similar to MMNI, we can write a mixed PL. model as P( or, 3
= _ O
-q=p (3 zi,) Despite the same parameter representation, past work focusing on learning the parameters of general class of K-PL models usually assumes the availability of ranking data or partial preference data, also known as the top-/ order data, which is the linear order of the top alternatives from the choice set [Oh et 2015 Liu et al.) 2019 Zhao and Xial 2019). This is in general very hard acquire many real-world applications. In this paper, we focus on learning MMNL model using repetitive choice data, which only assumes that we have records of which single alternative the decision maker chooses from the choice set. Such data is more widely available, which enables us to conduct a real-world case study with our algorithm (see Appendix B). In addition learning algorithms, many works focus on characterizing the identifiable conditions under different assumptions for mixed MNL or PL models (Zhao et al. 2016 Zhang et al. 2022 Zhao al. 2022
k=1 p=1
1.2.3 Motivation
While in some cases, it suffices learn only the aggregated MMNL model, there are many situ- ations where we want to obtain accurate estimation of the parameters for each MNL component. For instance, many important applications in operations research can benefit from knowing such individual mixture choice preferences, from minimizing the risk of model misspecification to offering personalized service such as product recommendation. Its capability of capturing the heterogeneity in the population makes it fundamental tool for demand prediction in revenue management and supply chain management. An important application that benefits from MMNL demand models the multi-prodee pricing problem (see Appendix C for concrete problem formulation).
3

---

Each individual preference vectors 35s can be thought as a representation of the taste over different attributes with respect to different consumer types, which can be utilized by manufacturers and retailers to better design their product design and marketing strategies. As an example, Kamakura and Russell [1989] used MMNI. to model brand preferences and created market structure that links the pattern of brand switching with price elasticities. The result provides 'managerially useful description of brand competition", which then in turn allows them to explore the characteristics of competition between national brands and private labels.
From another point of view, we can also think of B3g 's as interpretable user embeddings. The concept of embedding is widely used in a variety of machine learning settings, including natural language processing and computer vision, which is a relatively www-dimeniional vector that can be used to quantify similarities and distances between complex and/or non- non-numerical objects.
1.3
Main Results
The main contribution of this paper to provide a learning algorithm for MMNL models with provable convergence guarantees of the estimators. Specifically we will show that each of the generated choice probability vectors from our algorithm is within e-ball to one of the ground truth vectors with a high probability. Subsequently, we provide the sample complexity analysis of the algorithm. We name our algorithm the Stochastic Subregion Frank-Wole (SSRFW) algorithm, while is inspired by the Frank-Woo Wolfe framework by Jagabathula et al. [2020] but has carefully constructed stochastic feasible region that learned from the data.
The SSRFW framework assumes to work with repeated choice data, such as consumer panel or survey data. This is reasonable assumption in many world scenarios (cf. Revelt and Train [1998] and Brownstone et al. (000]). We assume the data consists of historical choice decisions from the target population of size N. Specifically, for each time period
.,,..,, T, we assume each decision maker i's choice x are i.d categorical random variables with support æ E [M]] and probability mass function (PMF) specified by qı if he/she is of type k.
For the MMNL to learnable, we need to assume that none of the individual MNL is identical the rest. Let F and G be the cumulative distribution functions (CDF) for two different choice probability distribution Q and qı" respectively. Define the Saanoggrov-Smmrnno (KS) distance between the two distributions as [Ks(qR, qie) sup, F(x)) G(x)]. Definition I below is adapted from a similar
assumption introduced in Kalai et al. 2010 to impose some regularity the MMNI. model.
Definition 1. We call a mixed multinomial logit model (MMNL) g e-standard for any given tolerance e
1. ming o% 26
2. 3kd S 2 4kk
3. DKs(ge, )1) #
Definition] [ Nimposes the following properties on the MMNL model: 1) none of the mixtures has zero weight; 2) all parameters in the model are bounded; and 3) each individual logit model has some level of separation from others. We consider this as a very mild assumption in our setting, as later we will see that we will not predetermine the number of mixtures (K) and if mixture/component not present in the observed data (since it has zero weight), the algorithm will simply not learn it. This is also one of the advantages of our algorithm over the traditional approaches, such EM.
The main result is as follows:
Theorem 1. Let g Ek a&qu be mixed multinomial logit (MMNL) model over a set of M
items. Assume M 2 K. For any > O, 0 < 5 < 1, Algorithm I outputs an MMNL g- -K Quqq where K' 2 K such that, with probability 2 1 - 5. there exists a many-to to- mapping म. k' k. [K',, k E [K] such that
1g (x)))) < <,,.K'
and
Σ áv - as
< <,, V k.
Wor(k)=

---

where ||
||| is the Euclidean norm. The number of samples required by Algorithm 1 is ( log( :
The outline of the paper is as follows: Section and Section 3 will introduce the main algorithms, namely the Stochastic Subregion Frank-Wolfe (SSRFW) and the Q-construction algorithm, where the output of the latter serves as an imput to the former. In Section 4, we will discuss the theoretical properties of the SSRFW algorithm. We conduct simulation studies in Section 5, All the detailed proof's and parts of simulation studies are included the appendix.
2 Stochastic Subregion Frank- Wolfe Algorithm
Recall that observe each decision maker i's choice at time L: x!! (as realization of x!"). Let Y [0.-..,,, ...... _] RMM, where y = if x (t) probability vector for period can be expressed as
j. The observed aggregated choice
N
Y i=1
where yd E RRM, Define P (a( E R | E R2). the set of all valid logit vectors for the given choice set and their features. Denote P its closure. Since g Σ au by definition we have g E CPn
Our main learning objective is to minimize the difference between the observed (from data) and theoretical aggregated choice probability, as follows
min gEConv
C(g) = min
g y"1
geConv
Note that problem is non-convex problem in og and B3 and is in general hard to solve. We adopt the conditional gradient approach proposed by Dagabathula et al. (2020), which we will refer to as the original FW algorithm. Each iteration is consisted of two steps: 1) the supporting finding step which searches for a new direction represented by choice probability vector (kk) (as new latent class k) via solving linear optimization subproblem in the search space Conv (P), and 2) the proportion update step which updates the mixture proportion a,, i = O, ..... ki assigned to all choice probability vectors obtained so far. The final learning outcome g is E anq (8) where
(0) = =0) E P is a random initialization. This setup corresponds to a fully-corrective variant of the generic Frank-Wolfe algorithm, and is guaranteed to find the optimal solution to the optimization problem Paggi 2013 for the decision variable g-
The problem with the original FW approach that the feasible region Conv(P) contains the complete set of all logit vectors and their limiting points. The former corresponds to non-boundary types, whose choice model can be characterized by standard MNL choice model, and the latter corresponds to boundary types, whose standard MNL parameters become unbounded, resulting infinite utility for some options and zero for the rest. Such broad search space Conv(P) in FW generates mixture compositions with a considerable number of the boundary types. More specifically, this problem is rooted to the linear subproblem solved at each support finding step, whose optimal solution always lies on the extreme point of the feasible region which contains many of these limiting choice probability vectors P \ P its extreme points. Often, each limiting choice probability vector corresponds to an unbounded Be and cause the learning outcome unusable for downstream applications. Though significant effort is made to justify the legitimacy of these boundary types the paper, the convergence result only applies to the population's aggregate choice probability, i.e. g - Ex 0s-qu, which is an Frank-Wolfe property. In other words, their method fails to recover each of the individual logit models (Bx) and the corresponding mixture weights (ax) accurately.
To remedy this issue and seek convergence guarantees for the general MMNL estimation problem, we designed the Stochastic Subregion Frank-Wolfe (SSRFW) algorithm, as shown in Algorithm []]
Note in the above algorithm, after à iterations, have generated & choice probability vectors ((t), 1,...,k k. all of which will be the output of the learning algorithm. The mixture weight
5

---

Algorithm 1: Stochastic Subregion Frank-Wolff
Input: data y, from Algorithm
Initialization: & = 0; ( [1) random g( (0)) chosen from
while stopping condition not met do 1 x+* h +
Compute (*)) arg min (VL (g!h-1))
g 1g(h-1))
veConv
Compute !k) arg min L acAc
(k)
(0)
((a)
ag
B
Update (8) = 00 g0)
(k)
(0)
+
end Output:choice prob. q (0)
(k)
mixture weights, (k) E Ax RR+11
estimator is slightly different. a (*) R++ represents the mixture weight vector for the k + choice probability vectors we have generated so far, where a² refers to the 8-th element in the vector, corresponding to the mixture associated with ((s), In other words, the vector length of increases by for each iteration and only the final vector (f) is outputted from the algorithm.
The key different in SSRFW is that it takes in an additional input Q from the Q Construction Algorithm (discussed in detail in Section 3). which is the key to eliminate the possibility of producing boundary types. In particular, the new feasible region Conv( () is subset of Conv(P) where each element q E Q will be learned from data and guaranteed to within an e-ball of the true choice probability vector for some mixture type with high probability. Subsequently. this also ensures the SSRFW algorithm to recover the mixture weight for each latent class. Note that the extreme points of Conv( Q) by definition is subset of Q. Since we only care about the extreme points of the feasible region, we can safely ignore any points Conv( \ Q and replace Conv( with Algorithm
Since Frank-Wolfe will converge to the optimal solution [aggi 2013 and as we will see in Section 4 that the optimal solution problem is an interior point of the feasible region Conv( (), the stopping condition can simply be set
y S.
3
Q Construction Algorithm
Our primary goal in this section is to learn the input set Q for SSRFW (Algorithm 1 In particular for
this set Q {ge J.m...... we require that V dr. there exists some mapping [Z] [K] such that
GE (e) < e, where (n(()) is of the ground truth logit vectors. Once this property holds, each
of the SSRFW 's q outputs will also be e-close to some true choice probability vector as the algorithm
always selects an extreme point of Conv( Q) in each iteration.
The two major components for the Q Construction Algorithm is first computing a distance score matrix (Section 3.1 and then creating subsamples that contain decision makers from only mixture type through random seeding and non-uniform sampling (Section 3.2). Section 3.3
presents the
algorithm and discusses the properties of the learning outcomes.
3.1 Distance Score Matrix
We first define distance matrix S, whose element Ssy measures the dissimilarity between any two decision makers and j in term of their choice decisions. Recall that each z are realizations of i.i.d categorical random variables with PMF qu if i is from mixture k. Denote its associated empirical cumulative distribution function (CDF) as Fr(2; i) when a total of T decisions are observed. In
A {(ai,.... ak+1) #+1 Dku ag o, 2 0)

---

particular,
T
(r =
Σ 1
sxp [M]
T
lo
(=1
Define the distance score function for each pair of decision makers
Sij == si, (1;((x;) Fr (z; (2:5)]l
where ||
|/ represents the infinity norm.
3.2
Subsample Construction
We now want create number of subsamples such that each subsample of decision makers contains only one mixture type with high probability. Each of these subsamples is generated following a procedure that contains random seeding step, and subsampling step. These steps repeated L times to create a Q set size L.
Random Seeding. This step randomly samples a decision maker E [N] from the population and can be done with simple uniform sampling technique. The selected decision maker called a seed. We will generate L seeds and later investigate how large L needs to be in order to cover all mixture types in Q
Subsampling Strategy. This step generates index set I(i) for a seed such that, with high probability, decision makers whose indices belonging to this set are of the same type as [(t) called the subsample originated from seed i.
Given selected seed we first calculate an accepting probability Pjts which determines the likelihood of another decision maker j being accepted the corresponding subsample [(i). is designed be a monotonically decreasing function with respect to the distance score we defined in Section 3.1
Pyle (s(i.j))
simple example can be Pjlı = 1 - j). Intuitively, when s(i. is small, seed and decision maker j's empirical CDF is close each other, indicating there is a higher chance that they of the same type and sharing the same choice probability q. Subsequently, Pyte will be larger compared to a large value of sii,j") for another person j', resulting we accept j with higher probability than j'. This is consistent with objective that we want each subsample to be composed of decision makers of the same type as the seed.
For implementation, we first initiate an empty index set [(2) for the selected seed i. Then we repeat the following two steps until we reach a desired subsample size n: 1) draw random sample from set [N] T(i), 2) accept this sample into the index set with probability PyІs and reject with Palé-
3.3 Algorithm and Properties
The remaining step obtain the whose elements ÿr are computed as the average of historical
T
1
r, Finally,
choice decisions from decision makers in the subsamples Lt i.c. q
nf
SEL
1=1
present the Q Construction Algorithm in Algorithm
Discussion We next discuss a few properties the two algorithms.
SSRFW returns the logit vectors q instead of the parameters 3. We can simply perform maximum likelihood estimation under the single MNL setting where, if the extreme point corresponding to subsample is selected during an SSRFW iteration, the likeliiiddood is
T 1(())= = -E Y!
log exp( 2,y) EmEIM exp(3 2m
HEL JE[M]
and we have 3 arg max Le()). With the assumption of linear utility in B,, we can obtain the optimal solution using MLE (c.f. McFadden 1974).

---

Algorithm 2: The Q construction algorithm
Input: score matrix S, number of subsamples I,, subsample size Initialization: Q = set()
for é to do
2
Choose seed: i ~ U (0, N)
Initiate: 4 set()
while 4/ + n do
{ randomnnsseee @attoom_smmmeme_ee (M) It)
Generate u ~ (0,
if < Pjjs then (j)
end
10
end
T Σ
11
Compute de-
nT
iET tml
12
Q.add((:))
13
end
Output: Q
In general, we prefer learning Br 's compared to qu s. This makes the algorithm more robust when item attributes change over time which results in change in the q 's whereas Gr values persist. To achieve this, we just need to add two additional steps after the index set is created:
MNL paramter estimation: apply the MLE step outlined above; create (Bie (=1)..
Map B to Q: fix item attributes of current interest, calculate Q { = 3. E B}
Next, stochasticity in our algorithm comes from two parts, namely the realization of each decision makers' choices, as well as the randomness in the process of generating subsamples. Yet we will show that under appropriate assumptions, SSRFW can still recover all individual mixture parameters with high probability.
In addition, we do not impose any restriction on the subsamples created during construction be mutually exclusive. As long as they only contain one single mixture type (i.e. with high purity), it creates a legitimate estimation of q or B3 with MLE.
Last but least, this is a more robust strategy than clustering algorithms segment the population. Not only do we not require the number of mixtures K as hyper-parameter each decision maker's data being used more than once for the estimation can be thought of special bootstrap mechanism that has custom weights tailored to our objective in creating homogeneous subsamples.
Theory of the SSRFW Algorithm
Theorem I is the main result of this paper. To prove it, will break down this section into two parts, where the first part will examine the provable convergence properties of the estimators, and the second part will focus on the sample complexity. We will outline the proof sketch but all detailed proofs can be found in the Appendix.
Provable Convergence. Assume K' is the total number of iterations that the algorithm has per- formed before reaching the stopping criteria. To simplify notation, we denote the generated outcome from k-th iteration q (()) in SSRFW as q& to indicate they are the estimators and the final updated
(K") dy. In other words, we have sSsra mixture weight corresponding to (*)), ag
K" E .u.q
We use a similar notation r for the original FW approach.
Figure describes the necessary components and steps of proving the result. In particular, Property and (purple boxes) are adapted from existing results of the Frank-Wolfe framework [Jaggi] 2013) In Appendix I.I.I prove Property 1 in Lemma 2 Property 2 follows from the fact that each of

---

Property 1: E. auqul S
1g"
Property 3: >i qu Conv(
gut 2n aceul s
Og au
Property 2: V.. .ur C(Conv (P))
AK = x(k)
S... O qul S
Property A E(Conv( 3 l1q qull
FW properties
SSA properties
Figure Proof sketch of Thoerem
the Os, obtained by solving linear optimization problem, has to be an extreme point (denoted by (.)) of the feasible region.
On the other hand, Property 3 and 4 (red boxes) hold as a consequence of the stochastic subregion we constructed in Algorithm - Property 3 claims that with high probability, the ground truth aggregated choice probability vector is still an interior point of the shrunken feasible region Conv () (proof in Appendix A..1.2.. Together with Property 1, this tells us that Ig S5RFM Σ a S c. That is to say the aggregated choice probability vector of SSRFW also converges to the ground truth. To achieve Property from the Q construction algorithm, we need to supply enough of data, the details of which will be discussed in the sample complexity part. Combining Property 2 and 4, we have W,, 3 = x (k') s.t. 19 qul S e. Finally, by doing some algebraic manipulation, we can obtain
EM age
ak S.
Sample Complexity. Recall we use T as the number of choice data records for each decision maker. Denote Tmin as the minimum number required and T Tin 2
Define in-types to be the decision makers who share the same type as the seed, and our-rypes to be the ones of different types. We focus on the following three properties:
probability of accepting the in-types,
probability of rejecting the out-types,
probability of covering all mixture types.
Consider seed i, with type k; = k. Assume there is a total of m. decision makers in the population that is of type k:
Accepting In-Types. Denote the empirical choice probability for the seed i as F, and that for
another decision maker j as Gm, with and m representing the number of independent price experiments from i and j. respectively. Recall that s(i,j) = F (z) Gm |loe = sup, [z) (z)) For exposition simplicity, we use and to denote F F(x) and G(x) the subsequent presentation if no confusion incurred.
Theorem (Sample Complexity I). Assume and j are of the same type. Define Pats =1 s(i,j) Ag 0, 6 we can achieve Pite 1 with probability least 1 8 with sample size Tiin O log (1)).
Theorem 2 indicates that with sufficient amount of data, there is high probability to accept en-type-who shares the same type as the seed-into the subsample if is chosen from the population after the random draw.
Rejecting Out-Types. Assume a decision maker j is of different type from the seed i. We use F and G denote their CDFs. Let F, and Gmm represent the corresponding empirical CDFs based on

---

samples drawn from the two different distributions with sample size n and m, respectively. s(i, j) is defined the same as before.
Theorem 3 (Sample Complexity II). Assume F and G correspond to the choice CDFs of two different rypes and sup, (f()) G(xz) 2 &. Let Fn. and Gm be independent empirical distribution based on m and n. i.idd. samples frawnfrrmm F and G. respectively. Denote Tiin min (m,n.) Define
Pals s(i.j). 46 > 0, > O, we can achieve Pja < E+ with probability at least 1 with Tmin = ( log
Theorem 3 complements Theorem 2 to rule out the out-types from being selected with high probability. The more separable (larger {) their underlying distributions are, the lower accepting probabilities are.
Lemma For E 0 and < 1, with minimum of choice records T required by Theorem
2 and Theorem 3 V
E
Q qe (=(0) S with probability at least 1 - 6, where q =
RT Eier,
Lemma D states that with enough data samples, the Q-construction algorithm achieves Property 4, as shown in Figure which SSRFW's desired provable convergence builds upon. With Lemma T we conclude the proof for Theorem |
The complete proof can be found in Appendix A.4
As final remark, discuss the number of subsamples (i.e. L) needed in the Q-construction process. We distinguish this from sample complexity as we are not requesting more data points with larger L. Instead, we should view L computational complexity factor. Recall that each moving direction derived from the support finding step is viewed as learned logit vector of one mixture component. Therefore, we want the D-construction algorithm to include least one qe for each mixture &s in order for it to be picked in SSRFW This impose requirement on L. Theorem states that the expected number of the number of subsamples is controlled by the smallest mixture weight.
Theorem 4. Assume 01 S a2 S a.g The expected number of subsamples L we need to construct
bounded by a1
log m1
Intuitively, this means if we have one mixture component that is very under-representte, then we need to create more subsamples to ensure that one of the seeds belongs to this mixture.
5 Conclusion
In this work, we propose the SSRFW algorithm, which provides an end-to-end solution for learning MMNL models using historical choice data. This novel approach utilizes a carefully designed sampling method to construct meaningful search space. Not only does resolve the drawback of the original Frank-Wolfs approach with boundary-type issues but also enables us to obtain provable guarantees for the model estimates and sample complexity. Another major advantages of our algorithm is that we do not assume prior knowledge of the number of mixtures K in the model. It is also more robust than traditional unsupervised approaches such as clustering method and EM.
In marketing science and operations research, MMNL provides fundamental tool for demand prediction, which many downstream applications such as pricing and inventory position rely on. For instance, consider pricing model that tries to maximize the revenue by solving the following
-MM optimization problem: maxp Σ/= P, Ex= 0xg9; (Bx: P) where pj- 2 0, V E [M]] This requires an accurate estimation of each individual MNL components. Due to page limit, include our simulation studies and application in real-world setting in the appendix. We will also briefly discuss the requirements the datasets in order to apply our algorithm.
Acknowledgments and Disclosure of Funding
The research of Yan was partly supported by Nanyang Technological University startup grant and MOE Academic Research Fund Tier [Grant RG17/21] and Tier 2 [Grant MOF2019-1221-0455 and NOL Fellowship grant [NOL2IRPO4)
10

---

References
Francis Bach. Learning with submodular functions: A convex optimization perspective. Foundations and Trends in Machine Learning, 6(2-3):15--773, 2013.
Steven Berry, James Levinsohn, and Ariel Pakes. Automobile prices in market equilibrium. Econo- metrica: Journal of the Econometric Society, pages 841-890, 1995.
David Brownstone, David S. Bunch, and Kenneth Train. Joint mixed logit models of stated and revealed preferences for alternative-fuel vehicles. Transportation Research Part B: Methodological, 34(5):315-333 2000. ISSN 0191-2615.
Flavio Chierichetti, Ravi Kumar, and Andrew Tomkins. Learning a mixture of two multinomial logits. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 961-969, PMLR, 10-15 Jul 2018.
Arthur P Dempster, Nan M Laird, and Donald Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1): 1-22, 1977.
Marguerite Frank and Philip Wolfe. An algorithm for quadratic programming. Naval research logistics quarterly, 3(1-2):95-110, 1956.
Zaid Harchaoui, Anatoli Juditsky, and Arkadi Nemirovski. Conditional gradient algorithms for norn-regularized smooth convex optimization. Mathematical Programming, 152(1-2);75-112, 2015.
Srikanth Jagabathula, Lakshminarayanan Subramanian, and Ashwin Venkataraman. A conditional gradient approach for nonparametric estimation of mixing distributions. Management Science, 2020.
Martin Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In Proceedings of the 30th international conference on machine learming, number CONF, pages 427-435, 2013.
Martin Jaggi and Marek Sulovsky. A simple algorithm for nuclear norm regularized problems. In ICML, 2010.
Adam Tauman Kalai, Ankur Moitra, and Gregory Valiant. Efficiently learning mixtures of two gaussians. In Proceedings of the Forty-Second ACM Symposium on Theory of Computing, STOC 100 pages 553-562, 2010.
Wagner Kamakura and Gary J. Russell. A probabilistic choice model for market segmentation and elasticity structure. Journal of Marketing Research, 26(4):379-3) -30,, 1989. ISSN 00222437. URL httt:://ww..r jstor org http:/staaeeee////eeeee//
Simon Lacoste-Julien and Martin Jaggi. On the global linear convergence of frank-wolfe optimization variants. In Advances in neural information processing systems, pages 496-504, 2015.
Ao Liu, Zhibing Zhao, Chao Liao, Pinyan Lu, and Lirong Xia. Learning plackett-luce mixtures from partial preferences. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01): 4328 4335, Jul. 2019.
D. McFadden and K. Train. Mixed mnl models for discrete response. Journal Applied Econometrics, 15(5):447-470, 2000.
Daniel McFadden. Frontiers Econometrics. Academic Press, New York, 1974.
Sewoong Oh, Kiran K Thekumparampil, and Jiaming Xu. Collaboratively learning pref- erences from ordinal data. In C. Cortes, N. Lawrence, D.. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, Cur- ran Associates, Inc., 2015. URL https: http:://prroae
wttpssserrr.p...pprrre///// Paper pdf
R. Plackett. Random permutations. Journal of the Royal Statistical Society: Series B (Method- ological), 30(3):517-534, 1968.

---

R. L. Plackett. The analysis of permutations. Journal of the Royal Statistical Society. Series C (Applied Statistics), 24(2):193-200) 1975.
David Revelt and Kenneth E. Train. Mixed logit with repeated choices: Households' choices of appliance efficiency level. Review of Economics and Statistics, 80:647-657, 1998.
Wenpin Tang. Learning arbitrary mixture of two multinomial logits, 2020.
Kenneth E Train. Em algorithms for nonparametric estimation of mixing distributions. Journal of Choice Modelling, 1(1):40-69, 2008.
Kenneth E Train. Discrete choice methods with simulation. Cambridge university press, 2009.
James. G. Wendel. A problem in geometric probability. Mathematica Scandinavica, 11(1): 0991111, 1962.
Zhilin Yang. Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the softmax bottleneck: A high-rank mn language model. In ICLR, 2018.
Zhilin Yang, Thang Luong, Ruslan Salakhutdinov, and Quoe V. Le. Mixtape: Breaking the softmax bottleneck efficiently. In NeurlPS, 2019.
Xiaomin Zhang, Xucheng Zhang, Po-Ling Loh, and Yingyu Liang, On the identifiability of mixtures of ranking models, 2022.
htatpp:://axxiv.org/abb////// 13132
Zhibing Zhao and Lirong Xia. Learning mixtures of plackett-luce models from structured partial orders. Advances in Neural Information Processing Systems, 32, 2019.
Zhibing Zhao, Peter Piech, and Lirong Xia. Learning mixtures of plackett-luce models. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages
14. PMLR, 20-22 Jun 2016.
Zhibing Zhao, Ao Liu, and Lirong Xia. Learning mixtures of random utility models with features from incomplete preferences. In Lue De Raedt, editor, Proceedings of the Thirty- First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 3780-3786. org, 2022. doi: (244966/jjcii222222/53 URL hotph:ttpoooo org 10. 24963 ijcai 2022/525
12

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Appendix B.
Did you include the license the code and datasets? [No] The code and the data are proprictary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix B.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 1, Section 4, and Appendix
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 4, and Appendix A.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes] See Appendix
B.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix B.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Appendix B.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] See Appendix B.
(b) Did you mention the license of the assets? [Yes] See Appendix B.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "yZXbjS5n9eV", "reviewer": "Reviewer_zoBe", "summary": "This paper studies the recovery of mixed multinomial logits using Frank-Wolfe algorithms. The authors proved that the proposed algorithm SSRFW converges to the true parameter of each individual mixture model. They also provided a sample complexity analysis of the proposed algorithm. \n", "questions": "Regarding the convergence of Algorithm 1 and Algorithm 2, I wonder whether the authors considered the computational complexity of Algorithm 2 in the analysis of the complexity of SSRFW. Specifically, how many iterations do step 4 of Algorithm 2 take to converge? And what is the order of $L$ in Algorithm 2 in order to get a good estimate of the input set $\\mathcal{Q}$?\n\nCan the authors elaborate more on the reason for obtaining the individual mixture? For constrained optimization with an aggregated objective, isn’t it enough to just know the aggregate prediction?\n\nWhat is $\\alpha$ in Line 4 of Algorithm 1? It does not appear in the objective function. How are you going to minimize $\\mathcal{L}$ over it? Moreover, the footnote says $\\alpha$ is a $k+1$ dimensional vector, while in Line 171 of the paper, the authors say $\\alpha^{(k)}=\\alpha$ is a $k$-dimensional vector (by Line 4 of Algorithm 1). This inconsistency makes it hard to understand the algorithm.\n", "limitations": "For the experiment results, I think it would be more reasonable to also compare the proposed method with those in recent related papers such as Chierichetti et al. [2018] and Jagabathula et al. [2020]. Though these baseline algorithms may work in a reduced setting or have different theoretical guarantees, their methods should still be applicable to the empirical studies in this paper. Therefore, it would be interesting to see how they compare with one another in the experiments. \n\nLin 336: “reply” => “rely”\n", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is generally well written and the results are interesting. Some clarification on the sample complexity of the subroutine should be added to the analysis. The comparison in the experiments could also be improved. See my specific questions in the next section. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N6imrnvFwFN", "reviewer": "Reviewer_VDBV", "summary": "The authors provide theoretical results on learning mixed multinomial logits. The main contribution is a new algorithm that learns both mixture weights and component-specific logit parameters with provable convergence guarantees for an arbitrary number\nof mixtures.", "questions": "See the main comments.", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n1. The paper is clearly written and the theoretical results look solid. \n2. Sample complexity is established for the proposed algorithm.\n\n[Weaknesses]\n1. The proposed algorithm performs EM-like updates. While sample complexity is provided, the algorithm could take many iterations to converge, and potentially converge to a local optima. How does the algorithm address these issues?\n2. Both \\alpha and \\beta depend on the parameter \\epsilon in Definition 1. Since the magnitude of z could be arbitrary, is the condition on \\beta necessary?\n3. I think the authors forgot to include the simulation studies as suggested in the abstract. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KXOc8nPKdq", "reviewer": "Reviewer_oo6h", "summary": "The paper proposed a new method for learning mixtures of multinomial logits based on historical choice data. The proposed method is based on Frank-Wolfe while restricting to a narrowed set of estimated choice probability vectors. The paper provided theoretical results on convergence and sample complexity. The paper provided numerical experiments to demonstrate the performance of the proposed method. \n", "questions": "- Line 34 : \"But it does not guarantee the estimators’ convergence.\" should better be \"But it does not guarantee the estimators’ convergence to the true parameters\", because EM algorithms do converge to something. \n\n- The authors could clarify why the problem considered in the paper is important and particularly relevant to the ML community. \n\n- The authors may provide more details on the underlying model for the historical choice data. Specifically, how were the parameters $z_j$'s (line 51) connected to the decision maker's choice probability q_k? Also, what's the usual relationship between N and T？ Do we consider a large N regime, large T regime, or large N large T regime?\n\n- Theorem 1: what is the sample dependency on K (or L) and M? Also, is there a convergence guarantee for $\\beta$'s in addition to q and \\alpha?\n\n- The method doesn't need to choose K but needs to choose an additional parameter L. How do you choose L in practice? \n\n- Algorithm 1 step 3 : $v$ was not defined. \n\n- Algorithm 1 : what's the stopping condition? ", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths. The proposed method seems effective for learning multinomial logit mixtures. The theoretical results are strong and solid. The paper is overall well-written. \n\nWeaknesses. The paper specifically considers historical choice data, which should be reflected in the title. It seems a narrow topic of learning multinomial logit mixtures using historical choice data, which may reduce the paper's significance. The Preliminaries subsection is a bit brief and it would be more helpful to formally define the historical choice data. \n\n\n- Line 35: typo of \"Frank-Wolf (FW)\"\n\n- Line 330: typo of \"Not only does it resolves ...\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8hasLbOqoOD", "reviewer": "Reviewer_Ldhd", "summary": "The authors provide theoretical analysis of a new \"Stochastic Subregion Frank-Wolfe Algorithm\" for learning mixed multinomial logits. The authors provide provable guarantees showing a polynomial number of samples is sufficient to achieve the performance guarantees. Simulations are provided in the supplementary material.", "questions": "Below are several questions that may be worth a clarification:\n\n* The results conveyed through Theorem 1 only depend on $\\epsilon$ and $\\delta$, and do not take into account other problem parameters such as $K$, $M$, etc. How do these parameters affect the sample complexity?\n* Are there any information-theoretic impossibility results that can be used for comparison? For example, results that show a relation between feasible expected error compared to number of samples? Is the dependence on $\\epsilon$ and $\\delta$ already theoretically optimal, or is it possible that algorithms with better complexity exist?\n* What is the computational complexity of the algorithms compared to previous algorithms such as MLE or previous algorithms that achieve sub-linear sample complexity for aggregate probabilities?\n* In the supplementary material, several details are swept under the rug. For example, line 457 states that \"enough sample\"s are sufficient to give a performance guarantee by law of large numbers, but the whole discussion of that section relies on a precise characterization of the sample complexity --- can this be clarified? In other words, what is the precise sample complexity for the statement of line 457 to be true and does it affect the final sample complexity claims in the main text?\n* Finally, in the simulations, although the authors claim that \"EM achieves a marginally better performance\" in certain scenarios in line 655, Figure 6-(b) does not suggest that and it appears the EM with suitably chosen $K$ has much better performance especially at larger epochs. While the authors clarify that EM \"suffers from its instability\", it does not appear to be a significant factor in this experiment. Can this be clarified?", "limitations": "Limitations are not thoroughly discussed in this submission. It would be beneficial to discuss limitations in more detail.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the topic and results are interesting. The smoothness of reading is affected by some typos that can be improved through further polishing.\n\n**Originality.** The authors provide a novel algorithm that can be considered a variation of the Franke-Wolfe algorithm with a component that takes into account a \"feasible region\". This concept is quite interesting. Using this concept, the hyperparameter $K$ is not required in the algorithm which makes it appealing compared to algorithms that require knowledge of $K$.\n\n**Quality.** The paper overall is interesting. The algorithms presented make sense. See below sections for a few questions (see below Questions section) and several notational confusions (see below clarity). \n\n**Clarity.** The clarity in the paper is mostly obstructed by typos and notational issues. Below are several points during my review:\n* Clarity of notations and typos. For example:\n    - defining \"$\\|\\| \\cdot \\|\\|$\" before it appears in Theorem 1.\n    - in the description of Theorem 1, the summation should be over $k'$ and not $k$. \n    - there are multiple typos throughout the submission but minimally affects the overall read. It would be beneficial to fix these typos in further revisions.\n\n**Significance.** The MMNL model, Frank-Wolfe algorithm and related topics benefit a range of real world applications and therefore related research, including this paper, is important. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept