Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: 1.The goal of the paper is to connect flexible choice modeling with a modern approach to ML architecture to make said choice modeling scalable, tractable, and practical. 
2. The approach of the paper is well motivated intuitively, but could more explicitly show that PCMC-Net is needed to fix inferential problems with PCMC and that e.g. SGD and some regularization + the linear parameterization suggested by the original PCMC authors isn't scalable in itself. 
3. The approximation theorem is useful and clean, and the empirical results are intriguing. While consideration of more datasets would improve the results, the metrics and baselines considered demonstrate a considerable empirical case for this method. 

My "weak accept" decision is closer to "accept" than "weak reject." (Edit 11/25/19: I raised my score in to accept in conjunction with the author's improvements in the open discussion phase)

Improvement areas(all relatively minor):
- While I personally enjoy the choice axioms focused on by the PCMC model and this paper, stochastic transitivity, IIA, and regularity are probably more important to emphasize than Contractibility. Because the properties of UE and contractibility were not used, it may be more appropriate to use this space to introduce more of the literature on neural-nets-as-feature-embeddings stuff. 
- This paper could be improved by generalizing to a few other choice models- in particular the CDM (https://arxiv.org/abs/1902.03266) may be a good candidate for your method. This is more a suggestion for future work if you expand this promising initial result. 
- Hyper-parameter tuning: I noticed that several of your hyper parameters were set to extremal values for the ranges you considered. If you tuned the other algorithms' hyper parameters the same way, it could be the case that the relative performance is explained by the appropriateness of those ranges. Would be interesting to have a more in-depth treatment of this, but I do understand that it's a lot of work. 


Specific Notes:
Theorem 1 is nice, and the proof is clean, but doesn't explicitly note that a PCMC model jointly specifies a family of distributions \pi_S for each S \in 2^U obtained by subsetting a single rate matrix Q indexed by U. It's clear that PCMC-Net will still approximate under this definition, as \hat q_ij  approximates each q_ij because \hat q_ij doesn't depend on S. While the more explicit statement is true with the same logic in the theorem, the notational choice to have "X_i" represent the "i-th" element in S is confusing at first, as e.g. X_1 is a different feature vector for S = {2,3} and S={1,3}. I don't see this issue as disqualifying, but it took me a while to realize that there wasn't more than a notational abuse problem when I returned to the definitions where the indexing depended on the set S under consideration. 


Typos/small concerns:
-Above equation (1), the number of parameters in Q_S is |S|(|S|-1) rather than (|S|-1)^2, as each of the |S| alternatives has a transition rate to the other |S|-1 alternatives. 
-Below equation (3), I think you mean j \in S rather than 1<= j <= |S|, as S may not be {1,2,...,|S|}. Later I noticed that you always index S with {1,\dots,|S|}, but using i \in S in combination with 1<=j<=|S| was a bit confusing. 
-X_i as the i-th element of S is a bit of an abuse of notation, as it surpasses dependence on S
-In Figure 1, you show X_0 in a vector that is referred to as "S." It is my understanding that X_0 represents user features. As the user is not in the set, this is confusing. The use of a vertical ellipsis to connect \rho(X_0) to \rho(X_1) is also confusion, as \rho(X_1) is input into the Cartesian product while X_0 is input into the direct sum. 

Overall, nice job! Really enjoyed the paper and approach, good to see connections made between these literatures so that progress in discrete choice can be used at scale. 

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: Summary:  This paper enables a feature-based parametrization and amortized inference of Pairwise Choice Markov Chains (PCMCs), a model for decisions in the face of a set of alternative choices (e.g. the rock-paper-scissors game).  Previous approaches to fitting PCMCs have leveraged sequential least squares programming, making optimization unstable, the model prone to overfitting, and test-time inference difficult.  The authors propose parametrizing PCMCs with neural networks to fix these issues.  Relying on universal function approximation results, the authors show that their PCMC-Net can represent arbitrary transition matrices.  The experiments report results on a dataset of airline booking behavior, comparing PCMC-Net with four other baselines from the literature.

Pros:  Although I was previously unfamiliar with the PCMC model, using a neural network parametrization seems novel and well motivated.  Moreover, the airline data experiment seems to validate that PCMC-Net is indeed effective, besting the other baselines in all three metrics.  

Cons:  I have two main concerns over the paper’s experimental rigor…

#1 Lack of Simulation Studies: The paper makes claims about the representation properties of PCMC-Net but fails to validate them with simulation studies.  

#2 Lack of data sets: Only one experiment on one data set is reported. 

Final Evaluation:  While I found the paper’s methodology well motivated and sensible, the experiments do not thoroughly validate the method as they contain no simulation studies and only one data set.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper introduces a novel approximate inference method, called PCMC-Net, for models from the family of Pairwise Choice Markov Chains (PCMC). The method relies on training a neural network. Consequently, the authors claim that inference is amortized, but its computational complexity is still quadratic in the number of choice alternatives due to separate processing of all pairs of alternatives. PCMC-Net bakes the definition of PCMC into the neural net structure and therefore satisfies the theoretical properties of contractability and uniform expansion, which are desired properties of choice models
Moreover, since choice probabilities are a function of choice candidates’ features (and features of an individual making the choice), this method allows for new (unseen) choice candidates at test time, which was not possible with previously proposed maximum-likelihood (ML) inference. The approach is evaluated on modelling the choice of airline itinerary, on which it outperforms all considered baselines by a significant margin.

I recommend REJECTing this paper. This paper tackles the problem of efficient inference and test-time generalization (to unseen choice alternatives) for choice modelling, and the proposed approach is interesting, seems to be theoretically sound, and outperforms evaluated baselines. Experimental evaluation is insufficient, however, with the method assessed only on a single dataset---in which case it is unclear if the method is better than baselines in general, or whether it is a quirk of the considered dataset. Moreover, the authors do not compare to ML inference in PCMC, which seems to be the closest possible baseline; instead, the authors only mention that ML would overfit on this dataset. Finally, the paper is full of complicated terms and cumbersome notation, which makes it difficult to read. Technical terms are often used without definition (e.g. framing effects, Luce’s axiom, asymmetric dominance), which makes the paper inaccessible to an inexperienced reader like myself.

I think that this work could be improved in the following ways. The exposition should be made simpler and easier to follow (especially section 2), and all technical terms should be appropriately defined. Additionally, the method should be evaluated on at least one more dataset and compared to ML inference for PCMC. I am happy to increase my score if (all) the above points are addressed.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper presents an approach for choice modeling that leverages neural network features in a continuous-time Markov Chain whose stationary distribution represents the choice distribution. Nonlinear features can be computed for both alternatives and individuals, and the resulting model beats all baselines in terms of log-likelihood and accuracy on an airline itinerary choice prediction task.

Overall, this paper presented a simple but effective approach for using neural networks in the PCMC class of models. The experimental section is too limited, with results on only one dataset and no comparison of different architectural choices for how to incorporate neural networks into PCMC models, or analysis pointing toward what the features are learning that allows them to improve over earlier approaches. The text was also confusing in a number of places (possibly due to my lack of knowledge in choice modeling), and there’s no discussion of related work incorporating neural networks into ranking-based models.

Comments:
* Knowing nothing about choice modeling, I found the introduction hard to follow with lots of jargon that may be inaccessible to the broader ML community. It may be useful to specify the set of desired properties for these models up front, and then highlight how the different existing models do or don’t satisfy these properties (e.g. uniform expansion, regularity, efficiency,  framing effects, etc.)
* How is the proposed approach “amortized inference”? 
* What’s the triangle notation in P(a |> c) ?
* It’d be useful to say more as to why contractability/uniform expansion are useful components of a choice model.
* “Additive smoothing at the cost of some efficacy” - efficacy in what sense? Expressivity? Or worse performance? The same smoothing technique (minimum of epsilon) seems to be used in this approach.
* Theorem 1 and proof do not consider some of the architectural choices of PCMC-Net (e.g. cartesian product layers, does d_a have to go to infinity?)
* Contractability: why is this property desirable if you don’t take advantage of it computationally?
* Why is SGD + dropout training stable but the original MLE problem not?
* Comparison to baselines using low-rank or other simple parameterizations of Q?
* Table 1 should be moved to supplement
* “Usual rule of thumb” -> cite something here? Not familiar with this
* What is the impact of \epsilon and dropout probability on model performance? From proofs I had expected \epsilon tiny (1e-9) but you use 0.5
* Would be useful to show how model performs on smaller dataset to gain intuition
review_title: Official Blind Review #2

Meta-review: This submission proposes to use neural networks in combination with pairwise choice markov chain models for choice modelling. The deep network is used to parametrize the PCMC and in so doing improve generalization and inference.

Strengths:
The formulation and theoretical justifications are convincing.
The improvements are non-trivial and the approach is novel.

Weaknesses:
The text was not always easy to follow.
The experimental validation is too limited initially. This was addressed during the discussion by adding an additional experiment.

All reviewers recommend acceptance.

Decision: Accept (Poster)