Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a neural network architecture, called Scaled Neural Multiplicative Model (SNMM). This architecture was designed with two main objectives in mind: first, the model can be fitted properly to reflect actual sales data, even though this data tends to be scarce and sparse. Second, the model is concave, which ensures it can be used by an off-the-shelf optimizer such as cvxpy to find the value of a subset of the input features that maximizes the outcome. Based on this model, the paper defines the problem of finding a shelf space assignment that maximizes sales. The paper evaluates the accuracy of SNMM on 3 public datasets, car sales, puzzle sales, and wine quality, as well as a private datasets of sales for an unspecified retailer.
strength_and_weaknesses: Strengths:
* The paper proposes a model that can be used directly by a convex optimizer, such as cvxpy.

Weaknesses:
* It's not clear that the model actually performs well. On the puzzle dataset, its accuracy was second to last, and on the wine quality dataset the model was the least accurate. That said, the model does well on the car sales dataset and the private dataset.
* Since the paper proposes an end-to-end solution, it would be useful to validate how well the model works with the optimizer to find an optimal assignment of shelf space.
* Other approaches have been proposed to solve similar problems under the umbrella of differentiable optimization. The related work section should mention a few of them, and it would be useful to add one of these to the baselines.
* Two of the datasets used in the evaluation are unrealistically small, with less than 200 datapoints. One of the dataset is not public, which will prevent other researchers from replicating the results and comparing new approaches against SNMM. Is there really nothing better available ?

clarity,_quality,_novelty_and_reproducibility: The paper needs to be edited to focus on its main contribution. Section 3 and 4.2 can be completely eliminated. In particular, it's unclear why linear models are mentioned at all. 

In table 1, 2, and 3, the authors compare their approach to DNN, SCNN, RTL, and Cal Lin. It would be useful to point to the papers that describe these models to enable third parties to reproduce these results.

In table 4, the comparison is made against various versions of GLM, NAM, and NMM. It would be useful to let the reader know where to get details on these versions. It would also be beneficial to explain why the comparison is not made against the same baseline as for the previous experiments.



summary_of_the_review: The contribution of this paper is fairly small: it modifies the neural multiplicative model to ensure its concavity by using the sigmoid trick. Furthermore, the evaluation of the new architecture is limited, and does not convincingly demonstrate that the model works well in practice.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of optimizing shelf space in retail stores (i.e., how many fixtures to assign to each of several brands within a category of products for sale) using DNNs. Finding a tractable optimization formulation for such a problem is non-trivial, but the authors propose the scaled neural multiplicative model (SNMM) with shape constraints that leads to a tractable optimization problem. The authors evaluate this formulation empirically across a variety of synthetic/smaller datasets, then perform a case study on real-world data. Although results for SNMM are inconsistent on smaller/synthetic datasets, it seems to provide a performance benefit on real-world data relative to several baseline techniques.

strength_and_weaknesses: Strength:
* The paper studies an interesting a relevant applied problem. The problem is well explained and familiar to any reader, which (in my opinion) makes the paper interesting/fun to read. The proposed method is explained well and is novel to the best of my knowledge.
* The authors study a novel combination of neural networks and GAMs for effectively imposing shape constrained that are needed to model retail space allocation. 
* The proposed technique performs well on the real-world datasets consistently. 

Weakness:
* SNMM does not provide a consistent performance benefit, though it seems to improve performance in some cases. The validation MSE on the puzzle sales dataset is especially concerning, as it seems to be the worst of the considered methods. A similar result is observed on the wine dataset. It seems like a deeper discussion of why SNMM performs so poorly is needed here, instead of claiming that the error is within the range of other methods. 
* Most of the baselines that outperform SNMM on the smaller datasets (e.g., DNN, RTL, SCNN) are not actually considered in the real-world experiments where SNMM performs the best. 

Small Comments:
* In related work, you use the acronym GAM without stating what this means. 
* The discussion on data sparsity and the difficulty of modeling/recording retail behavior is useful and provides good context for solving the proposed problem. 

Questions:
* What are a few examples of real world applications the authors think this approach could be extended to? It would be nice to maybe give a few examples in the text to give readers an idea. 
* Is (3) seen as a formulation that is easier to explain/interpret in comparison to (2)?
* Do the changes in (9a)/(9b) solve the problem of reduced capacity mentioned at the beginning of the section
* Why are different baselines explored on real data (Sec 5.2) vs. synthetic/small problems (Sec 5.1)? 
clarity,_quality,_novelty_and_reproducibility: Clarity/quality/novelty are good.

By nature, the main experimental results of this work are not reproducible because they use a proprietary dataset. I don't think this is necessarily a reason for rejection, but it is a bit concerning given that the real-world data is where the benefit of the method is observed. 
summary_of_the_review: To begin, I want to emphasize that this review reflects my initial impression of the work, and that I am completely open to further discussion with other reviewers/authors. My final score will mostly be based upon the subsequent discussion.

Currently, my main concern with the work are:
* Benefits are only observed on real-world datasets, while on the smaller/synthetic data the proposed method is actually quite a bit worse.
* No substantial discussion is provided regarding why the method performs so poorly compared to baselines on some datasets.
* Baselines on real-world data (where we see the benefit) are different from the smaller/synthetic experiments. The methods that outperform SNMM actually are not considered in the real-world case.

I think the work could be greatly improved by performing further analysis on the cases where SNMM performs poorly and providing a better understanding of why the real-world case is so different (plus explaining the problem with the selected baselines). Overall, the results on the real-world dataset are quite promising, and I believe this paper can be valuable/useful given further discussion/modifications.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper considers a real-world problem of ​​shelf space allocation for retail stores using DNN models. It focuses on the selection of predictive model families considering the particular convexity constraint for the specific problem formulation. It then proposes a Scaled Neural Multiplicative Model (SNMM) which aims to improve model expressivity under the constraints and is empirically demonstrated on both synthetic and real-world datasets.
strength_and_weaknesses: **Strength**
* The paper considers an optimization problem that is practically relevant. The collected real-world dataset could be a contribution to the community if open-sourced.


**Weakness**
* ***The novelty and the significance of the paper seem quite limited***. In particular, the paper targets a very specific optimization problem (i.e., shelf space allocation) and the proposed SNMM method seems to be a very straightforward generalization of the NMM method (by adding a scaler parameter) which is not convincingly justified with empirical evidence. It is crucial to discuss how the proposed method generalizes beyond the specific problem formulation and how significant it is. See “Novelty & Significance” in the next section for details. 
* ***The presentation of the paper is not very clear or easy to follow***. In particular, the specific optimization problem considered, the key issue that the proposed SNMM aims to address and the significance of SNMM need to be more clearly presented. There are also some unjustified claims and missing mathematical definitions that need to be clarified. See “Clarity & Quality” in the next section for details. 
* ***The experimental evaluation is not reproducible or convincing***. There are a lot of missing experimental details (even missing introduction of baselines) in the paper and no further experimental details are included for reproducibility. See “Reproducibility” in the next section for details.
* ***Potential risk of plagiarism***. In Sec 5.1.1-5.1.3, the starting sentences for introducing the datasets are all **copied** from [1], which may be considered a violation of academic integrity.

[1] Diminishing Returns Shape Constraints for Interpretability and Regularization.  Gupta et al. NeurIPS 2018.

clarity,_quality,_novelty_and_reproducibility: **Novelty & Significance**

The novelty of the paper is not very well presented, and from what it stands now, seems quite limited. In particular, 
1. The formulation targets a very specific optimization problem, which significantly limits the scope of the paper. It is important to justify the generality of the proposed method by either 1) reformulating the problem in a more general way and demonstrating how the method still fits, or 2) incorporating a discussion of how the proposed method generalizes beyond the specific problem formulation.
2. The proposed SNMM simply modifies the NMM by adding a scaler parameter for the input feature. This modification, however, seems to be minor and marginal. Furthermore, there may be many other ways that could be considered, e.g., by utilizing the composition of (monotonically increasing) convex functions, but are not discussed or compared in the paper. It is crucial to justify the significance of the proposed SNMM compared with potential variants.
3. The empirical evaluation of the proposed SNMM is not convincing. In particular, SNMM seems to work poorly on some evaluated setups (e.g., Table 2) and leads to marginal improvements on the real-world dataset (Tables 3, 4, 6), which makes its significance even more questionable. See “Reproducibility” for more details. 


**Clarity & Quality**

The current presentation of the paper is not clear and sometimes confusing. It took me some time to figure out what problem the paper tries to address and how the paper goes toward it. There are also several unjustified claims and mathematical unclarities in the paper. I detail my confusion and questions below:
1. *The problem formulation is not clear*. The paper focuses on the specific problem of ​​shelf space allocation, but the specific description of the paper and detailed problem formulation is not presented beforehand, it would be much clear if the problem is well presented at the start of Section 4 or 3.
2. *The proposed method and the contributions are not clear*.
    * The key issue that the proposed SNMM aims to address, which (to my understanding) is to increase the expressivity of the NMM under the introduced constraint, needs to be very clearly stated out.
    * In the paragraph that introduces SNMM, it is loosely claimed that “Generally, constraining neural network models leads to reduction in their capacity” which is not necessarily true and needs further discussion or empirical evidence. It is also important to discuss how the proposed SNMM breaks through the expressivity bottleneck under the constraint to justify the proposed method. 
    * In Eq. (9b), the purpose for introducing the $\max$ operator is not explained.
    * In Sec 4.4, how the optimization problem is solved is not discussed.
3. *Some statements are not well explained or justified*. Here are a few:
    * In Eq. (8), the two linear layers are composed without any linearity which still falls into the linear family. How does it increase the expressivity as claimed above? 
    * At the start of Sec 3.2, the two reasons for the advantages of multiplicative models over linear models are not sufficiently supported. More explanations or some references are needed.
4. *Missing mathematical definitions*. Here are a few:	
    * In Sec 3.1, the equation for $F_\theta(x)$, both $\phi$ and $\mathbb{P}$ are not defined.
    * In Eq. (3), the notation $\phi_p(w_i^p)$ is confusing to me, what doe $p$ mean as sub-script and super-script respectively?
    * In Eq. (14), the notation $N$ is not defined.


**Reproducibility**

The reproducibility of the experimental results is poor. In particular, in Sec 5.1, the experimental setups (e.g., how the models are trained and how the optimization problem is solved) and even the baselines considered in Table 1 & 2 are not introduced. In Sec 5.2, the dataset used seems not publicly available, but the dataset descriptions are not detailed enough for reproducibility. There are no further experimental details included for reproducing the results in the paper.

summary_of_the_review: In summary, the paper has some crucial flaws: 1) lack of novelty and significance;  2) unclear presentation and lack of soundness 3) reproducibility concerns and potential violation of academic integrity. Thus I recommend a ‘strong reject’ for the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a variant on the neural multiplicative model which can be used for demand modeling, and applies the method to shelf space allocation. The reviewers feel like there might be value in introducing this problem formulation if it were being released as a public benchmark that others could build on. However, the reviewers have a number of significant objections, including: (1) limited novelty, (2) clarity of both the problem formulation and the method, (3) the lack of an apparent improvement (it seems to underperform various baselines), and (4) the lack of reproducibility due to the lack of comparisons on publicly available datasets. I hope that the authors take the reviewers' comments into account in resubmission.
justification_for_why_not_higher_score: See above.
Decision: Reject