Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper introduces the duration-of-stay estimation problem in the warehouse storage application and describes a way to formulate the problem, prepare datasets, design loss functions and train models. The approach seems to significantly reduce the distance traveled in warehouses and therefore reduce labor costs.

This paper is an application paper where the problem studied seems to be limited to a small audience. I am not sure if it aligns well with ICLR. However, I do think the problem is interesting since it can directly lead to significant real-world improvements by improving the machine learning task. It is also nice to see the authors will publish the datasets to enable future research along this direction.

The paper is well written. It introduces the context of the problem very clearly, and outlines the important factors, and how machine learning can help solve the problem. I also like the detailed description of the problem formulation, datasets, and the loss function design.

I think the experiment section needs more work. For example, given that the input features are limited, can a GBDT do the job as well? Is neural network really needed here to learn a complicated representation? By comparing some simpler baselines, we can have a clearer idea into this question. Also, it is better if the paper can include some ablation studies to single out the important factors, for instance, which input features are most important, which kind of modeling enables the potential of which input features.

It is interesting the proposed approach seems to work well in real-world scenario. But it is not clear to me the comparison presented in Figure 3 really leads to increased efficiency of the warehouse. It is true more pallets are placed in nearby locations, but is the prediction of DoS accurate enough to prevent constantly retrieving pallets placed farther away? In other words, Figure 3 only describes the state after adopting the algorithm, but it does not say whether this configuration is better or not since one can come up with a totally random algorithm to place more pallets in nearby locations without leveraging the input features. 


=============Update==============
After considering the rebuttal and the updated text, I have changed my rating to weak accept.

The authors have added more experiment comparisons and also some ablation studies to provide more insights of the problem. Also some clarification to Figure 3 is also added.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper tackles the problem of assigning pallets to storage positions in a warehouse so as to minimize total travel time. Previous literature has shown that if durations of stay (DoS) is known in advance then the optimal storage position is inversely proportional to this. The paper proposes to predict a DoS distribution for each pallet using a 19 point CDF with a neural network. The authors show that the proposed network is better than a few other neural baselines and that it can predict the DoS reasonably well.
The authors report significant positive real life results in two warehouses and releases their dataset.

The paper is interesting and seems to make significant progress on an important real-life issue. The release of a large realistic dataset is a major contribution to this field.

Despite this I will not recommend this paper for publishing at ICLR, simply because I think it falls outside the scope of the conference.

I would recommend the authors seek to publish in a venue more focused on logistics or operational research (OR), where I think the paper could have a great impact.

I would have appreciated a more in-depth description of the problem. It seems that the general problem would be to minimize the expected cost (distance moved) over the lifetime of the warehouse. Under what simplifying assumptions does that reduce to minimizing the expected cost of each pallet individually as done here? 

Similarly, I would have preferred a more formal definition of the loss, e.g. start from the expected distance moved and derive the (approximate) loss from there. Approximations are fine as long as you tell the reader what you're approximating, why you need to do it, and what the biases are.

I would also move the detailed description and figures of the actual neural network to the appendix since it's not central to the understanding of the paper (aside from the CDF formulation).


------------ UPDATE ------------

The area chair didn't get back to me, but I'll assume the paper is in scope and as such have changed my rating to weak reject. 

I'd prefer if the paper spend less space arguing that a specific network architecture is the right choice and prioritize introducing the problem more formally. I would move the simulations of different storage strategies (figure 2, blue and black lines) into the introduction, as part of introducing the reader to the problem and evaluation, and explain how figure 2 is related to what you really want to minimize: "total pallet distance moved over time x". I wonder if the pallet percentile distribution (figure 2) is actually interesting, other than as a way to compute the "total pallet distance moved over time x".

I fear the authors feel the need to introduce a "novel" NN in order to publish at ICLR and thus spend a lot of effort describing something that is really not very interesting (even giving it a name). Combining textual and non-textual features is not a new concept, and combining a RNN layer and a CNN layer is not novel. I'm skeptical that this exact network structure is needed. It's clear you should combine the textual and non-textual features, but is this *exact* network structure needed? I would find that very surprising. How about MLP(concat(CNN(text), RNN(text), non-text))?

Another gripe is the heuristic nature of the loss function. It's clear predicting the DoS distribution better is the right thing to do, but is the sum of L2 distances to the emprical log CDF the *right* measure or a heuristic? What is the right thing to do? What is it approximating if anything? Why?
I would expect the right thing to do is to maximize the probability of the observed duration of stay under the model, i.e max_\theta p(DoS|x, \theta), i.e. min_\theta \sum_i -log(p(DoS_i|x_i, \theta). If this is log-normally distributed, then a log-normal distribution is probably a good distribution. Or perhaps a mixture of log-normals. I think you can even use the 19 point CDF output layer and evaluate the data likelihood under this, but I *don't* think the 19 point L2 distance to the empirical probability distributions is the correct thing to do. Is that even defined if there's less than 19 pallets with the same inputs x?
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper introduces a model to predict the duration of stay of goods in a warehouse, and releases an extensive dataset of warehousing records used in the experiments.

I found the paper very interesting to read, and I believe it could serve as an inspiration to researches more interested in applied ML, since is shows that relatively standard architectures can make a big impact in solving real-world problems.

I have some questions on the experiments:
- The model used in the experiments is a combination of a CNN and a GRU architecture. One thing that is not clear to me is why the author chose to limit the sequential input to the model to only 5 words (in particular in the GRU branch).
- Did you make any analysis on which information is more important for the model when making predictions? Is the sequential information describing the type of product dominating the non-sequential information?
- What are the generalization/transferability properties of the model across different locations? 
- Based on the results on the extended training set it seems clear that due to the temporal shift in the data the model will have to be retrained fairly often. Did you experiment with this in your real-life implementation?

To me, the released dataset is one of the key contributions of this work. However, I have some doubts on the reproducibility of the experiments.
- Due to regulations/NDA you were not able to release some details of the data that seem really important to me. My guess is for example that the brand name is a key information to have when predicting DoS. Is the data really meaningful even without it?
- To verify the above question as well as to create a baseline for people interested in using the public dataset, you should add your results on the public version of dataset (even in appendix).

Overall I liked the paper so I argue for acceptance, provided that I receive a satisfactory answer to the above concerns. 

Small typo:
- Assumption 1: ".. exists a function f" -> ".. exists a function g"

review_title: Official Blind Review #1

Meta-review: Thanks to the authors for the submission and the active discussion. The paper applies deep learning to the duration-of-stay estimation problem in the warehouse storage application. The authors provide problem formulation and describe the pipeline of their solutions, including datasets preparation and loss functions design. The reviewers agree that this is a good application paper that showcases how deep learning can be useful for a real-world problem. The release of the dataset can also be a nice contribution. A major debate during the discussion is whether this paper is in scope of ICLR given that it is mostly a straightforward application existing techniques. After several rounds of discussion, reviewers think that this should fit under the category "applications in vision, ... , computational biology, and others." Overall, this paper can be a good example of applying deep learning to real-world problems.

Decision: Accept (Spotlight)