Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a two-step method to determine which floor a mobile phone is on inside a tall building. 
An LSTM RNN classifier analyzes the changes/fading in GPS signals to determine whether a user has entered a building.  Using the entrance point's barometer reading as a reference, the method calculates the relative floor the user has moved to using a well known relationship between heights and barometric readings.

The paper builds on a simple but useful idea and is able to develop it into a basic method for the goal.  The method has minimal dependence on prior knowledge and is thus expected to have wide applicability, and is found to be sufficiently successful on data collected from a real world context.  The authors present some additional explorations on the cases when the method may run into complications.

The paper could use some reorganization.   The ideas are presented often out of order and are repeated in cycles, with some critical details that are needed to understand the method revealed only in the later cycles.   Most importantly, it should be stated upfront that the outdoor-indoor transition is determined using the loss of GPS signals.  Instead, the paper elaborates much on the neural net model but delays until the middle of p.4 to state this critical fact. However once this fact is stated, it is obvious that the neural net model is not the only solution.

The RNN model for Indoor/Outdoor determination is compared to several baseline classifiers.  However these are not the right methods to compare to -- at least, it is not clear how you set up the vector input to these non-auto-regressive classifiers. You need to compare your model to a time series method that includes auto-regressive terms, or other state space methods like Markov models or HMMs.

Other questions:

p.2,  Which channel's RSSI is the one included in the data sample per second?

p.4,  k=3, what is k?

Do you assume that the entrance is always at the lowest floor? What about basements or higher floor entrances?  Also, you may continue to see good GPS signals in elevators that are mounted outside a building, and by the time they fade out,  you can be on any floor reached by those elevators.

How does each choice of your training parameters affect the performance?  e.g. number of epoches, batch size, learning rate.   What are the other architectures considered?  What did you learn about which architecture works and which does not?  Why?

As soon as you start to use clustering to help in floor estimation, you are exploiting prior knowledge about previous visits to the building.  This goes somewhat against the starting assumption and claim.
review_title: A simple but useful method that serves a practical purpose well; improvements needed in writing and experimental comparisons.

final_score: rating: null
review_content: Can you explain why in the table 1 in the revision from 29th October the validation and test accuracy of the LSTM are 0.949 and 0.911 and in the most recent version they have dropped to 0.935 and 0.898 (worse than the baselines)?

Also I agree with the statement by reviewer 2:

"The RNN model for Indoor/Outdoor determination is compared to several baseline classifiers.  However these are not the right methods to compare to -- at least, it is not clear how you set up the vector input to these non-auto-regressive classifiers. You need to compare your model to a time series method that includes auto-regressive terms, or other state space methods like Markov models or HMMs."

It seems like no changes have been made to address this.
review_title: Drop in accuracy (?) and baselines

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors motivate the problem of floor level estimation and tackle it with a RNN. The results are good. The models the authors compare to are well chosen. As the paper foremost provides application (and combination) of existing methods it would be benefitial to know something about the limitations of their approach and about the observed prequesits. 
review_title: The paper combines existing methods to outperform baseline methods on floor level estimation. Limitations of their approach are not explored.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Update: Based on the discussions and the revisions, I have improved my rating. However I still feel like the novelty is somewhat limited, hence the recommendation.

======================

The paper introduces a system to estimate a floor-level via their mobile device's sensor data using an LSTM to determine when a smartphone enters or exits a building, then using the change in barometric pressure from the entrance of the building to indoor location. Overall the methodology is a fairly simple application of existing methods to a problem, and  there remain some methodological issues (see below).

General Comments
- The claim that the bmp280 device is in most smartphones today doesn’t seem to be backed up by the “comScore” reference (a simple ranking of manufacturers).  Please provide the original source for this information.
- Almost all exciting results based on RNNs are achieved with LSTMs, so calling an RNN with LSTM hidden units a new name IOLSTM seems rather strange - this is simply an LSTM.
- There exist models for modelling multiple levels of abstraction, such as the contextual LSTM of [1]. This would be much more satisfying that the two level approach taken here, would likely perform better, would replace the need for the clustering method, and would solve issues such as the user being on the roof.  The only caveat is that it may require an encoding of the building (through a one-hot encoding) to ensure that the relationship between the floor height and barometric pressure is learnt. For unseen buildings a background class could be used, the estimators as used before, or aggregation of the other buildings by turning the whole vector on.
- It’s not clear if a bias of 1 was added to the forget gate of the LSTM or not. This has been shown to improve results [2].
- Overall the whole pipeline feels very ad-hoc, with many hand-tuned parameters.  Notwithstanding the network architecture, here I’m referring to the window for the barometric pressure, the Jaccard distance threshold, the binary mask lengths, and the time window for selecting p0.
- Are there plans to release the data and/or the code for the experiments? Currently the results would be impossible to reproduce.
- The typo of accuracy given by the authors is somewhat worrying, given that the result is repeated several times in the paper.

Typographical Issues
- Page 1:  ”floor-level accuracy” back ticks
- Page  4:   Figure  4.1→Figure  1;  Nawarathne  et  al  Nawarathne  et  al.→Nawarathne et al.
- Page 6:  ”carpet to carpet” back ticks
- Table 2:  What does -4+ mean?
- References.  The references should have capitalisation where appropriate.For example,  Iodetector→IODetector,  wi-fi→Wi-Fi,  apple→Apple, iphone→iPhone, i→I etc.

[1]  Shalini Ghosh, Oriol Vinyals, Brian Strope, Scott Roy, Tom Dean, and LarryHeck. Contextual LSTM (CLSTM) models for large scale NLP tasks. arXivpreprint arXiv:1602.06291, 2016.
[2]  Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever.  An empirical exploration of recurrent network architectures.  InProceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 2342–2350,2015
review_title: A fairly simple application of existing methods to a problem, and there remain some methodological issues

Meta-review: Reviewers agree that the paper is well done and addresses an interesting problem, but uses fairly standard ML techniques.
The authors have responded to rebuttals with careful revisions, and improved results. 
Decision: Accept (Poster)