Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: In this paper the authors propose a novel architecture, called Mass-Conserving LSTM (MC-LSTM) based on LSTM. The authors base their work over the hypothesis that the real world is based over conservation laws related to mass, energy, etc. Thus, they propose that also the quantities involved in deep learning models should be conserved. To do so, they aim at exploiting the memory cells of the LSTM as mass accumulators and then force the conservation laws via the model equations. The authors finally show successfully the potential of this novel network into three experimental settings where several types of “conservation” are required (e.g. mass conservation, energy conservation, etc).

Pros:
+ they deal with a problem which can arise in non-laboratory scenarios in a novel way. Moreover, their results show that their method is great at dealing with situations where conservation is required, which may be particularly important for real-world scenarios
+ the paper explains in-depth all the decisions made, and it is well written. Moreover, I found really interesting how they deal with the related work and special cases. It shows a really in-depth understanding of up-to-date literature in the field. 

Cons:
- (minor) the authors focus their experimental section to settings where mass (or energy, etc) conservation is required. It would be interesting to see how it performs also in settings where it is not required as well, thus showing whether this method also generalizes to different settings. 

------

Authors' response addressed properly my request.

review_title: The experimental evaluation shows that the proposed method is great at dealing with situations where conservation is required, which may be particularly important for real-world scenarios

final_score: rating: 7: Good paper, accept
review_content: The paper provides an interesting and novel LSTM structure named MC-LSTM, which extends the inductive bias of LSTM to deal with some real-world problems limited by conservation laws. The authors do some experiments related to traffic forecasting and hydrology to illustrate the effectiveness of MC-LSTM.  The new architecture is well-suited for predicting some physical systems, which is valuable.
##########################################################################

Reasons for score: 

Overall, I vote for accepting. 
I deem that the novel architecture based on LSTM that conserves quantites is useful and interesting.
My major concern is about some explanation about definitions and some additional ablation models (see cons below). 
Hopefully the authors can address my concern in the rebuttal period. 

##########################################################################Pros: 

1. The paper takes one of the most important issue of some real-world systems:  conservation laws, which is important and
should be expressed through LSTM units.

2.  This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed LSTM. 
The entire structure is organized well and the formulas are very detailed.

##########################################################################

Cons: 

1. In Basic gating, the formula about input, why do you use softmax operator? Because in basic LSTM, there is sigmoid.
 It would be better to provide more details about it.

2. What are the advantages of MC-LSTM in terms of speed and resource consumption?
It would be more convincing if the authors can provide more cases in the rebuttal period. 

##########################################################################

Questions during rebuttal period: 

Please address and clarify the cons above 
 
review_title: Recommendation to Accept

final_score: rating: 6: Marginally above acceptance threshold
review_content: <Summary>
1. Many real-world systems follow conservation laws.
2. This paper incorporates conservation laws into the RNN as an inductive bias.
3. This paper proposes MC-LSTM and shows that MC-LSTM follows conservation laws empirically and theoretically.
4. MC-LSTM utilizes a positive left-stochastic matrix to redistribute mass.
5. This paper validates the MC-LSTM on arithmetic tasks, traffic forecasting, pendulum, rainfall tasks.

<Strengths>
1. Incorporating mass conservation laws into the neural network is important.
2. This paper proposes MC-LSTM, a simple and effective mass-conserving model.
3. The motivation of the research and the proposed methods are straightforward.
4. Related works section provides a comparison between MC-LSTM and others such as the Markov chain, and it is also very interesting.

<Weaknesses>
1. This paper shows extensive experimental results, but there are less qualitative results. 
2. This paper missing some important related works, such as a physics-guided recurrent neural network model (PGRNN)

<Questions and Additional Feedback>
1. Is there empirical (or theoretical) analysis for long-term gradient vanishing?
2. I wonder how MC-LSTM actually works. Is there qualitative analysis for a, i, o, and R?
3. What is the reason that Eq. (5) and Eq. (6) utilize L1 norm?
4. Does MC-LSTM can be extended to represent (2*mass) or (mass^2)-conserving properties?
5. MC-Transformer will be one of a good extension.

<Missing Reference>
1. Lagrangian Neural Networks
2. Physics-Guided Machine Learning for Scientific Discovery: An Application in Simulating Lake Temperature Profiles
3. Discovering physical concepts with neural networks

<Typos>
1. One of the greatest success stories of deep learning are => One of the greatest success stories of deep learning is
2. The former represent => The former represents

<After Rebuttal>
Thank you for your detailed response.
I will keep my positive score because my concerns are resolved partially.
review_title: MC-LSTM is an interesting work.

final_score: rating: 7: Good paper, accept
review_content: #### Summary

The authors proposed a new family of LSTMs (i.e. MC-LSTMS) which can be shown to have a mass conservation property for the LSTM memory cells. They have shown in various applications that these models lead to on par or better performance compared to state of art approaches. However, after examining the paper, I am not yet fully convinced that 1) a general unified MC-LSTMs can achieve good performance in all the mentioned cases 2) the conserved mass is interpretable (and corresponds to the problem’s invariant, conserved quantity). I have listed those concerns in the question sections as authors might have precise answers. 

#### Pros

The authors in this paper have proposed a new family of LSTMs (i.e. MC-LSTMs) which can be shown to have a conservation property for the mass held by the LSTM memory cells. The authors have applied their models in three different prediction tasks involving quantity conservation, namingly summation(and subtraction), traffic prediction and rainfall runoff modelling. The proposed models (with some variants) achieve better or at least on par with current state of art models.

These different applications illustrate the usefulness of the models; the authors also reference appropriately to current state of art approaches, helping reviewers to well situate the paper’s contribution.

#### Questions:

Well I understand there are differences for each application (motivating to choose different neural architectures), it would be very interesting to know the performance of one quite general architecture (compared to say standard LSTM implemented in torch). In the paper, we have seen in the experiments:
- time - independent R^t used in arithmetics
- time - dependent R^t used in hydrology experiments
- Hypernetwork based R^t used in pendulum experiments
And also they vary on the auxiliary/mass inputs choices. 
What would be the performance if the authors use a quite general architecture (for example the second setting) for all the experiments please?

Just for confirmation, in the experiments, the LSTMs involve forgetting gate right (contrary to what described in eq (1))

I don’t find explanations for the r value for the hydrology experiments. Meanwhile the r value for the traditional r value is different from what is chosen for the MC-LSTM. Why is this please?

#### Minor issues:

There seems some formatting issues at the end of “Mechanisms beyond storing are required for real-world applications” in Introduction. 

In related work, Hamiltonian approaches (Greydanus et al. 2019) don’t seem to assume access knowledge to time derivatives w.r.t. Inputs. Rather, it favours the conservation parametrised by the Hamiltonian.  

#### Minor suggestions:

In abstract, “expressed through continuity equations”, these aspects don’t seem to be addressed by MC-LSTMs, so I propose to not include this phrase in the abstract. 

I suggest adding some citations In the introduction where the authors said “LSTM to excel at speech, text, and language tasks” http://nlpprogress.com/. 

The authors mention in the abstract and introduction about the interpretability but only show the mass interpretation in the hydrology experiments. For which I would suggest the authors show it in at least one other experiments (e.g. traffic) to 1) be more convincing 2) highlight this property (which may reveal to be very beneficial for production systems)

I would be good to mention forgetting gate around equation (1) as those are used for later experiments in the paper. 

#### Thank you for the authors to having answered my questions and having addressed all my comments. My main concern was about the generalisation of the proposed architecture and the results at the current revision are convincing to me. I believe this direction of the research together with the approach taken make a nice contribution to the conference. In consequence, I have raised my score from 5 to 7.  
review_title: Nice idea, extensive experiments with clear references; having concerns about the architecture generalisation.

Meta-review: The paper proposes a variant of recurrent neural networks based on Long Short-Term Memory. Unlike the standard LSTM, the proposed mass-conserving LSTM subtracts the output hidden state of the LSTM from its current cell state, thus preserving the "mass" stored in the cell states at each step. A left-stochastic recurrent weight matrix is also used to conserve the "mass" across the time steps. Empirical experiments demonstrated the effectiveness of the proposed MC-LSTM on a range of datasets such as addition & arithmetic tasks, traffic forecast, and rainfall modeling models. 

Several issues were clarified during the rebuttal period in a way that satisfied the reviewers. However, some concerns still remain unanswered:

1) This is an empirical paper that proposes a modified LSTM that brings forward a few different ideas: L1 norm, stochastic transition matrices, and subtracting the output hidden states. An ablation study is a MUST in such an applied work. It has been pointed out by other reviewers that there are many prior references on LSTMs variants. It would greatly strengthen the paper by considering more diverse baselines. There is no experiment nor discussion on how much each modification helps wrt the final accuracy. Thus it remains unclear how the results can generalize to other problems.

2) Although the results seem convincing across various datasets that mass conservation seems to help, the datasets are non-standard benchmarks in the machine learning conferences thus there is a lack of competitive prior baselines. As the proposed LSTM has a different number of parameters compared to the standard LSTM, is it fair to compare the different architectures under the same number of neurons? What happens if we compare the architectures with the same number of parameters? And how well does the model scale as we vary the hidden size? It would be helpful to keep the contributions into perspective by using standard RNN benchmark datasets such as Penn TreeBank or Wiki-8.

Overall, the basic idea seems interesting, but the lack of ablation studies significantly hurt the contribution and the positioning of the paper. Given the current submission, the paper needs further development, and non-trivial modifications, to be broadly appreciated by the machine learning community.


Decision: Reject