MC-LSTM: Mass-conserving LSTM
Authors: Pieter-Jan Hoedt
Frederik Kratzert
Daniel Klotz
Christina Halmich
Markus Holzleitner
Grey Nearing
Sepp Hochreiter
Günter Klambauer
Venue:  (2021)
Keywords: LSTM
RNN
mass-conservation
neural arithmetic units
inductive bias
hydrology

Abstract: The success of Convolutional Neural Networks (CNNs) in computer vision is mainly driven by their strong inductive bias,  which is strong enough to allow CNNs to solve vision-related tasks with random weights, meaning without learning. Similarly, Long Short-Term Memory (LSTM) has a strong inductive bias towards storing information over time. However, many real-world systems are governed by conservation laws, which lead to the redistribution of particular quantities —e.g. in physical and economical systems.   Our novel Mass-Conserving LSTM (MC-LSTM) adheres to these conservation laws by extending the inductive bias of LSTM to model the redistribution of those stored quantities. MC-LSTMs set a new state-of-the-art for neural arithmetic units at learning arithmetic operations, such as addition tasks, which have a strong conservation law, as the sum is constant overtime. Further, MC-LSTM is applied to traffic forecasting, modeling a pendulum, and a large benchmark dataset in hydrology, where it sets a new state-of-the-art for predicting peak flows. In the hydrology example, we show that MC-LSTM states correlate with real world processes and are therefore interpretable.

Under review as a conference paper at ICLR 2021
MC-LSTM: MASS-CONSERVING LSTM
Anonymous authors
Paper under double-blind review
ABSTRACT
The success of Convolutional Neural Networks (CNNs) in computer vision is mainly driven by their strong inductive bias, which is strong enough to allow CNNs to solve vision-related tasks with random weights, meaning without learning- Similarly, Long Short-Teri Memory (LSTM) has strong inductive bias towards storing information over time. However, many real-world systems are governed by conservation laws, which lead to the redistribution of particular quantities e.g. in physical and economical systems. Our novel Mass-Conserving LSTM (MC-LSTM) adheres to these conservation laws by extending the inductive bias of LSTM to model the redistribution of those stored quantities. MC-LSTMs set new state-of-the-art for neural arithmetic units learning arithmetic operations, such addition tasks, which have a strong conservation law, as the sum is constant over time. Further, MC-LSTM is applied to traffic forecasting, modeling a pendulum, and a large benchmark dataset in hydrology, where it sets a new state-of-the-ar for predicting peak flows. In the hydrology example, we show that MC-LSTM states correlate with real world processes and are therefore interpretable]
INTRODUCTION
Inductive biases enabled the success of CNNs and LSTMs. One of the greatest success stories of deep learning is Convolutional Neural Networks (CNNs) (Fukushima] 1980, LeCun & Bengio 1998) Scchidhhuerr 20155 LeCun et al. 2015) whose proficiency can be attributed to their strong inductive bias towards visual tasks (Cohen Shashua 2017 Gaier & Hap 2019). The effect of this inductive bias has been demonstrated by CNNs that solve vision-related tasks with random weights, meaning without learning (He et al., 2016 Gaier & Hal 2019 Ulyanov et al. 2020). Another success story is Long Short-Term Memory (LSTM) (Hochreiter) 1991 Hochreiter & Schmidhuber 1997). which has a strong inductive bias toward storing information through its memory cells. This inductive bias allows LSTM excel at speech, text, and language tasks (Sutskever al., 2014 Bohnntt et al.) 2018 Kochkina et al., 2017]] Liu & Guo 2019), as well as timeseries prediction. Even with random weights and only a learned linear output layer LSTM is better at predicting timeseries than reservoir methods (Schmidhuber et al., 2007). In a seminal paper on biases in machine learning, Mitchell (1980) stated that "biases and initial knowledge are at the heart of the ability to generalize beyond observed data", ". Therefore, choosing an appropriate architecture and inductive bias for deep neural networks is key to generalization.
Mechanisms beyond storing are required for real-world applications. While LSTM can store information over time, real-world applications require mechanisms that go beyond storing. Many real-world systems are governed by conservation laws related to mass, energy, momentum, charge, or particle counts, which are often expressed through continuity equations. In physical systems, different types of energies, mass or particles have to be conserved (Evans & Hanney| 2005 Rabitz et al.) 1999) van der Schaft et al. 1996), in hydrology it is the amount of water (Freeze & Harlan 1969; Beven 201 1b, in traffic and transportation the number of vehicles (Vanajakshi & Rilent 2004; Xiao & Duan 2020: Zhao et al. (017). and in logistics the amount of goods, money products. A real-world task could be to predict outgoing goods from a warchouse based on a general state of the warchouse, ie., how many goods are in storage, and incoming supplies. If the predictions are not precise, then they do not lead to an optimal control of the production process. For modeling such systems, certain inputs must be conserved but also redistributed across storage locations within the system. We will
All code reproduce the results will be made available GitHuh.

---

Under review as a conference paper at ICLR 2021
refer conserved inputs mass, but note that this can be any type of conserved quantity. We argue that for modeling such systems, specialized mechanisms should be used to represent locations & whereabouts, objects, storage & placing locations and thus enable conservation.
Conservation laws should pervade machine learning models in the physical world. Since a large part of machine learning models are developed to be deployed in the real world, in which conservation laws are omnipresent rather than the exception, these models should adhere to them automatically and benefit from them. However, standard deep learning approaches struggle at conserving quantities across layers or timesteps (Beucler et al. [2019b] Greydanus et al., 2019)) Song Hopke T996. Yitian& Gu 2003), and often solve a task by exploiting spurious correlations (Szegedy et al.) 2014 Lapuschkin et al. 2019). Thus, an inductive bias of deep learning approaches Via mass conservation over time an open system, where mass can be added and removed, could lead to a higher generalization performance than standard deep learning for the above-mentioned tasks.
A mass-conserving LSTM. In this work. we introduce Mass Mass-Conserving LSTM (MC-LSTM), a variant of LSTM that enforces mass conservation by design. MC-LSTM is recurrent neural network with an architecture inspired by the gating mechanism in LSTMs. MC-LSTM has strong inductive bias to guarantee the conservation of mass. This conservation is implemented by means of left-stochastic matrices, which ensure the sum of the memory cells in the network represents the current mass in the system. These left-stochastic matrices also enforce the mass to be conserved through time. The MC-LSTM gates operate as control units on mass flux. Inputs are divided into a subset of mass inputs, which are propagated through time and are conserved, and subset of auxiliary inputs, which serve as inputs to the gates for controlling mass fluxes. We demonstrate that MC-LSTMs excel at tasks where conservation of mass is required and that it is highly apt at solving real-world problems in the physical domain.
Contributions. We propose a novel neural network architecture based LSTM that conserves quantities, such as mass, energy, or count, of a specified set of inputs. We show properties of this novel architecture, called MC-LSTM, and demonstrate that these properties render it powerful neural arithmetic unit. Further, we show its applicability in real-world areas of traffic forecasting and modeling the pendulum. In hydrology, large-scale benchmark experiments reveal that MC-LSTM has powerful predictive quality and can supply interpretable representations.
2 MASS-CONSERVING LSTM
The original LSTM introduced memory cells to Recurrent Neural Networks (RNNs), which alleviate the vanishing gradient problem (Hochreiter 1991). This is achieved by means of fixed recurrent self-connection of the memory cells. If we denote the values in the memory cells at time t by c', this recurrence can be formulated as
c' c'=c'-++ f(x', h'')),
where æ and h are, respectively, the forward inputs and recurrent inputs, and ʃ is some function that computes the increment for the memory cells. Here, we used the original formulation of LSTM without forget gate (Hochreiter & Schmidhuber 1997), but in all experiments we also consider LSTM with forget gate (Gers al. 2000)..
MC-LSTMs modify this recurrence to guarantee the conservation of the mass input. The key idea is use the memory cells from LSTMs as mass accumulators. or mass storage. The conservation law is implemented by three architectural changes. First, the increment, computed by f in Eq. [ has to a
distribute mass from inputs into accumulators. Second, the mass that leaves MC-LSTM must also disappear from the accumulators Third, mass has to be redistributed between mass accumulators. These changes mean that all gates explicitly represent mass fluxes.
Since, in general, not all inputs must be conserved, we distinguish between mass inputs, æ, and auxiliary inputs, a. The former represents the quantity to be conserved and will the mass accumulators in MC-LSTM. The auxiliary inputs used to control the gates. To keep the notation uncluttered, and without loss of generality, we use a single mass input at each timestep, z', to introduce the architecture.
2

---

Under review as a conference paper at ICLR 2021
The forward pass of MC-LSTM timestep can be specified as follows:
mist
miot R² -c'cc-l +
(2)
R
c' = (1 o') O m
(3)
h' o' o' o mioe
(4)
where i' and o' are the imput- and output gates, respectively, and R is positive left-stochastic matrix, i.e., 1d = 1, for redistributing mass in the accumulators. The total mass moot is the redistributed mass, R' c²-1, plus the mass influx, new mass, i' x². The current mass in the system is stored in cf.
a.'
Figure I: Schematic representation of the main operations in the MC-LSTM architecture (adapted from: Olah 2015).
Note the differences between Eq. and Eq. First, the increment of the memory cells longer depends on h Instead, mass inputs are distributed by means of the normalized i (see Eq. 5) Furthermore, R' replaces the implicit identity matrix of LSTM redistribute mass among memory cells. Finally, Eq. (3) introduces
of as a forget gate on the total mass, Mynt- Together with Eq. (4) this assures that no outgoing mass stored in the accumulators. This formulation has some similarity to Gated Recurrent Units (GRU) (Cho et al.) (2044), however the gates are not used for mixing the old and new cell state, but for splitting off the output.
Basic gating and redistribution. The MC-LSTM gates at timestep l are computed as follows:
d²1
softmax Wi a' + Uj
||[-|]] + bi)
(5)
c
o oWW o UU. |c- b.)
(6)
R softmax( Br):
where the softmax operator is applied column-wise, is the logistic sigmoid function, and W. b, Wo. ba and B, are learnable model parameters. Note that for the input gate and redistribution matrix, the requirement is that they column normalized. This can also be achieved by other means than using the softmax function. For example, an alternative way to ensure a column column-normalizee normalized matrix R' is use normalized logistic, ē (rkky) = Σ., J. Also note that MC-LSTMs compute the gates from the memory cells, directly. This is in contrast with the original LSTM, which uses the activations from the previous time step. The accumulated values from the memory cells, c', are normalized to counter saturation of the sigmoids and to supply probability vectors that represent the current distribution of the mass across cell states We use this variation e.g. in our experiments with neural arithmetics (see Sec. 5.1)
a(re))
Time-dependent redistribution. It can also be usefuul to predict redistribution matrix for each sample and timestep, similar to how the gates are computed:
Rd = softmax W. U.
11 + B,
(8)
||
where the parameters W, and U, are weight tensors and their multiplications result in K K matrices. Again, the softmax function is applied column- This version collapses to a time-independent redistribution matrix if W, and U, are equal to 0. Thus, there exists the option to initialize W, and U, with weights that are small in absolute value compared to the weights of B,, favour learning iime-independent redistribution matrices. We use this variant in the hydrology experiments (see Sec. 5.4)
Redistribution via a hypernetwork. Even more general, a hypernetwork (Schmidhuber 1992 Ha et al. 2017) that we denote with be used to procure R. The hypernetwork has produce
3

---

Under review as a conference paper at ICLR 2021
d'aa.....'''e.. ......,,c"") ....,"'-).. Notably, hypernetwork column- normalized, square matrix R' g(a". ..... d'.
can be used to design an autoregressive version of MC-LSTMs, if the network additionally predicts auxiliary inputs for the next time step. We use this variant in the pendulum experiments (see Sec. 5.3
3 PROPERTIES
Conservation. MC-LSTM guarantees that mass is conserved over time. This is a direct conse- quence of connecting memory cells with stochastic matrices. The mass conservation ensures that no mass can be removed or added implicitly, which makes it easier to learn functions that generalize well. The exact meaning of this mass conservation formalized in Theorem T
Theorem (Conservation property). Let m = = C be the mass contained in the system and mi =kk= hi be the mass efflux, or, respectively, the accumulated mass in the MC-LSTM storage and the outputs at time T. At any timestep T, we have:
m m
mi.
t=1
That is, the change of mass in the memory cells is the difference between the input and oulput mass, accumulated over time.
The proof by induction over (see Appendix C). Note that is still possible for input mass to be stored indefinitely in a memory cell so that it does not appear at the output. This can be a useful feature if not all of the input mass is needed at the output. In this case, the network can learn that one cell should operate as a collector for excess mass in the system.
Boundedness of cell states. In each timestep T, the memory cells, ct. are bounded by the sum of mass inputs Ei= x² + +m that is lcol < >i= x' + m. Furthermore, if the series of mass inputs converges, lim, Ex=r" x = mm then also the sum of cell states converges (see Appendix,
Corollary
Initialization and gradient flow. MC-LSTM with R² has a similar gradient flow to LSTM with forget gate (Gers et al. 2000). Thus, the main difference in the gradient flow is determined by the redistribution matrix R. The forward pass of MC-LSTM without gates c' RR --1 leads
to the following backward expression Bo R'. Hence, MC-LSTM should be initialized with redistribution matrix close to the identity matrix to ensure stable gradient flow as in LSSTs. For random redistribution matrices, the circular law theorem for random Markov matrices (Bordenave) ct al. 2012) can be used to analyze the gradient flow in more detail, see Appendix, Section D
Se
Computational complexity. Whereas the gates in traditional LSTM are vectors, the input gate and redistribution matrix of an MC-LSTM are matrices in the most general case. This means that MC-LSTM is, in general, computationally more demanding than LSTM. Concretely, the forward pass for a single timestep MC-LSTM requires O(K3 K2(M + L) + K KML) Multiply-Accumun operations (MACs), whereas LSTM takes O(K² + K(M + L)) MACs per timestep. Here, M, L and are the number of mass inputs, auxiliary inputs and outputs, respectively. When using a time- independent redistribution matrix cf. Eq. 77) the complexity reduces to O(K² M + KML) MACs.
Potential interpretability through inductive bias and accessible mass in cell states. The repre- sentations within the model can be interpreted directly as accumulated mass. If one mass energy quantity is known, the MC-LSTM architecture would allow to force particular cell state to represent this quantity, which could facilitate learning and interpretability. An illustrative example is the of rainfall runoff modelling, where observations, say the soil moisture or groundwater-state, could be used to guide the learning of explicit memory cell of MC-LSTMM.
4

---

Under review as a conference paper at ICLR 2021
4 SPECIAL CASES AND RELATED WORK
Relation to Markov chains. In a special case MC-LSTM collapses to a finite Markov chain, when e" is a probability vector, the mass input is zero x 0 for all t, there is no input and output gate, and the redistribution matrix is constant over time R2 R. For finite Markov chains, the dynamics are known to converge, if is irreducible (see e.g. Hairer (2018) Theorem 3. 3.13.)). Awiszus Rosenhahn (2018) aim to model a Markov Chain by having a feed- forward network predict the state distribution given the current state distribution. In order to insert randomness to the network, a random seed is appended to the input, which allows to simulate Markov processes. Although MC-LSTMS are closely related to Markov chains, they do not explicitly learn the transition matrix, as is the case for Markov chain neural networks. MC-LSTMs would have to learn the transition matrix implicitly.
Relation to normalizing flows and volume-conserving neural networks. In contrast to normal- izing flows (Rezende & Mohamed| 2015 Papamakarios et al.) 2019), which transform inputs in each layer and trace their density through layers or timesteps. MC-LSTMs transform distributions and do not aim to trace individual inputs through timesteps. Normalizing flows thereby conserve information about the input in the first layer and can use the inverted mapping to trace an input back to the initial space. MC-LSTMs are concerned with modeling the changes of the initial distribution over time and can guarantee that multinomial distribution is mapped to multinomial distribution. For MC-LSTMs without gates, the sequence of cell states c", ,..,,, c, constitutes a normalizing flow an initial distribution (ce) ) is available. In more detail, MC-LSTM can be considered a linear flow with the mapping c²+1 R' c¹ and p(c'+t)= =ar=))=p(ee) p(c')l det R'I-1 in this case. The gate providing the redistribution matrix (see Eq. 8) the conditioner in normalizing flow model. From the perspective flow trained in supervised fashion. Deco & Brauer (1955) proposed volume-conserving neural networks, which conserve the volume spanned by imput vectors and thus the information of the starting point of input is kept. In other words, they are constructed so that the Jacobians of the mapping from one layer to the next have a determinant of 1. In contrast, the MC-LSTMs determinant of the Jacobians (of the mapping) is smaller than 1 (except for degenerate cases), which means that volume of the inputs is not conserved.
of normalizing flows, MC-LSTM can be considered
Relation to Layer-wise Relevance Propagation. Layer-wise Relevance Propagation (LRP) (Bach et al. 2015) is similar to our approach with respect to the idea that the sum of a quantity, the relevance Q' is conserved over layers 1. LRP aims to maintain the sum of the relevance values E/=1 Q E/= Q backward through classifier in order to obtain relevance values for each input feature.
Relation to other networks that conserve particular properties. While a standard feed-forward neural network does not give guarantees aside from the conservation of the proximity of datapoints through the continuity property. The conservation of the first moments of the data distribution in the form of normalization techniques (loffe & Szegedy (2013) has had tremendous success. Here, batch normalization (loffe Szegedy| 2015) could exactly conserve mean and variance across layers, whereas self eeff-normalization (Klambauer et al. 2017) conserves those approximately. The conservation of the spectrul norm of each layer the forward pass has enabled the stable training of generative adversarial networks (Miyato et al. 2018). The conservation of the spectral norm of the errors through the backward pass of an RNN has enabled the avoidance of the vanishing gradient problem (Hochreiter) 1991 Hochreiter & Schmidhuber) 1997). In this work, explore an architecture that exactly conserves the mass of a subset of the input, where mass is defined as physical quantity such as mass energy.
Relation to neural networks for physical systems. Neural networks have been shown to discover physical concepts such as the conservation of energies (Iten et al., 2020), and neural networks could allow to learn natural laws from observations (Schmidt & Lipson 2009 Cranmer et al. (020b). MC-LSTM can be seen as a neural network architecture with physical constraints (Karpatne et al.)
Beucler et al. 2019c). It is however also possible to impose conservation laws by using other means, e.g. initialization constrained optimization or soft constraints (as, for example, proposed by Karpatne et al.] 2017: Beucler et al. [209cc at Via et al.) 2019). Hamiltonian neural networks (Grey' danus et al.) (019) and Symplectic Recurrent Neural Networks make energy conserving predictions by using the Hamiltonian (Chen et al. 2019). function that maps the inputs to the quantity that
2017
5

---

Under review as a conference paper at ICLR 2021
Table 1: Performance of different models on the LSTM addition task in terms of the MSE. MC-LSTM significantly (all p-values below .05) outperforms its competitors, LSTM (with high initial forget gate bias), NALU and NAU. Error bars represent 95%-confidence intervals across 100 runs.
reference
seq length b
input range
count
combo
Nan
MC-LSTM
0.004 + 0.003
0.009 = 0.004
0.8 + 0.5
0.6 - 0,4
4.0 + 2.5
LSTM
0.008 + 0.003
0.727 = 0.169
21.4
9.5 0.6
54.6 + 1.0
0
NALU
0.060 = 0.008
0.059 = 0.009
2
7.1
63.7 + 0.6
93
NAU
0.248 = 0.019
0.252 = 0.020
28.3 = 0.5
9.1 = 0.2
68.5
+ 0.8
24
a training regime:
summing 2 out of 100 numbers between and 0.5.
b longer sequence lengths:
summing 2 out of 000 numbers between 0 and 0.5.
e more mass the input:
summing 2 out of 100 numbers between and 5.0.
d higher number of summands:
summing 20 out of 100 numbers between 0 and 0.5.
combination of previous scenarios:
summing 10 out of 500 numbers between 0 and 2.5.
Number of runs that did converge.
needs to be conserved. By using the symplectic gradients, it possible to move around in the input space, without changing the output of the Hamiltonian. Lagrangian Neural Networks (Cranmer et al., 2020a), extend the Hamiltonian concept by making it possible to use arbitrary coordinates as inputs.
All of these approaches, while very promising, assume closed physical systems and are thus to restrictive for the application we have mind. RRaissi et al.] (20T9) propose to enforce physical constraints simple feed-forward networks by computing the partial derivatives with respect to the inputs and computing the partial differential equations explicitly with the resulting terms. This approach, while promising, does require exact knowledge of the governing equations. By contrast, our approach is able to learn its own representation of the underlying process, while obeying the pre-specified conservation properties.
EXPERIMENTS
In the following, we discuss the experiments we conducted demonstrate the broad applicability and high predictive performance of MC-LSTM in settings where mass conservation is required. For more details on the datasets and hyperparameter selection for each experiment, we refer to Appendix B
5.1 ARITHMETIC TASKS
Addition problem. We first considered problem for which exact mass conservation is required. One example for such a problem has been described in the original LSTM paper (Hochreiter & Schmidhuber] 1997), showing that LSTM is capable of summing two arbitrarily marked elements in sequence of random numbers. We show that MC-LSTM is able to solve this task, but also generalizes better to longer sequences, input values in different range and more summands. the results of this method comparison and shows that MC-LSTM significantly outperformed the other models all tests (p-value < 0.03, Wilcoxon test). In Appendix B.1.5. provide a qualitative analysis of the learned model behavior for this task.
Recurrent arithmetic. Following Madsen & Johansen (200),, the inputs for this task are sequences of vectors, uniformly drawn from [1, 2 For each vector in the sequence, the sum over two random subsets is calculated. Those values are then summed over time, leading to two values. The target output obtained by applying the arithmetic operation to these two values. The auxiliary input for MC-LSTM is sequence of ones, where the last element is - to signal the end of the sequence.
We evaluated MC-LSTM against NAUS and Neural Accumulators (NACs) directly in the framework Madsen & Johansen (022). NACs and NAUs use the architecture as presented in (Madsen & Johansen 2020). That is, a single hidden layer with two neurons, where the first layer recurrent. The MC-LSTM model has two layers, of which the second one is fully connected linear layer. For subtraction an extra cell was necessary to properly discard redundant input mass.

---

Under review as a conference paper at ICLR 2021
Recurrent arithmetic task results. MC-LSTMs for addition and subtraction/multipiccttioo Table 2:
have two and three neurons, respectively. Error bars represent 95% 25%-confee intervals.
addition
subtraction
multiplication
success rate
updates
success rate"
updates
success rate
updates
MC-LSTM
96% +2%
4.6 105
81%
-6%
1.2 10
67%
+8%
1.8 10
-6%
-9%
-10%
NAU NMU
88% +5%
8.1 104
60%
+9%
6.1 10*
34% +10%
8.5 104
-8%
10%
NAC
56%
3.2 10
86%
4.5- 104
0% +4%
10%
8%
00
NALU
10% +7%
1.0 106
0% +49%
1% + +1%
4.3 . 109
-0%
a Percentage of runs that generalized to longer sequences. Median number of updates necessary to solve the task.
Succoss
For testing, we took the model with the lowest val- idation error, c.f. early stopping. The performance was measured by the percentage of runs that suc- cessfully generalized to longer sequences. General- ization is considered successful if the error is lower than the numerical imprecision of the exact opera- tion (Madsen & Johansen 2020). The summary in Tab. shows that MC-LSTM was able to significantly outperform the competing models (p-value 0.03 for addition and 3 for multiplication, proportion test).
MC-LSTM
NAU
10
20g
400
600
B00
1000
Sequence length
In Appendix [.1.3]] we provide qualitative analysis of the learned model behavior for this task.
Figure MNIST arithmetic task results for MC-LSTM and NAU. The task is to correctly predict the sum of a sequence of presented MNIST digits. The success rates are depicted on the y-axis in dependency of the length of the sequence (x-axis) of MNIST digits. Error bars represent 95 95%--confidenc intervals.
Static arithmetic. To enable a direct comparison with the results reported in Madsen & Johansen (2020), we also compared MC-LSTM on the static arithmetic task, see Appendix B.1.3
MNIST arithmetic. We tested that feature extractors can be learned from MNIST images (LeCun al. 1998) to perform arithmetic on the images (Madsen & Johansen 2020). The input is sequence of MNIST images and the target output is the corresponding sum of the labels. Auxiliary inputs are all 1, except the last entry, which 11, to indicate the end of the sequence. The models are the same as in the recurrent arithmetic task with CNN to convert the images to (mass) inputs for these networks. The network is learned end-to-end. Lg-regularization added to the output of CNN to prevent its outputs from growing arbitrarily large. The results for this experiment are depicted in Fig. 2 MC-LSTM significantly outperforms the sate-of-theeart NAU (p-value 0.002, Binomial test).
5.2 INBOUND-OUTBOUND TRAFFIC FORECASTING
We examined the usage of MC-LSTMs for traffic fore- casting in situations in which inbound and outbound traffic counts of a city are available (see Fig. 3). For this type of data,
conservation-o--veliee
principle (Nam)
Drew (996) must hold, since vehicles can only leave the city if they have entered it before or had been there in the first place. Based on data for the traffic4cast 2020 challenge (Kreil et al. 2020). we constructed dataset to model inbound and outhound traffic in three differ- cities: Berlin, Istanbul and Moscow. We compared MC-LSTM against LSTM, which is the state-of-the-ar method for several types of traffic forecasting situations (Zhao et al. 2017; Tedjopurnomo et al. 2020), and found that MC-LSTM significantly outperforms LSTM in this traffic forecasting setting (all p-values < 0.01, Wilcoxon test). For details, see Appendix B.2
Figure 3: Schematic depiction of inbound- outbound traffic situations that require the -ffvaiiiie---- principle. All ve- hicles on outbound roads (yellow arrows) must have entered the city center before (green arrows) or have been present in the first timestep.

---

Under review as a conference paper at ICLR 2021
5.3 PENDULUUM WITH FRICTION
In the area of physics, we examined the usability of MC-LSTM for the problem of modeling a swing- ing pendulum with friction. Here, the total energy is the conserved property. During the movement of the pendulum, kinetic energy is converted into potential energy and vice-versa. This conversion between both energies has to be learned by the off-diagonal values of the redistribution matrix. A
qualitative analysis of a trained MC-LSTM for this problem can be found in Appendix B.3.1
Accounting for friction, energy dissipates and the swinging slows over time until towards fixed point. This type of behavior presents a difficulty for machine learning and is impossi- ble for methods that assume the pendulum to be closed system, such Hamiltonian net- works (Greydanus et al.) (211)). We generated 120 datasets of timeseries data of pendulum where we used multiple different settings for initial angle, length of the pendulum, and the amount of friction. We then selected LSTM and MC-LSTM models and compared them with respect to predictive MSE. For an exam- ple, see Fig. + Overall, MC-LSTM has signif- icantly outperformed LSTM with mean MSE of 0.01 (standard deviation 0,04) compared to 0.05 (standard deviation 0.15; with p-value 6,9e-8, Wilcoxon test).
(b)
Potential
Energy
Kinetic
Time
Time
Figure 4: Example for the pendulum-modelin exercise. (a) LSTM trained for predicting energies of the pendulum with friction in auto-regressive fashion, (b) MC-LSTM trained in the same set- ting. Each subplot shows the potential- and kinetic energy and the respective predictions.
5.4 HYDROLOGY: RAINFALL RUNOFF MODELING
We tested MC-LSTM for sange hydrological modeling following Kratzert et al. (201cc). An ensemble of 10 MC-LSTMs was trained on 10 years of data from 447 basins using the publicly-
available CAMELS dataset (Newman et al. 2015 Addor et al. 2017a). The mass input is precipitation and auxiliary inputs are: daily min. and max. temperature, solar radiation, and vapor pressure, plus 27 basin characteristics related to geology. vegetation, and climate (described by Kratzert et al. (019c). All models besides MC-LSTM and LSTM were trained by different research groups with experience using each model. More details are given in Appendix B.4.2
As shown in Tab. 3 MC-LSTM performed better with respect to the Nash-Suter Suthlfffeee Efficiency (NSE; the R2 between simulated and observed runoff) than any other mass-conserving hydrology model, al- though slightly worse than LSTM.
Table 3: Hydrology benchmark results. All val- represent the median (25% and 75% per- centile in sub- and superscript, respectively) over the 447 basins. Only the two best per- forming hydrological models are included, An extended version can be found in Tab, B.7
NSE is often not the most important metric in hydrol- ogy, since water managers are typically concerned primarily with extremes (e.g. floods). MC-LSTM performed significantly better (p = 0.025, Wilcoxon test) than all models, including LSTM, with respect to high volume flows (FHV), at or above the 98th per- centile flow in each basin. This makes MC-LSTM the current state-of-the-arr model for flood predic- tion. MC-LSTM also performed significantly better than LSTM on low volume flows (FLV) and over- all bias, however there other hydrology models that are better for predicting low flows (which is important, e.g. for managing droughts).
Model
MC
FHVb
NSE
MC-LSTM
/
11.77
7.0
0.7440.814
23
LSTM mHM
X
-15.7
0.763
23.8
676
/
-18.6
-2777
0.666
0.588
0.70
HB Vub
18.5
0.6769-749
0.749
27.8
Mass conservation (MC).
b Top 2% peak flow bias: (-00, oo), values closer zero are dexirable.
"; Nash-Sutcliffe Efficiency: (-o0, 1), values cloxer
to one are desiruble.
Model states and environmental processes. It is an open challenge to bridge the gap between the fact that LSTM approaches give generally better predictions than other models (especially for flood prediction) and the fact that water managers need predictions that help them understand not only how much water will be in river given time, but also how water moves through basin.
8

---

Under review as a conference paper at ICLR 2021
Snow processes difficult observe and model. Kratzert et al. (2019a) showed that LSTM learns to track snow in memory cells without requiring snow data for training. We found similar behavior in MC-LSTMs,, which has the advantage of doing this with memory cells that are mass storages. Figure S shows the snow as the sum over a subset MC-LSTM memory states and snow water equivalent (SWE) modeled by the well-established Snow-17 snow model (Anderson 1973) (Pearson correlation coefficient 2 0.91). It is important to remember that MC-LSTMs did not have access any snow data during training. bias to predict how much water will be stored as snow under different conditions by using simple combinations or mixtures of the internal states. Future work will determine whether this is possible with other difficult-to-observe states and fluxes.
the best case is possible to take advantage of the inductive
400
Water Equivalent (mm)
Sum MC-LSTM 'snow" cells
200
SWe
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
Date
Figure Show-water-cqqiivalen (SWE) from a single basin. The blue line SWE modeled
by Newman et al. (2015). The orange line is the sum over MC-LSTM memory cells (Pearson correlation coefficient r 2 0.8).
6 CONCLUSION.
We have demonstrated that with the concept of inductive biases an RNN can be designed that has the property to conserve mass of particular inputs. This architecture is highly proficient as neural arithmetic unit and is well-suited for predicting physical systems like hydrological processes, in which water mass has to be conserved.
REFERENCES
Nans Addor, Andrew J Newman, Naoki Mizukami, and Martyn P Clark. The camels data set: catchment attributes and meteorology for large-sample studies. Hydrology and Earth System Sciences (HESS), 21(10):523353313 2017a.
Nans Addor, Andrew J. Newman, Naoki Mizukami, and Martyn P. Clark. Catchment attributes for large-sample studies. Boulder, CO: UCAR/NCAR, 2017b.
Eric A Anderson. National weather service river forecast system: Snow accumulation and ablation model. NOAA Tech. Memo. NWS 177, 87 pp., 1973.
Maren Awiszus and Bodo Rosenhahn. Markov chain neural networks. 2018 IEEE/VVE Conference Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2261-22617, 2018.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klbur-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):1 2015.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems 28, pp.
Curran Associates, Inc., 2015.
Tom Beucler, Michael Pritchard, Stephan Rasp, Pierre Gentine, Jordan Ott, and Pierre Baldi, Enforc- ing analytic constraints in neural-networks emulating physical systems, 2019a.
Tom Beucler, Stephan Rasp. Michael Pritchard, and Pierre Gentine. Achieving conservation of energy in neural network emulators for climate modeling. arXiv preprint /0X662190..06622, 2019b.

---

Under review as a conference paper at ICLR 2021
Tom Beucler, Stephan Rasp. Michael Pritchard, and Pierre Gentine. Achieving conservation of energy in neural network emulators for climate modeling. ICML Workshop "Climate Change: How Can Al Help?", 2019c.
Keith Beven.
Deep learning. hydrological processes and the uniqueness of place. Hydrological Processes, 34(16):3608-33) 2020.
Keith J Beven. Rainfall-unoof modelling: the primer. John Wiley & Sons, 2011.
Bernd Bohnet, Ryan McDonald, Goncalo Simoes, Daniel Andor, Emily Pitler, and Joshua Maynez. Morphosyntactic tagging with a meta-bilstm bilstm model over context sensitive token encodings. arXiv preprint arXiv:1805.082.0 08237, 2018.
Charles Bordenave, Pietro Caputo, and Djalil Chafai, Circular law theorem for random markov matrices. Probability Theory and Related Fields, 152(3-4:751-779, 2012.
Zhengdao Chen, Jianyu Zhang, Martin Arjovsky, and Léon Bottou. Symplectic recurrent neural networks. arXiv preprint 1909:1999,,,334, 2019.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of the Conference Empirical Methods in Natural Language Processing, pp. 1724-1734. Association for Computational Linguistics, 2014.
N. Cohen and A. Shashua. Inductive bias of deep convolutional networks through pooling geometry. In International Conference on Learning Representations, 2017.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. arXiv preprint arXhv:2003.04630. 2020a.
Miles Cranmer, Alvaro Sanchez Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. Advances in Neural Information Processing Systems, 33, 2020b.
Zhiyong Cui, Kristian Henrickson, Ruimin Ke, and Yinhai Wang. Traffic graph convolutional recurrent neural network: A deep learning framework for network-scale traffic learning and forecasting. IEEE Transactions on Intelligent Transportation Systems, 2019.
Gustavo Deco and Wilfried Brauer. Nonlinear higher-order statistical decorrelation by volume- conserving neural architectures. Neural Networks, 8(4):525-553 535, 1995. ISSN 0893-6080.
Stanislas Dehaene, The number sense: How the mind creates mathematics. Oxford University Press, 2 edition, 2011. 1. ISBN 9780199753871.
Martin R Evans and Tom Hanney. Nonequilibrium statistical mechanics of the zero-range process and related models. Journal of Physics A: Mathematical and General, 38(1)),,195 2005.
R Allan Freeze and RL Harlan. Blueprint for physically-based, digitally-simulated hydrologic response model. Journal of Hydrology, 93))2377-588 1969.
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4):193-202, 1980.
A. Gaier and D. Ha. Weight agnostic neural networks. In Advances in Neural Information Processing Systems 32, pp. 5364-5378, Curran Associates, Inc., 2019.
C. R. Gallistel. Finding numbers in the brain. Philosophical Transactions of the Royal Society B: Biological Sciences, 373(1740), 2018. doi: 10.1099//stb.2017.000
Felix A. Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with Istm. Neural Computation, (110):2451-2411 2000.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems 32, pp. 15353-15363, Curran Associates, Inc., 2019.
10

---

Under review as a conference paper at ICLR 2021
David Ha, Andrew Dai, and Quoe Le. Hypernetworks. In International Conference Learning Representations, 2017.
M. Hairer. Ergodic properties of markov processes. Lecture notes, 2018.
K. He, Y. Wang, and J. Hoperoft. A powerful generative model using random weights for the deep image representation. Advances Neural Information Processing Systems 29, pp. 631-639. Curran Associates, Inc., 2016.
Sepp Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. PhD thesis, Technische Universität München, 1991.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pp. 148-456. PMLR, 2015.
Raban Iten, Tony Metger, Henrik Wilming, Lídia Del Rio, and Renato Renner. Discovering physical concepts with neural networks. Physical Review Letters, 124(1):010508, 2020.
Xiaowei Jia, Jared Willard, Anuj Karpatne, Jordan Read, Jacob Zwart, Michael Steinbach, and Vipin Kumar. Physics guided rnns for modeling dynamical systems: A case study in simulating lake temperature profiles. In Proceedings of the 2019 SIAM International Conference Data Mining, pp. 558-566. SIAM, 2019.
Anuj Karpatne, Gowtham Atluri, James Faghmous, Michael Steinbach, Arindam Banerjee, Auroop Ganguly, Shashi Shekhar, Nagiza Samatova, and Vipin Kumar. Theory-guided data science: A new paradigm for scientific discovery from data. IEEE Transactions on Knowledge and Data Engineering, 29(10):2318-333 2017.
Diederik P. Kingma and Ba Jimmy. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems 30, pp. 971-980, 2017.
Elena Kochkina, Maria Liakata, and Isabelle Augenstein. Turing at semeval-2017 task 8: Sequential approach rumour stance classification with branch-Istm. arXiv preprint arXiv: 17X1v:1704.07221, 2017.
Frederik Kratzert, Daniel Klotz, Claire Brenner, Karsten Schulz, and Mathew Herreggeer Rainfall- runoff modelling using long short-term memory (Istm) networks. Hydrology and Earth System Sciences, 22(1 00060066622.... 2018.
Frederik Kratzert, Mathew Hermegger, Daniel Klotz, Sepp Hochreiter, and Günter Klambauer. Neuralihydrologyy-tteeeree LSTMs in Hydrology, pp. 347-362. Springer, 2019a.
Frederik Kratzert, Daniel Klotz, Mathew Hermegger, Alden K Sampson, Sepp Hochreiter, and Grey Nearing. Toward improved predictions in ungauged basins: Exploiting the power of machine learning. Water Resources Research, 55(12):1 1344-1 11354, 2019b.
Frederik Kratzert, Daniel Klotz, Guy Shalev, Günter Klambauer, Sepp Hochreiter, and Grey Nearing. Towards learning universal, regional, and local hydrological behaviors via machine learning applied to large-sample datasets. Hydrology and Earth System Sciences. 23(12):5089-5110 2019c.
Frederik Kratzert, Daniel Klotz, Sepp Hochreiter, and Grey Nearing. A note on leveraging synergy in multiple meteorological datasets with deep learning for rainfall-runoff modeling. Hydrology and Earth System Sciences Discussions, 2020: 1-26, 2020.
David P Kreil, Michael Kopp. David Jonietz, Moritz Neun, Aleksandra Gruca, Pedro Herruzo, Henry Martin, Ali Soleymani, and Sepp Hochreiter. The surprising efficiency of framing geo-spatial time series forecasting as a video prediction task insights from the iarai traffic4cast competition at neurips 2019. In NeurlPS 2019 Competition and Demonstration Track, pp. 232-241. PMLR, 2020.

---

Under review as a conference paper at ICLR 2021
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojeiech Samek, and Blaus-Rober Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 10(1):1-8, 2019.
Y. LeCun and Y. Bengio. Convolutional Networks for Images, Speech, and Time Series, pp. 255-258. MIT Press, Cambridge, MA, USA, 1998.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 2015.
Gang Liu and Jiabao Guo. Bidirectional lstm with attention mechanism and convolutional layer for text classification. Neurocomputing. 337:325-338, 2019.
Yang Liu, Zhiyuan Liu, and Ruo Jia. Deeppf: A deep learning based architecture for metro passenger flow prediction. Transportation Research Part C: Emerging Technologies, 101:18-34, 2019.
Andreas Madsen and Alexander Rosenberg Johansen. Neural arithmetic units. In International
Conference on Learning Representations, 2020.
M. Mitchell. The need for biases in learning generalizations. Technical Report CBM-TR-117, Rutgers University, Computer Science Department, New Brunswick, NJ, 1980.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. International Conference on Learning Representations, 2018.
Naoki Mizukami, Martyn P. Clark, Andrew J. Newman, Andrew W. Wood, Ethan D. Gutmann, Bart Nijssen, Oldrich Rakovec, and Luis Samaniego. Towards seamless large-domain parameter estimation for hydrologic models. Water Resources Research, 53(9):8020-8044 2017.
Naoki Mizukami, Oldrich Rakovec, Andrew J Newman, Martyn P Clark, Andrew W Wood, Hoshin V Gupta, and Rohini Kumar. On the choice of calibration metries for "high-flow" estimation using hydrologic models. Hydrology and Earth System Sciences, 236)):601-2214, 2019.
Do H Nam and Donald Drew. Traffic dynamics: Method for estimating freeway travel times in real time from flow measurements. Journal of Transportation Engineering, 122(3):185-191, 1996.
Grey S. Nearing, Yudong Tian, Hoshin V. Gupta, Martyn P. Clark, Kenneth W. Harrison, and Steven V. Weijs. A philosophical basis for hydrological uncertainty. Hydrological Sciences Journal, 61)) 1666-1678, 2016.
AJ Newman, K Sampson, MP Clark, A Bock, RJ Viger, and D Blodgett. A large-sample watershed- scale hydrometeorological dataset for the contiguous USA. Boulder, CO: UCARNCAR, 2014.
AJ Newman, MP Clark, Kevin Sampson, Andrew Wood, LE Hay, A Bock, RJ Viger, D Blodgett, Brekke, JR Arnold, al. Development of large-sample watershed-seale hydrometeorological data set for the contiguous USA: data set characteristics and assessment of regional variability in hydrologic model performance. Hydrology and Earth System Sciences, 19(1):209 2209 2015.
Andrew Newman, Naoki Mizukami, Martyn P Clark, Andrew Wood, Bart Nijssen, and Grey Nearing, Benchmarking of physically based hydrologic model. Journal of Hydrometeorology, 18(8):2215-22 2225, 2017.
Andreas Nieder. The neuronal code for number. Nature Reviews Neuroscience, 17 186):366-382, doi: https://doi.org/10.. http:////.....000....0.0066.60
Christopher Olah. Understanding L.STM networks, 2015. URLlhttpps:/cclaah.githti fo/
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Technical report, DeepMind, 2019.
12

---

Under review as a conference paper at ICLR 2021
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Herschel Rabitz, Ömer Alis, Jeffrey Shorter, and Kyurhee Shim. Efficient input- input-output model representations. Computer physics communications, 117(1-2):11-20, 1999.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
Oldrich Rakovec, Naoki Mizukami, Rohini Kumar, Andrew J Newman, Stephan Thober, Andrew w Wood, Martyn P Clark, and Luis Samaniego. Diagnostic evaluation of large-domain hydro- logie models calibrated across the contiguous united states. Journal of Geophysical Research: 2019.
Atmospheres, (3924)::3991-1444)
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference Machine Learning, volume 37, pp. 1530-1538. PMLR, 2015.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dy- namics of learning in deep linear neural networks. In International Conference Learning Representations, 2014.
Schmidhuber, D. Wierstra, M. Gagliolo, and F. Gomez. Training recurrent networks by Evolino. Neural Computation, 19(3):757-779, 2007.
Jürgen Schmidhuber. Learning to control fast- memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139, 1992.
Jürgen Schmidhuber: Deep learning in neural networks: An overview. Neural networks, 61:85-117, 2015.
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data, science, 324(5923)811-85, 2009.
Jan Scibert, Mare J. P. Vis, Elizabeth Lewis, and H. J. van Meerveld. Upper and lower benchmarks in hydrological modelling. Hydrological Processes, 32(8):1 1000::112011100 120-1125, 2018.
SL Sellars. "grand challenges" in big data and the earth sciences. Bulletin of the American Meteoro- logical Society, E955,E5995,,,, 2018.
Xin-Hua Song and Philip K Hopke. Solving the chemical mass balance problem using an artificial neural network. Environmental science & technology, 30(2):531-535, 1996.
Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112, 12,,, 2014.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
David Alexander Tedjopurnomo, Zhifeng Bao, Baihua Zheng, Farhana Choudhury, and AK Qin. A survey on modern deep neural network for traffic prediction: Trends, methods and challenges.
IEEE Transactions on Knowledge and Data Engineering, 2020.
E. Todini. Rainfall-runoff modeling past, present and future. Journal of Hydrology. 100(1): 341-352, 1988. ISSN 0022-1694.
Andrew Trask, Felix Hill, Scott E Reed, Jack Rae, Chris Dyer, and Phil Blunsom. Neural arithmetic logic units. In Advances in Neural Information Processing Systems 31, pp. 8035-8044, Curran Associates, Inc., 2018.
13

---

Under review as a conference paper at ICLR 2021
D. Ulyanov, A. Vedaldi, and V. Lempitsky. Deep image prior. International Journal of Computer Vision, 128(7):1867-1888 2020.
A. J. van der Schaft, M. Dalsmo, and B. M. Maschke. Mathematical structures in the network representation of energy-conserving physical systems. In Proceedings of 35th IEEE Conference on Decision and Control, volume 1, pp. 201-206, 1996.
Lelitha Vanajakshi and LR Rilett. Loop detector data diagnostics based on conservation-oo--vhiiles principle. Transportation research record, 1870(1):162- 169, 2004.
Xinping Xiao and Huiming Duan. A new grey model for traffic flow mechanics. Engineering Applications of Artificial Intelligence, 88: 88:103350, 2020.
LI Yitian and Roy R Gu. Modeling flow and sediment transport in river system using artificial neural network. Environmental management, 31(1):0122-014, 2003.
Zheng Zhao, Weihai Chen, Xingming Wu, Peter CY Chen, and Jingmeng Liu. Lstm network: a deep learning approach for short-term traffic forecast. IET Intelligent Transport Systems, 11(2):68-75, 2017.
14

---

Under review as a conference paper at ICLR 2021
A
NOTATION OVERVIEW
Most of the notation used throughout the paper, summarized in Tab. A.
Table A.1: Symbols and notations used in this paper.
Definition
Symbol/Notation
Dimension
mass input at timestep / auxiliary imput timestep cell state at timestep t limit of sequence of cell states hidden state at timestep redistribution matrix input gate
x² or x a' e²
M or
K
h'
K
R
K
K
K
output gate
K
mass
m
K K K
input gate weight matrix input gate weight matrix output gate weight matrix output gate weight matrix identity matrix
W
W.
U;
K K
U.
K X K
K
K X
input gate bias
b.
K
output gate bias
K
arbitrary differentiable function
hypernetwork function (conditioner)
redistribution gate bias stored mass
BR
K x
K
me my m Po
mass efflux
limit of series mass inputs timestep index
an arbitrary timestep
last timestep of sequence
T W,
redistribution gate weight tensor redistribution gate weight tensor arbitrary feature index arbitrary feature index arbitrary feature index
KXXXL
U
KxKx
B EXPERIMENTAL DETAILS
In the following, provide further details on the experimental setups.
B. NEURAL ARITHMETIC
Neural networks that learn arithmetic operations have recently come into focus (Trask et al. 2018 Madsen & Johansen 2020). Specialized neural modules for arithmetic operations could play role for complex AI systems since cognitive studies indicate that there is a part of the brain that enables animals and humans perform basic arithmetic operations (Nieder 2016 Gallistel] 20188.. Although this primitive number processor can only perform approximate arithmetic, it is fundamental part of our ability to understand and interpret numbers (Dehaene 201T).
B.1.1 DETAILS ON DATASETS
We consider the addition problem that proposed in the original LSTM paper (Hochreiter & Schmidhuber 1997). We chose input values the range [o, 0.3] in order to able to use the fast standard implementations of L.S.. For this task, 20 000 samples were generated using fixed
15

---

Under review as a conference paper at ICLR 2021
random seed create dataset, which split in 50% training and 50% validation samples. For the test data, a different random seed was used.
A definition of the static arithmetic task is provided by (Madsen & Johansen 2020). The following presents this definition and its extension to the recurrent arithmetic task (c.f. Trask et al. 2018).
The input for the static version is a vector, randomly from a uniform distribution. The target. y, is computed as
(11 2) 100, consisting of numbers that drawn
a+c
b+c Σ
-
Jk
R=a
For the recurrent variant, the input consists of where N. a Sbsuba C N and D {+, - 2i
a sequence of T vectors, denoted by x E U(1, 2) 2)20 I (I,..., Th and the labels are computed as
b+c
y
-
k=b
For these experiments, fixed datasets were used. Instead, samples were generated on the fly. Note that since the subsets overlap, i.e., inputs are re-used, this data does not have mass conservation properties.
For more detailed description of the MNIST addition data, we refer to Traak et al. 2018) and the appendix of (Madsen & Johansen 2020).
B.1.2 DETAILS ON HYPERPARAMETERS
For the addition problem, every network had a single hidden layer with 10 units. The output layer was a linear, fully connected layer for all MC-LSTM and LSTM variants. The NAU (Madsen & Johansen 2020) and NALU/NAC (Trask et al., 2018) networks used their corresponding output layer. Also, we used more common L2 regularization scheme with low regularization constant (10-1) to keep the weights ternary for the NAU, rather than the strategy used the reference implementation from Madsen & Johansen (2020). Optimization was done using Adam (Kingma & Jimmy (2015) for all models. The initial learning rate was selected from {0.1, 0.05, 0.01, 0.005, 0.001} on the validation data for each method individually. All methods were trained for 100 epochs.
The weight matrices of LSTM were initialized in a standard way, using orthogonal and identity matrices for the forward and recurrent weights, respectively. Biases were initialized be zero, except for the bias in the forget gate, which was initialized 3. This should benefit the gradient flow for the first updates. Similarly, MC-LSTM is initialized that the redistribution matrix (cf. Eq. 7 is (close to) the identity matrix. Otherwise we used orthogonal initialization (Saxe et al.] 2014). The bias for the output gate was initialized -3, This stimulates the output gates stay closed (keep mass in the system), which has a similar effect as setting the forget gate bias in LSTM. This practically holds for all subsequently described experiments.
For the recurrent arithmetic tasks, we tried to stay as close as possible to the setup that was used by Madsen & Johansen (2020). This means that all networks had again single hidden layer. The NAU, Neural Muluplication Unit (NMU) and NALU networks all had two hidden units and, respectively, NAU, NMU and NALU output layers. The first, recurrent layer for the first two networks was NAU and the NALU network used recurrent NALU layer. For the exact initialization of NAU and NALU, we refer to (Madsen Johansen] 2020).
The MC-LSTM models used a fully connected linear layer with Iz-regularization for projecting the hidden state to the output prediction for the addition and subtraction tasks. It is important to use free linear layer in order to compensate for the fact that the data does not have mass-conserving propertics. However, is important to note that the mass conservation MC-LSTM is still necessary to solve this task. For the multiplication problem, MC-LSTM with an extra scalar parameter to allow the conserved mass to be re-scaled if necessary. This multiplicative layer is described in more detail in Appendix B.1.3
used multiplicative, non-recurrent variant of
Whereas the addition could be solved with two hidden units, MC-LSTM needed three hidden units to solve both subtraction and multiplication. This extra unit, which refer the trash cell, allows
16

---

Under review as a conference paper at ICLR 2021
MC-LSTMs to get rid of excessive mass that should not influence the prediction. Note that, since the mass imputs are vectors, the input gate has to be computed in a similar fashion as the redistribution matrix. Adam again used for the optimization. We used the same learning rate as Madsenn & Johansen 2020) (0.001) to train the NAU, NMU and NALU networks. For MC-LSTM the learning rate was increased to 0.01 for addition and subtraction and 0.05 for multiplication after manual search on the validation set. All models were trained for two million update steps.
In a similar fashion, we used the same models from (Madsen & Johansen 2020) for the MNIST addition task. For MC-LSTM, we replaced the recurrent NAU layer with a MC-LSTM layer and the output layer was replaced with a fully connected linear layer. In this scenario, increasing the learning rate was not necessary. This can probably be explained by the fact that training CNN to regress the MNIST images is the main challenge during learning. We also used a standard I-greguulrizzttinn on the outputs of CNN instead of the implementation proposed in (Madsen & Johansen 2020) for this task.
B.1.3 STATIC ARITHMETIC
This experiment should enable a more direct comparison to the results from Madsen & Johansen (2020) than the recurrent variant. The data for the static task equivalent that of the recurrent task with sequence length one. For more details on the data, we refer to Appendix B.1.1 or (Madsen & Johansen 2020).
Since the static task does not require recurrent model, we discarded the redistribution matrix in MC-LSTM. The result layer with only input and output gates, which we refer to as an Mass- Conserving Fully Connected (MC-FC) layer. We compared this model to the results reported in (Madeen & Johansen 2020). using the code base that accompanied the paper. All NALU and NAU networks had single hidden layer. Similar to the recurrent task, MC-LSTM required two hidden units for addition and three for subtraction. Mathematically, an MC-FC with K hidden neurons and M inputs can be defined as MC-FC:
R + R
y, where
diag(o) I
I = softmax(B,)
o (b.))
where the softmax operates on the row dimension to get column-normalized matrix, for the input gate.
Using the log-exp transform (c.f. Trask al. 2018), multiplicative MC-FC with scaling parameter, a, can be constructed as follows, eCp(MC-FC[loo (x)) ax). The scaling parameter is necessary to break the mass conservation when it is not needed. By replacing the output layer with this multiplicative MC-FC, it can also be used to solve the multiplication problem. This network also required three hidden neurons. This model was compared to a NMU network with two hidden neurons and NALU network.
All models were trained for two million updates with the Adam optimizer (Kingma Jimmy 2015). The learning rate was set to 0.001 for all networks, except for the MC-FC network, which needed lower learning rate 0.0001, and the multiplicative MC-FC variant, which was trained with learning rate 0.01. These hyperparameters were found using manual search.
Since the input consists of a vector, the input gate predicts a left-stochastic matrix, similar to the redistribution matrix. This allows to verify generalization abilities of the inductive bias in MC-LSTMs. The performance was measured in a similar way as the recurrent task, except that generalization was tested over the range of the input values (Madsen & Johansen, 2020). Concretely, the models were trained on input values in [1, 2] and tested on input values in the range [2, 6]. Table B.2 shows that MC-FC is able match or outperform both NALU and NAU this task.
1. COMPARISON WITH TIME MEEE-DEPNNDET MC-LSTM
We used MC-LSTM with a time-independent redistribution matrix, as Eq. 77. to solve the addition problem. This resembles another form of inductive bias, since we know that no redistribution across cells necessary to solve this problem and results also in a more efficient model, because less parameters have to be learned. However, for the sake of flexibility, use the more general deeendennd redistribution matrix (cf. Eq. 8). The results this experiment can be found in Table B.3
also verified that it is possible to
17

---

Under review as a conference paper at ICLR 2021
Table B.2: Results for the static arithmetic task. MC-FC is mass-conserving variant of MC-LSTM based on fully-connected layers for non-recurrent tasks. MC-FCs for addition and subtrac- tion/multillication have two and three neurons, respectively. Error bars represent 95% confidence
intervals.
addition
subtraction
multiplication
success rate
updates
success rate
updates
success rate
updates
MC-FC
100%
+0%
2.1 10
100%
+0%
1.6 10a
100% +0%
1.4 10
-4%
-4%
-4%
NAU / NMU
100%
+0%
1.8 104
100%
+0%
5.0 10
98%
+1%
1.4 . 106
-0%
NAC
100%
+0%
2.5 105
100%
+0%
9.0 103
31%
+10%
2.8 10
8%%
NALU
14%
+88%
1.5 106
14%
+8%
1.9 106
0% -0%
+4%
5%
5%
a Percentage of runs that generalized to a different input range.
Median number of updates necessary to solve the task.
Table B.3: Performance of different models on the LSTM addition task in terms of the MSE. MC-LSTM significantly (all p-values below .05) outperforms its competitors, LSTM (with high
initial forget gate bias), NALU and NAU. Error bars represent -o%fconneeeec intervals across 100
runs.
reference*
seq length
input range
count
combo
NaN
MC-LSTM
0.013 + 0.004
0.022 + 0.010
2.6 + 0.8
2,2 + 0.7
13.6 + 4.0
0
MC-LSTM
0.004 + 0.003
0.009 + 0.004
0.8 + 0.5
0.6 + 0.4
4.0 + 2.5
0
LSTM
0.008 + 0.003
0.727 = 0.169
21.4
9.5 9 0.6
54.6 + 1.0
0
NALU
0.060 0.008
0.059
= 0.009
2.2
7.4
63.7
+ 0.6
93
NAU
0.248 + 0.019
0.252 = 0.020
28.3
+ 0.5
9.1 0.2
68.5
+ 0.8
24
training regime:
summing 2 of 100 numbers between and 0.5.
b longer sequence lengths:
summing 2 out of 000 numbers between 0 and 0.5.
more mass the imput:
summing 2 of 100 numbers between 0 and 5.0.
d higher number of summands:
summing 20 out of 100 numbers between 0 and 0.5.
combination of previous scenarios:
summing 10 out of 500 numbers between and 2.5.
Number of runs that did not converge.
MC-LSTM with time-dependent redistribution matrix.
Although the performance of MC-LSTM with time-dependent redistribution matrix is slightly worse than that of the more efficient MC-LSTM variant, it still outperforms all other models the generalisation tasks. This can partly be explained by the fact that is harder to train a time-dependent redistribution matrix, and the training budget is limited to 100 epochs.
B. QUALITATIVE ANALYSIS OF THE MC-LSTM MODELS TRAINED ON ARITHMETIC TASKS
Addition Problem. To reiterate, we used MC-LSTM with 10 hidden units and linear output layer. The model has to learn to sum all mass inputs of the timesteps, where the auxiliary input (the marker) equals a² 1, and ignore all other values. the final timestep - where the auxiliary input equals a' 11 - the network should output the sum of all previously mark mass inputs.
In our experiment, the model has learned to store the marked input values in a single cell, while all other mass inputs mainly end up in single, different cell. That is, single cell leams to accumulate the inputs to compute the solution and the other cells are used as trash cells. In Fig. B. we visualize the cell states for single input sample over time, where the orange and the blue line denote the mass accumulator and the main trash cell, respectively.
We that at the last time step where the network is queried return the accumulated sum the value that particular cell does not drop to zero (i.e., not the entire value that is actually accumulated is removed from the system). For this particular model, the corresponding output gate
18

---

Under review as a conference paper at ICLR 2021
value for this cell at the last time step is 0.18134. That is, only 18.134% of the actual accumulated value is returne.. However, the weight of the linear layer that corresponds to this cell for this model is 5.5263. If we multiply these two values, the result is 1.0021, which means the model recovers the value stored in the cell state. For all other cells (grey lines), either the output gate at the last time step, the weight of the linear layer, the cell value itself is zero. That means that the model output is only determined by the value of the single cell that acted as accumulator of the marked values (orange line).
We also analyzed MC-LSTM without linear output layer for the same addition task. In this case, the model output is determined as the sum over the outgoing mass. As before, the model also uses single cell to store the accumulated values of the marked timesteps. However, because no scaling can be learned from the linear output layer, the model learned to fully open the output at the query timestep.
1.2
trash cell
main cell
other cells
Cell state value
0.6
0.4
20
40
60
80
100
timestep
Figure B.1: MC-LSTM cell states over time for model trained to solve the addition problem (see Appendix B...II. Each line denotes the value of one particular cell over time, while the two vertical grey indicator lines denote the timesteps, where the auxiliary input was I (i.e., those numbers have to be added).
Recurrent Arithmetic. In the following we take closer look at the solution that is learned with MC-LSTM. Concretely, we look at the weights of a MC-LSTM model that successfully solves the following recurrent arithmetic task:
T
Σ
y =
where O E (-,+), given sequence of imput vectors x' E R (the only purpose of the colors is to provide an aid to readers). We highlight the following observations:
1. For the addition task (i.e., D= = +). MC-LSTM has two units (see Appendix 1.1 for details on the experiments). Trask et al. (2018); Madsen & Johansen (2020) fixed the number of hidden units to two with the idea that each unit can learn one term of the addition operation (D). However, if we take a look at the input gate of our model, we find that the first cell is used to accumulate {zf +... + 2 + 0.52% + 0.5zf + xá + +xio) and the second cell collects (0.5.g + x4 + 0.5z)). ) Since the learned redistribution matrix is the identity matrix, these accumulators operate individually.
This means that, instead computing the individual terms, MC-LSTM directly computes the solution, scaled by factor 1/2 in its second cell. The first cell accumulates the rest of the mass, which it does not need for the prediction. In other words, operates as some sort of trash cell. Note that due to the mass-conservation property, it would be impossible to
19

---

Under review as a conference paper at ICLR 2021
compute each side of the operation individually. After all, 24 appears both sides of the central operation (D), and therefore the data is not mass conserving.
The output gate is always open for the trash cell and closed for the other cell, indicating that redundant mass is discarded through the output of the MC-LSTM in every timestep and the scaled solution is properly accumulated. However, in the final timestep prediction is to be made, the output gate for the trash cell is closed and opened for the other cell. That is, the accumulated solution is passed to the final linear layer, which scales the output of MC-LSTM by factor of two to get the correct solution.
when the
2.
For the subtraction task (i.e., D = -), similar behavior can be observed. In this case, the final model requires three units to properly generalize. The first two cells accumulate zg and zi,, respectively. The last cell operates as trash cell and collects (af +.. + 25 +2 +o)) The redistribution matrix is the identity matrix for the first two cells. For the trash cell, equal parts (0.4938) are redistributed to the two other cells. The output gate operates in a similar fashion as for addition. Finally, the linear layer computes the difference between the first. two cells with weights I, 1 and the trash cell is ignored with weight O.
Although MC-LSTM with two units was not able generalize well enough for the Madsen & Johansen 2020) benchmarks, it did turn out to be able to provide a reasonable solution (albeit with numerical flaws). With two cells, the network learned to store (0.5xl +............... + 0.5z4+0..5zz +0.5x1)) in one cell, and (0.5zf + in the other cell. With a similar linear layer as for the three-unit variant, this solution should also compute correct solution for the subtraction task.
...... +0.5xz +0.5x +ag + 0.55z +0.5x1p)
B.2 INBOUND-OUTBOUNE TRAFFIC FORECAST
Traffic forecasting considers large number of different settings and tasks (Tedjopurnomo et al.) 2020). For example whether the physical network topology of streets can be exploited by using graph neural networks combined with LSTMs (Cui et al.) 2019). Within traffic forecasting mass conservation translates to a oonnservationoo--veiicee
principle. Generally, models that adhere to this principle are desired (Vanajakshi & Rileu (2004] [Zhao et al.) (017) since they could be useful for long-term forecasts, Many recent benchmarking datasets for traffic forecasts are usually uni-directional and are measured at few streets. Thus conservation laws cannot be directly applied (Tedjopurnomo et al., 2020).
We demonstrate how MC-LSTM can be used in traffic forecasting settings. A typical setting for vehicle conservation is when traffic counts for inbound and outbound roads of a city are available. In this case, all vehicles that come from an inbound road must either be within a city or leave the city on outbound road. The setting is similar to passenger flows in inbound and outbound metro (Liu et al.] 2019), where LSTMs have also prevailed. We were able to extract such data from a recent dataset based on GPS-locations (Kreil et all 2020) of vehicles at fine geographic grid around cities, which represents good approximation of a vehicle conserving scenario.
An approximately mass-conserving traffic dataset Based on the data for the traffic4cast 2020 challenge (Kreil et al. (2020), we constructed a dataset to model inbound and outbound traffic of three different cities: Berlin, Istanbul and Moscow. The original data consists of 181 sequences of multi-channel images encoding traffic volume and speed for every five minutes in four (binned) directions. Every sequence corresponds to a single day in the first half of the year. In order to get the traffic flow from the multi- multi-channel images at every timestep. we defined frame around the city and collected the traffic-volume data for every pixel on the border of this frame. This illustrated in Fig. 3 For simplicity, we ignored the fact that a single-pixel frame might have issues with fast-moving vehicles.
By taking into account the direction of the vehicles, the inbound and outbound traffic can be combined for every pixel on the border of our frame. To get a more tractable dataset, we additionally combined the pixels of the four edges of the frame to end up with eight values: four values for the incoming traffic, i.e: one for each border of the frame, and four values for the outgoing traffic. The inbound traffic would be the mass input for MC-LSTM and the target outputs are the outbound traffic along the different borders. The auxiliary input is the current daytime, encoded as a value between zero and one.
20

---

Under review as a conference paper at ICLR 2021
Table B.4: The hyperparameters resulting from the grid search for the traffic forecast experiment
hidden
forget bias
initial state
learnable state
LSTM
10
0.01
0
Berlin
MC-LSTM
100
0.01
True
LSTM
100
0.005
5
Istanbul
MC-LSTM
50
0.01
0
False
LSTM
50
0.001
5
Moscow
MC-LSTM
10
0.01
False
To model the sparsity that is often available in other traffic counting problems, we chose three time-slots (6 am. 12 pm and 6 pm) for which we use fifteen minutes of the actual measurements - -i.. three timesteps. This could for example simulate the deployment of mobile traffic counting stations. The other inputs are imputed by the average inbound traffic over the training data, which consists of 181 days. Outputs are only available when the actual measurements are used. This gives total of 9 timesteps per day on which the loss can be computed. For training, this dataset randomly split in 85% training and 15% validation samples.
During inference, all 288 timesteps of the inbound and outbound measurements are used find out which model learned the traffic dynamics from the sparse training data best. For this purpose, we used the 18 days of validation data from the original dataset as test set, which are distributed across the second half of the year. In order to enable a fair comparison between LSTM and MC-LSTM, the data for LSTM was normalized to zero mean and unit variance for training and inference (using statistics from the training data). MC-LSTM does not need this pre- pre-processing step and is fed the raw data.
Model and Hyperparameters For the traffic prediction, we used L.STM followed by a fully connected layer baseline (c.f. Zhao et al. 2017 Liu et al. 2019). For MC-LSTM, chose to enforce end-to-end mass conservation by using a MC-FC output layer, which is described in detail in Appendix [B.1.]] For the initialization of the models, we refer to the details of the arithmetic experiments in Appendix B.I
For each model and for each city, the best hyperparameters were found by performing grid search on the validation data. This means that the hyperparameters were chosen to minimize the error on the nine 5-minute intervals. For all models, the number of hidden neurons was chosen from {10, 50, 100) and for the learning rate, the options were {0.100, 0.050, 0.010, 0.005, 0.001 }. All models were trained for 2 000 epochs using the Adam optimizer (Kingma & Jimmy 2015). Additionally, we considered values in (0, 5} for the initial value for the forget gate bias in LSTM. For MC-LSTM, the extra hyperparameters were the initial cell state value (E {0. 100)) i.e., how much cars are in each memory cell at timestep zero and whether or not the initial cell state should be trained via backpropagation. The results of the hyperparameter search can be found Tab, B.4
The idea behind tuning the initial cell state, is that unlike with LSTM, the cell state in MC-LSTM directly reflects the number of cars that can drive out of city during the first timesteps. If the initial cell state is too high or too low, this might negatively affect the prediction capabilities of the model. If would be possible to estimate the number of cars in a city at the start of the sequence, this could also be used to get better estimates for the initial cell state. However, from the results of the hyperparameter search (see Tab. B.4), we might have overestimated the importance of these hyperparameters.
Results. All models were evaluated on the test data, using the checkpoint after 2 200 epochs for fifty runs. An example of what the predictions of both models look like for an arbitrary day in an arbitrarily chosen city is displayed in Fig. B.2 The average Root MSE (RMSE) and Mean Absolute Error (MAE) are summarized in Tab, B.5 The results show that MC-LSTM is able to generalize significantly better than LSTM for this task. The RMSE of MC-LSTM is significantly better than
21

---

Under review as a conference paper at ICLR 2021
gat
LSTH
MCT
prokaed OOUF
31
30
20
obse vod DESE COUST
DESERNNN TRETI COUTE
Obeareed DISTIC CO.0T1
DESARAIC TTER COUTT
Figure B.2: Traffic forecasting models for outbound traffic in Moscow. An arbitrary day has been chosen for display. Note that both models have only been trained on data at timesteps 71-73, 143- 145, and 215-217. Colors indicate the four borders of the frame, i.e. north, east, south and west. Left: LSTM predictions shown dashed lines versus the actual traffic counts (solid lines). Right: MC-LSTM predictions shown in dashed lines versus the actual traffic counts (solid lines).
Table B.5: Results on outbound traffic forecast avg RMSE and MAE with 95% confidence intervals
over 50 runs
Istanbul
Berlin
Moscow
RMSE
MAE
RMSE
MAE
RMSE
MAE
MC-LSTM
7.3 + 0.1
28 + 2
13.6 + 1.8
66 + 1 25.5 + 1.1
27.8 + 1.1
LSTM
142.6 + 4.4
84 + 3
135.4 + 5.0
84
+ 3
45.6 + 0.8
31.7 + 0.5
LSTM (p-values 10-10, 8 and 4e-10 10 for Istanbul, Berlin, and Moscow, respectively, Wilcoxon test).
B.3 PENDULUM WITH FRICTION
In the area of physics, we consider the problem of modeling swinging pendulum with friction. The conserved quantity of interest is the total energy. During the movement of the pendulum, kinetic energy converted into potential energy and vice-versa. Neglecting friction, the total energy is conserved and the movement would continue indefinitely. Accounting for friction, energy dissipates and the swinging slows over time until fixed point reached. This type of behavior presents difficulty for machine learning and is impossible for methods that assume the pendulum to be closed systems, such as Hamiltonian networks (Greydanus al. 2019). We postulated that both energy conversion and dissipation can be fitted by machine learning models, but that an appropriate inductive bias will allow to generalize from the learned data with more ease.
To train the model, we generated set of timeseries using the differential equations for a pendulum with friction. We used multiple different settings for initial angle, length the pendulum, the amount of friction, the length of the taining--priood and with and without Gaussian noise. Each model received the initial kinetic and potential energy of the pendulum and must predict the consecutive timesteps. The time series starts always with the pendulum at the maximum displacement - i.e., the entire energy in the system is potential energy. We generated timeseries of potential- and kinetic energies by iterating the following settings/condition:: initial amplitude (f0.2, 0.4)), pendulum length (f0.75, 1)), length of training sequence in terms of timesteps ((100, 200, 400)), noise level ((0, 0.01)), and dampening constant ((0.0, 0.1, 0.2, 0.4, 0.8)). All combinations of those settings
22

---

Under review as a conference paper at ICLR 2021
were used to generate total of 120 datasets, for which we train both models (the auto-regressive LSTM and MC-LSTM).
We trained an auto-regressive LSTM that receives its current state and low-dimensional temporal embedding (using nine sinusoidal curves with different frequencies) to predict the potential and kinetic energy of the pendulum. Similarly, MC-LSTM is trained in autoregressive mode, where a hypernetwork obtains the current state and the same temporal embedding as LSTM. The model- setup is thus similar an autoregressive model with exogenous variables from classical timeseries modelling literature. To obtain suitable hyperparameters we manualy adjusted the learning rate (0.01), hidden size of LSTM (256), the hypernetwork for estimating the redistribution (a fully connected network with 3 layers, ReLu activations and hidden sizes of 50, 100, and 2 respectively), optimizer (Adam, Kingma& & Jimmy 2015) and the training procedure (crucially, the amount of additionally considered timesteps in the loss after a threshold is reached. See explanation of the used loss below), on a separately generated validation dataset.
For MC-LSTM, a hidden size of two was used so that each state directly maps to the two energies. The hypernetwork consists of three fully connected layers of size 50, 100 and 4, respectively. To account for the critical values at the extreme-points of the pendulum (i.e. the amplitudes - where the energy is present only in the form of potential energy - and the midpoint- - where only kinetic energy exists), we slightly offsee the cell state from the actual predicted value by using a linear regression with a slope of 1 1.02 and an intercept -0.01.
For both models, we used a combination of Pearson's 's correlation of the energy signals and the MSE as a loss function (by subtracting the former mean from the latter). Further, we used a simple curriculum to deal with the long autoregressive nature of the timeseries (Bengio et al. 2015): Starting a time window of eleven we added five additional timesteps whenever the combined loss was below -0.9.
Overall, MC-LSTM has significantly outperformed LSTM with mean MSE of 0.01 (standard deviation 0.04) compared to 0.05 (standard deviation 0.15; with a p-value 6,9e-8, Wilcoxon test).
B.3.1 QUALITATIVE ANALYSIS OF THE MC-LSTM MODELS TRAINED FOR A PENDULUM
In the following, we analyse the behavior of the simplest pendulum setup, i.e., the one without friction. Special to the problem of the pendulum without friction is that there are no mass in- or outputs and the whole dynamic of the system has to be modeled by the redistribution matrix. The initial state of the system is given by the displacement of the pendulum at the start, where all energy is stored as potential energy. Afterwards, the pendulum oscillates, converting potential to kinetic energy and vere-versa.
In MC-LSTM. the conversion between the two forms of energy has to be learned by the redistribution matrix. More specifically, the off-diagonal elements denote the fraction of energy that is converted from one form to the other. In contrast, the diagonal elements of the redistribution matrix denote the fraction of energy that is not converted.
In Fig. B.3 we visualize the off-diagonal elements of the redistribution matrix (i.e., the conversion of energy) for the pendulum task without friction, well the modeled potential and kinetic energy. We can see that an increasing fraction of energy is converted into the other form, until the total energy of the system is stored as either kinetic or potential energy. As soon as the total energy e.g. converted into kinetic energy, the corresponding off-diagonal element (the orange line of the upper plot in Fig. B.3 drops to zero. Here, the other off-diagonal element (the blue line of the upper plot in Fig. [B.3) starts to increase, meaning that energy is converted back from kinetic into potential energy. Note that the differences in the maximum values of the off-diagonal elements is not important, since at this point the corresponding energy is already approximately zero
B.4 HYDROLOGY
Modeling river discharge from meteorological data (e.g., precipitation, temperature) is one of the most important tasks in hydrology, and is necessary for water resource management and risk mitigation related to flooding. Recently, Kratzert et al. (2019c 2020) established -STM-based models as state-of-the-an art in rainfall runoff modeling, outperforming traditional hydrological models by large margin against most metrics (including peak flows, which is critical for flood prediction). However, the hydrology community is still reluctant to adopt these methods (e.g. Beven 2020). A recent
23

---

Under review as a conference paper at ICLR 2021
Fraction of moving Energy
0.6
Flow: Epot Exin
0.5
Flow: Exin Epot
0.4
0.3
0.1
4
8
10
0.20
Evi Epot
0.15
Energy in
0.10
0.05
0.00
8
10
Time in
Figure B.3: Redistribution of energies in pendulum learned by MC-LSTM. The upper plot shows the fraction of energy that is redistributed between the two cells that model Eot and Ekin over time. The continuous redistribution of energy results in the two time series of potential and kinetic energy displayed in the lower plot.
workshop on 'Big Data and the Earth Sciences' Sellars (2018) reported that "[m]any participants who have worked in modeling physical-basee systems continue to raise caution about the lack of physical understanding of ML. methods that rely on data-driven approaches."
One of of the most basic principles in watershed modeling is mass conservation. Whether water is treated as a resource (e.g. droughts) or hazard (e.g. floods), a modeller must be sure that they are accounting for all of the water in catchment. Thus, most models conserve mass (Todini] 1988), and attempt to explicitly implement the most important physical processes. The downside of this 'model everything' strategy is that errors introduced for every real-world process that is nor implemented in a model, or implemented incorrectly. In contrast, MC-LSTM able to learn any necessary behavior that can be induced from the signal (like LSTM) while still conserving the overall water budget.
B.4.1 DETAILS ON THE DATASET
The data used in all hydrology related experiments is the publicly available Catchment Attributes and Meteorology for Large-sample Studies (CAMELS) dataset (Newman et al. 2014 Addor et al. (0175). CAMELS contains data for 671 basins and is curated by the US National Center for Atmospheric Research (NCAR). It contains only basins with relatively low anthropogenic influence (e.g.. dams and reservoirs) and basin sizes range from geo- and oco-climatologies, as described by Newman et al. (2015) and Addor et al. (2017a). Out of all 671 basins, we used 447 these are the basins for which simulations from all benchmark models are available (see Sec. B.4.4). To reiterate, we used benchmark hydrology models that were trained and tested by other groups with experience using these models, and were therefore limited to the 447 basis with results for all benchmark models. The spatial distribution of the 447 basins across the contiguous USA (CONUS) shown in Fig. B.4
25 000 km² The basins cover range of different
For each catchment, roughly 30 years of daily meteorological data from three different products exist (DayMet, Maurer, NLDAS). Each meteorological dataset consist of five different variables: daily cumulative precipitation. daily minimum and maximum temperature, average short-wave radiation and vapor pressure. We used the Maurer forcing data because this is the data product that was used by all benchmark models (see Sec. B.4.4). In addition meteorological data, CAMELS also
24

---

Under review as a conference paper at ICLR 2021
6.2
Figure B.4: Spatial distribution of the 447 catchments considered in this study. The color denotes the SuthlSut Efficiency of the MC-LSTM ensemble for each basin, where a value of means perfect predictions.
includes set of static catchment attributes derived from remote sensing or CONUS -iide available data products. The static catchment attributes can broadly be grouped into climatic, vegetation or hydrological indices, as well as soil and topological properties. In this study, we used the same 27 catchment attributes as Kratzert et al. (2019c). Target data were daily averaged streamflow observations originally from the USGS streamflow gauge network, which are also included in the CAMELS dataset.
Training, validation and test set. Following the calibration and test procedure of the benchmark hydrology models, trained on streamflow observations from October 1999 through 30 September 2008 and tested on observations from October 1989 to 30 September 1999. The remaining period October 1980 30 September 1989) was used validation period for hyperparameter tuning.
B.4.2 DETAILS ON THE TRAINING SETUP AND MC-LSTM HYPERPARAMETERS
The general model setup follows insights from previous studies (Kratzert et al. 2018 2019c b 2020), where LSTMs were used for the same task. We use sequences of 365 timesteps (days) of meteorological inputs to predict discharge at the last timestep of the sequence (sequence-to-n prediction). The mass input æ in this experiment was catchment averaged precipitation (mm/day) and the auxiliary inputs were the 4 remaining meteorological variables (min. and max. temperature, short-wave radiation and vapor pressure) as well as the 27 static catchment attributes, which are constant over time.
We tested variety of MC-LSTM model configurations and adaptions for this specific task, which are briefly described below:
1. Processing auxiliary inputs with LSTM: Instead of directly using the auxiliary inputs in the input gate (5..5), output gate (Eq. (6) and time-dependent mass redistribution (Eq. R). we first processed the auxiliary inputs with LSTM and then used the output of this LSTM as the auxiliary inputs. The idea was to add additional memory for the auxiliary inputs, since in its base form only mass can be stored in the cell states of MC-LSTM. This could be seen as a specific adaption for the rainfall runoff modeling application, since information about the weather today and in the past ought to be useful for controlling the gates and mass redistribution. Empirically however, we could not see any significant performance gain and therefore decided not the more complex version with an additional L.T.M.
25

---

Under review as a conference paper at ICLR 2021
2. Auxiliary output + regularization to account for evapotranspiration: Of all precipita- tion falling in a catchment, only part ends as discharge in the river. Large portions of precipitation are lost to the atmosphere in form of evaporation (from e.g. open water sur- faces) and transpiration (from e.g. plants and trees), and to groundwater. One approach account for this "mass loss" is the following: instead of summing over outgoing mass (Eq. F. we used a linear layer to connect the outgoing mass to two output neurons, One neuron was fitted against the observed discharge data, while the second was used to estimate water loss due to unobserved sinks. A regularization term was added to the loss function to account for this. This regularization term was computed as the difference between the sum of the outgoing mass from MC-LSTM and the sum over the two output neurons. This did work, and the timeseries of the second auxiliary output neuron gave interesting results (i.e. matching the expected behavior of the annual evapotranspiration cycle), however results were not significantly better compared to our final model setup, which is why we rejected this architectural change.
3. Explicit trash cell Another way to account for evapotranspiration that we tested is to allow the model to use one memory cell as explicit "trash cell". That is, instead of deriving the final model prediction as the sum over the entire outgoing mass vector, we only calculate the sum over all but e-g. one element (see Eq. [3). This simple modification allows the model to use e.g. the first memory cell to discard mass from the system, which is then ignored for the model prediction. We found that this modification improved performance, and thus integrated it into our final model setup.
4. Input/output scaling to account for input/output uncertainty: Both, input and output data in our applications inherit large uncertainties (Nearing et al. 2016), which is not ideal for mass-congg models (and likely one of the reasons why LSTM performs so well compared all other mass-conserving models). To account for that, we tried three different adaptions. First, we used small fully connected network to derive time-dependent scaling weights for the mass input, which we regularized to be close to one. Second, we used a linear layer with positive weights to map the outgoing mass to the final model prediction, where all weights were initialized to one and the bias to zero. Third, we combined both. Out of the three, the input scaling resulted in the best performing model, however the results were worse than not scaling.
5. Time-dependent redistribution matrix variants: For this experiment, time-dependent redistribution matrix is necessary, since the underlying real-world processes (such as snow melt and thus conversion from snow into e-g. soil moisture or surface runoff) are time- dependent. Since using the redistribution matrix as proposed in Eq. ijs memory-demanding especially for models with larger numbers of memory cells, we also tried to use a different method for this experiment. Here, we learned fixed matrix (as in Eq. 7 and only calculated two vectors for each timestep. The final redistribution matrix was then derived as the outer product of the two time-dependent vectors and the static matrix. This resulted in lower memory consumption, however the model performance deteriorated significantly, which could be hint towards the complexity required to learn the redistributing processes in this problem.
6. Activation function of the redistribution matrix: We tested several different activation functions for the redistribution matrix in this experiment. Among those were the normalized sigmoid function (that is used e.g. for the input gate), the softmax function (as in Eq, g and the normalized ReLU activation function (see Eq. T8). We could achieve the best results using the normalized ReLU variant and can only hypothesize the reason for that: In this application (rainfall-ruma rumoff modelling) there are several state processes that are strictly disconnected. One example is snow and groundwater: groundwater will never turn into snow and snow will never transform into groundwater (not directly at least, it will first need to percolate through upper soil layers). Using normalized sigmoids or softmax makes it numerically harder (or impossible) to not distributed at least some mass between every cell because activations can never be exactly zero. The normalized ReLU activation can do so, however, which might be the reason that it worked better in this case.
As an extension to the standard MC-LSTM model introduced in Eq. (5) to Eq. (8), we also used the mass input (precipitation) in all gates. The reason is the following: Different amounts of precipitations can lead to different processes. For example, low amounts of precipitation could be absorbed by the
26

---

Under review as a conference paper at ICLR 2021
soil and stored soil moisture, leading to effectively immediate discharge contribution. Large amounts of precipitation on the other hand, could lead to direct surface runoff, if the water cannot infiltrate the soil at the rate of the precipitation falling down. Therefore, it is crucial that the gates have access to the information contained in the precipitation input. The final model design used in all hydrology experiments is described by the following equations:
moot = R c² + 1 x!
(10)
c² (1 o') o miot
i
(11)
h' o' @ m
(12)
9 = h
(13)
1=2
with the gates being defined by
d-1
8t = o(W; a' +U.
Vi v''x' bb)
(14)
|ct-1||1
dl-1
of a(W. a' U. |c'11l1
Vo x² + bo)
(15)
ct11 |cl-i|h + V, xf + B.).
R' ReLU W, a. a' +U
(16)
where ō is the normalized logistic function and ReLU is the normalized rectified linear unit (ReLU) that we define in the following. The normalized logistic function defined of the input gate is defined by:
(ic)) r(ix) = Σn (is))
(17)
In this experiment, the activation function for the redistribution gate is the normalized ReLU function defined by:
max( Sk, 0)
ReLU(ss)
(18)
Σx max(sx; 0)*
where s is some input vector to the normalized ReLU function.
We manually tried different sets of hyperparameters, because large-scale automatic hyperparameter search was not feasible. Besides trying out all variants as described above, the main hyperparameter that we tuned for the final model was the number of memory cells. For other parameters, such as learning rate, mini-batch size, number of training epochs, we relied on previous work using LSTMs on the same dataset.
The final hyperparameters are a hidden size of 64 memory cells and a mini-batch size of 256. We used the Adam optimizer (Kingma & Jimmy/ 2015) with scheduled learning rate starting at 0.01 then lowering the learning rate after 20 epochs to 0.005 and after another 5 epochs to 0.001. We trained the model for total number of 30 epochs and used the weights of the last epoch for the final model evaluation. All weight matrices were initialized as (semi) orthogonal matrices (Saxe et al. 2014) and all bias terms with a constant value of zero. The only exception was the bias the output gate, which we initialized to -3, to keep the output gate closed at the beginning of the training.
B.4.3 DETAILS ON THE LSTM MODEL.
For LSTM, we largely relied on expertise from previous studies (Kratzert et al. 2018 2019c 2020). The only hyperparameter we adapted was the number of memory cells, since used fewer basins
27

---

Under review as a conference paper at ICLR 2021
Table B.6: Model robustness of MC-LSTM and LSTM results the 10 different random seeds. For all n = 10 models, we calculated the median performance for each metric and report the mean and standard deviation of the median values in this table.
MC NSE
B-NSE
FLV4
FHVe
MC-LSTM Single
-0.021#0.003 0.737=0.003 -0.035=0.005
0.706=10003
-38.7+3.2
-3.9 -10.+=1.0
LSTM Single
x
13.6#3.4
Mass conservation (MC).
desirable. h; Nash-- efficiency: (-oo, 1]. values closer
5 3-NSE decomposition: (-o0, oo), values closer to zero are desirable.
Bottom 30% low flow bias: (-o0, oo), values closer zero are desirable.
: Top 2% peak flow bias: (-oc, 00), values closer zero desirable.
(447) than in the previous studies (531). We found that LSTM with 128 memory cells, compared to the 256 used in previous studies, resulted in slightly better results. Apart from that, trained LSTMs with the same inputs and settings (sequence-to-one with a sequence length of 365) as described in the previous section for MC-LSTM. We used the standard LSTT implementation from the PyTorch package (Paszke et al. 2019), i.c., with forget gate (Gers et al., 2000). We manually initialized the bias of the forget gate be 3 in order to keep the forget gate open the beginning of the training.
B.4.4 DETAILS ON THE BENCHMARK MODELS
The benchmark models were first collected by Kratzert et al. (2019c). All models were configured, trained and run by several different research groups, most often the respective model developers themselves. This was done avoid any potential favor our models. All models used the same forcing data (Maurer) and the same time periods to train and test. The models can be classified in two groups:
Models trained for individual watersheds. These are SAC-SMA NNwmann et al. 201 VIC (Newman et al. 2017), three different model structures of FUSH mHM (Mizukami jtt al. 2019) and HBV V (Seibert et al.) 2018). For the HBV model, two different simulations exist: First, the ensemble average of 1000 untrained HBV models (lower benchmark) and second, the ensemble average of 100 trained HBV models (upper benchmarks). For details see (Seibert et al. 2018).
2. Models trained regionally. For hydrological models, regional training means that one parameter transfer model was trained, which estimates snstershed-ppecffe model parameters through globally trained model functions from e.g. soil maps other catchment attributes. For this setting, the benchmark dataset includes simulations of the VIC model (Mizukami et al. 2017) and mHM (Rakovec al. 2019.
B.4.5
DETAILED RESULTS.
Table B.6
provides results for MC-LSTM and LSTM averaged over the 10 model repetitions.
Table B.7
provides the complete benchmarking results.
Provided by Nans Addor personal communication
28

---

Under review as a conference paper at ICLR 2021
Table B.7: Full hydrology benchmark results. All values represent the median (25% and 75% percentile in sub- and superscript, respectively) over the 447 basins. For both MC-LSTM and LSTM, metrics were derived from the ensemble mean prediction over the model repetitions (same as reported in the main paper).
MC²
NSE
B-NSE
FLVd
FHVe
MC-LSTM Ensemble
0.7440.814
0.641
-0.0200.013
0.066
-24.731.1
94.4
-1.77 7.0
LSTM Enssemble
X
0.763
0.676
-0.034
0.000
0.833
36.350.7
0.4
-15.7 8.6
13.5
SAC-SMA
0.603
-0.066
37.468.1
31.9
-20.4
19:3
VIC (basin)
0.5518.641
0.416
-0.0189.032
00001
-74.828
271.8 18.909
-28.1
40.1
VIC (regional)
0.307 402
-0.0749.023
-6.55
38.3
0.166
73.1
64.6
mHM (basin)
0.6666.728 730
-0.040
0.102
11.405.1 64.0
-18.6 9.5
27.7
mHM (regional)
0.5270.610
0.391
-0.0390
-1169
36.870.9
32.6
400.2 23.8
HBV (lower)
0,4170.550
0.276
-0.0230.055
-01114
23,9610
-11.9 56.2
HBV (upper)
0.676
-0.012
00.08
18.3675
62.9
-18.5
8.5
27.8
FUSE (900)
0.639
0..031
-10.5
-18.9 9.9
0.59
0.100 100
91.8
27.8
FUSE (902)
0.650
-0.047
0.004
-68.2
-19,4 8.9
0.570
0 0999
230.9
27.9
FUSE (904)
0.622
0.705
-0.067
0.019
-67.6
35.7
-21.4
11.3
10.135
238.6
33.0
Mass conservation (MC).
Nush-Sutcliffe efficiency: (-oc, values closer
desirable. B-NSE decomposition: (-00, oo), values closer to zero desirable.
Bottom 30% low flow bias: (-o0, oc), values closer to zero are desirable.
Top 2% peak flow bias: (- oo). values closer
desirable.
29

---

Under review as a conference paper at ICLR 2021
C THEOREMS & PROOFS
Theorem (Conservation property). Let m = Ex of and m = Ex hg be, respectively, the mass the MC-LSTM storage and the outputs time T. At any timestep T, we have:
m m
Σ
mi
That is, the change of mass the cell states is the difference between input and output mass, accumulated over time.
Proof. The proof is by induction and we use Mhot R' ct- + x² from Eq.
For T = 0, we have m = m + Σo Ei=1 x² Ei m which is trivially true when using the convention that Σi= =0
Assuming that the statement holds for we show that must also hold for T + 1.
Starting from Eq. 3). the mass of the cell states at time T + 1 is given by:
K )1
m T+1
T+1 m
Σ
og
where moo, is the k-th entry of the result from Eq. first term can be simplified as follows:
2 (at timestep t). The sum over entries in the
K
K
K
C T
+ +gx T+1
m
tot
A=
K
K
K
Σ
+ x
T+1
is
j=1
mm/
+
TT
The final simplification is possible because R and are (left-)stochaste) The mass of the outputs can then be computed from Eq. (4):
T+1 ogmi m
m
Ok
R=1
Putting everything together, we find
K
m
T+1
mot.k
T+1
Σ
Ogm
T+1
k=
h=1
m
+ x
T+1
m
T+1
T
T
m
x
m
+
T+1
m
T+1
t=1
T+1
m
+
mi
t=1
By the principle of induction, conclude that mass is conserved, as specified in Eq. 9
Corollary 1. In each timestep т, the cell states of are bounded by the sum of mass inputs Ei= x² +mo that is |c% I Ei x" + mo Furthermore, if the series of mass inputs converges limy Σi= x² m then also the sum of cell states converges.
30

---

Under review as a conference paper at ICLR 2021
Proof. Since ck 2 0, x² and m 2 0 for all ls and t,
cf
c S m
c
S
x²
+
m
(19)
where we used Theorem 1. Convergence follows immediately through the comparison test.
D ON RANDOM MARKOV MATRICES.
When initializing an MC-LSTM model, the entries of the redistribution matrix R of dimension K K are created from non-negative and iid random variables (si))s with finite means m and variances o² and bounded fourth moments. We collect them in matrix S. Next we assume that those entries get column-normalized to obtain the random Markov matrix R.
Properties of Markov matrices and random Markov matrices. Let At...., AK be the eigen- values and ......, K be the singular values of R, ordered such that |A1l 2 ... |Ax| and $1 2 ... 2 Sk- We then have the following properties for any Markov matrix (not necessarily random):
L.
1 I 1.
81 || S VK.
Furthermore, for random Markov matrices, we have
lim 81 (Bordenave al. 2012 Theorem 1.2)
For the reader's convenience briefly discuss further selected interesting properties of random Markov matrices in the next paragraph, especially concerning the global behavior of their eigenvalues and singular values.
Circular and Quartercircular law for random Markov matrices. In random matrix theory one major field interest concerns the behavior of eigenvalues and singular values when - cc. One would like to find out how the limiting distribution of the eigenvalues or singular values looks like. To discuss the most important results in this direction for large Markov matrices R, let us introduce some notation.
5n denotes the Dirac delta measure centered at a.
By pn = # Ek Ox, we denote the empirical spectral density of the eigenvalues of R.
Similarly we define the empirical spectral density of the singular values of R as: VR
> 8.y.
C, denotes the quartercireular distribution on the interval [0, and
Ug the uniform distribution on the disk {2 C: <o.
Then have K + oct
Quarter cirular law theorem: (Bordenave al. 2012 Theorem 1.1): VVER + Q. almost surely.
Cirular law theorem: [Bordenave et al. 2012
Theorem 1.3): V/KR + U, almost surely.
The convergence here understood in the sense of weak convergence of probability measures with respect to bounded continuous functions. Note that those two famous theorems originally appeared for S instead of / VKB R. Of course much more details on those results can be found in Bordenave
al. 2012)
31

---

Under review as a conference paper at ICLR 2021
Gradient flow of MC-LSTM for random redistributions. Here provide short note the gradient dynamics of the cell state random MC-LSTM, hence, at initialization of the model. Specifically want to provide some heuristics based the arguments about the behavior of large stochastic matrices. Let us start by recalling the formula for ch:
c' = (1 - o') о (R' cd x²).
(20)
Now investigate the gradient of of and i' for all Thus we approximately have:
de
0c' ll2 in the limit K + co. We assume that for K - oc,
Oc
det 2l2 Re |R'112.
(21)
R' is stochastic matrix, and $1 = || 2 its largest singular value. Theorem 1.2 from Bordenave
et al. (2012) ensures that | 2 for K + under reasonable moment assumptions the
distribution of the unnormalized entries (see above). Thus we are able conclude act lll s 1 for large K and all 4, which can prevent the gradients from exploding.
0c'
32


Reviews:
[{"review_id": "s18NnymVhw", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\n\nThe authors proposed a new family of LSTMs (i.e. MC-LSTMS) which can be shown to have a mass conservation property for the LSTM memory cells. They have shown in various applications that these models lead to on par or better performance compared to state of art approaches. However, after examining the paper, I am not yet fully convinced that 1) a general unified MC-LSTMs can achieve good performance in all the mentioned cases 2) the conserved mass is interpretable (and corresponds to the problem’s invariant, conserved quantity). I have listed those concerns in the question sections as authors might have precise answers. \n\n#### Pros\n\nThe authors in this paper have proposed a new family of LSTMs (i.e. MC-LSTMs) which can be shown to have a conservation property for the mass held by the LSTM memory cells. The authors have applied their models in three different prediction tasks involving quantity conservation, namingly summation(and subtraction), traffic prediction and rainfall runoff modelling. The proposed models (with some variants) achieve better or at least on par with current state of art models.\n\nThese different applications illustrate the usefulness of the models; the authors also reference appropriately to current state of art approaches, helping reviewers to well situate the paper’s contribution.\n\n#### Questions:\n\nWell I understand there are differences for each application (motivating to choose different neural architectures), it would be very interesting to know the performance of one quite general architecture (compared to say standard LSTM implemented in torch). In the paper, we have seen in the experiments:\n- time - independent R^t used in arithmetics\n- time - dependent R^t used in hydrology experiments\n- Hypernetwork based R^t used in pendulum experiments\nAnd also they vary on the auxiliary/mass inputs choices. \nWhat would be the performance if the authors use a quite general architecture (for example the second setting) for all the experiments please?\n\nJust for confirmation, in the experiments, the LSTMs involve forgetting gate right (contrary to what described in eq (1))\n\nI don’t find explanations for the r value for the hydrology experiments. Meanwhile the r value for the traditional r value is different from what is chosen for the MC-LSTM. Why is this please?\n\n#### Minor issues:\n\nThere seems some formatting issues at the end of “Mechanisms beyond storing are required for real-world applications” in Introduction. \n\nIn related work, Hamiltonian approaches (Greydanus et al. 2019) don’t seem to assume access knowledge to time derivatives w.r.t. Inputs. Rather, it favours the conservation parametrised by the Hamiltonian.  \n\n#### Minor suggestions:\n\nIn abstract, “expressed through continuity equations”, these aspects don’t seem to be addressed by MC-LSTMs, so I propose to not include this phrase in the abstract. \n\nI suggest adding some citations In the introduction where the authors said “LSTM to excel at speech, text, and language tasks” http://nlpprogress.com/. \n\nThe authors mention in the abstract and introduction about the interpretability but only show the mass interpretation in the hydrology experiments. For which I would suggest the authors show it in at least one other experiments (e.g. traffic) to 1) be more convincing 2) highlight this property (which may reveal to be very beneficial for production systems)\n\nI would be good to mention forgetting gate around equation (1) as those are used for later experiments in the paper. \n\n#### Thank you for the authors to having answered my questions and having addressed all my comments. My main concern was about the generalisation of the proposed architecture and the results at the current revision are convincing to me. I believe this direction of the research together with the approach taken make a nice contribution to the conference. In consequence, I have raised my score from 5 to 7.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qab9cJgkvZa", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper the authors propose a novel architecture, called Mass-Conserving LSTM (MC-LSTM) based on LSTM. The authors base their work over the hypothesis that the real world is based over conservation laws related to mass, energy, etc. Thus, they propose that also the quantities involved in deep learning models should be conserved. To do so, they aim at exploiting the memory cells of the LSTM as mass accumulators and then force the conservation laws via the model equations. The authors finally show successfully the potential of this novel network into three experimental settings where several types of “conservation” are required (e.g. mass conservation, energy conservation, etc).\n\nPros:\n+ they deal with a problem which can arise in non-laboratory scenarios in a novel way. Moreover, their results show that their method is great at dealing with situations where conservation is required, which may be particularly important for real-world scenarios\n+ the paper explains in-depth all the decisions made, and it is well written. Moreover, I found really interesting how they deal with the related work and special cases. It shows a really in-depth understanding of up-to-date literature in the field. \n\nCons:\n- (minor) the authors focus their experimental section to settings where mass (or energy, etc) conservation is required. It would be interesting to see how it performs also in settings where it is not required as well, thus showing whether this method also generalizes to different settings. \n\n------\n\nAuthors' response addressed properly my request.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nb_DnMAQGgu", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper provides an interesting and novel LSTM structure named MC-LSTM, which extends the inductive bias of LSTM to deal with some real-world problems limited by conservation laws. The authors do some experiments related to traffic forecasting and hydrology to illustrate the effectiveness of MC-LSTM.  The new architecture is well-suited for predicting some physical systems, which is valuable.\n##########################################################################\n\nReasons for score: \n\nOverall, I vote for accepting. \nI deem that the novel architecture based on LSTM that conserves quantites is useful and interesting.\nMy major concern is about some explanation about definitions and some additional ablation models (see cons below). \nHopefully the authors can address my concern in the rebuttal period. \n\n##########################################################################Pros: \n\n1. The paper takes one of the most important issue of some real-world systems:  conservation laws, which is important and\nshould be expressed through LSTM units.\n\n2.  This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed LSTM. \nThe entire structure is organized well and the formulas are very detailed.\n\n##########################################################################\n\nCons: \n\n1. In Basic gating, the formula about input, why do you use softmax operator? Because in basic LSTM, there is sigmoid.\n It would be better to provide more details about it.\n\n2. What are the advantages of MC-LSTM in terms of speed and resource consumption?\nIt would be more convincing if the authors can provide more cases in the rebuttal period. \n\n##########################################################################\n\nQuestions during rebuttal period: \n\nPlease address and clarify the cons above \n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "He8rYglDaY7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "<Summary>\n1. Many real-world systems follow conservation laws.\n2. This paper incorporates conservation laws into the RNN as an inductive bias.\n3. This paper proposes MC-LSTM and shows that MC-LSTM follows conservation laws empirically and theoretically.\n4. MC-LSTM utilizes a positive left-stochastic matrix to redistribute mass.\n5. This paper validates the MC-LSTM on arithmetic tasks, traffic forecasting, pendulum, rainfall tasks.\n\n<Strengths>\n1. Incorporating mass conservation laws into the neural network is important.\n2. This paper proposes MC-LSTM, a simple and effective mass-conserving model.\n3. The motivation of the research and the proposed methods are straightforward.\n4. Related works section provides a comparison between MC-LSTM and others such as the Markov chain, and it is also very interesting.\n\n<Weaknesses>\n1. This paper shows extensive experimental results, but there are less qualitative results. \n2. This paper missing some important related works, such as a physics-guided recurrent neural network model (PGRNN)\n\n<Questions and Additional Feedback>\n1. Is there empirical (or theoretical) analysis for long-term gradient vanishing?\n2. I wonder how MC-LSTM actually works. Is there qualitative analysis for a, i, o, and R?\n3. What is the reason that Eq. (5) and Eq. (6) utilize L1 norm?\n4. Does MC-LSTM can be extended to represent (2*mass) or (mass^2)-conserving properties?\n5. MC-Transformer will be one of a good extension.\n\n<Missing Reference>\n1. Lagrangian Neural Networks\n2. Physics-Guided Machine Learning for Scientific Discovery: An Application in Simulating Lake Temperature Profiles\n3. Discovering physical concepts with neural networks\n\n<Typos>\n1. One of the greatest success stories of deep learning are => One of the greatest success stories of deep learning is\n2. The former represent => The former represents\n\n<After Rebuttal>\nThank you for your detailed response.\nI will keep my positive score because my concerns are resolved partially.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a variant of recurrent neural networks based on Long Short-Term Memory. Unlike the standard LSTM, the proposed mass-conserving LSTM subtracts the output hidden state of the LSTM from its current cell state, thus preserving the "mass" stored in the cell states at each step. A left-stochastic recurrent weight matrix is also used to conserve the "mass" across the time steps. Empirical experiments demonstrated the effectiveness of the proposed MC-LSTM on a range of datasets such as addition & arithmetic tasks, traffic forecast, and rainfall modeling models. 

Several issues were clarified during the rebuttal period in a way that satisfied the reviewers. However, some concerns still remain unanswered:

1) This is an empirical paper that proposes a modified LSTM that brings forward a few different ideas: L1 norm, stochastic transition matrices, and subtracting the output hidden states. An ablation study is a MUST in such an applied work. It has been pointed out by other reviewers that there are many prior references on LSTMs variants. It would greatly strengthen the paper by considering more diverse baselines. There is no experiment nor discussion on how much each modification helps wrt the final accuracy. Thus it remains unclear how the results can generalize to other problems.

2) Although the results seem convincing across various datasets that mass conservation seems to help, the datasets are non-standard benchmarks in the machine learning conferences thus there is a lack of competitive prior baselines. As the proposed LSTM has a different number of parameters compared to the standard LSTM, is it fair to compare the different architectures under the same number of neurons? What happens if we compare the architectures with the same number of parameters? And how well does the model scale as we vary the hidden size? It would be helpful to keep the contributions into perspective by using standard RNN benchmark datasets such as Penn TreeBank or Wiki-8.

Overall, the basic idea seems interesting, but the lack of ablation studies significantly hurt the contribution and the positioning of the paper. Given the current submission, the paper needs further development, and non-trivial modifications, to be broadly appreciated by the machine learning community.


Decision: Reject