MC-LSTM: Mass-conserving LSTM

1 INTRODUCTION . Inductive biases enabled the success of CNNs and LSTMs . One of the greatest success stories of deep learning is Convolutional Neural Networks ( CNNs ) ( Fukushima , 1980 ; LeCun & Bengio , 1998 ; Schmidhuber , 2015 ; LeCun et al. , 2015 ) whose proficiency can be attributed to their strong inductive bias towards visual tasks ( Cohen & Shashua , 2017 ; Gaier & Ha , 2019 ) . The effect of this inductive bias has been demonstrated by CNNs that solve vision-related tasks with random weights , meaning without learning ( He et al. , 2016 ; Gaier & Ha , 2019 ; Ulyanov et al. , 2020 ) . Another success story is Long Short-Term Memory ( LSTM ) ( Hochreiter , 1991 ; Hochreiter & Schmidhuber , 1997 ) , which has a strong inductive bias toward storing information through its memory cells . This inductive bias allows LSTM to excel at speech , text , and language tasks ( Sutskever et al. , 2014 ; Bohnet et al. , 2018 ; Kochkina et al. , 2017 ; Liu & Guo , 2019 ) , as well as timeseries prediction . Even with random weights and only a learned linear output layer LSTM is better at predicting timeseries than reservoir methods ( Schmidhuber et al. , 2007 ) . In a seminal paper on biases in machine learning , Mitchell ( 1980 ) stated that “ biases and initial knowledge are at the heart of the ability to generalize beyond observed data ” . Therefore , choosing an appropriate architecture and inductive bias for deep neural networks is key to generalization . Mechanisms beyond storing are required for real-world applications . While LSTM can store information over time , real-world applications require mechanisms that go beyond storing . Many real-world systems are governed by conservation laws related to mass , energy , momentum , charge , or particle counts , which are often expressed through continuity equations . In physical systems , different types of energies , mass or particles have to be conserved ( Evans & Hanney , 2005 ; Rabitz et al. , 1999 ; van der Schaft et al. , 1996 ) , in hydrology it is the amount of water ( Freeze & Harlan , 1969 ; Beven , 2011 ) , in traffic and transportation the number of vehicles ( Vanajakshi & Rilett , 2004 ; Xiao & Duan , 2020 ; Zhao et al. , 2017 ) , and in logistics the amount of goods , money or products . A real-world task could be to predict outgoing goods from a warehouse based on a general state of the warehouse , i.e. , how many goods are in storage , and incoming supplies . If the predictions are not precise , then they do not lead to an optimal control of the production process . For modeling such systems , certain inputs must be conserved but also redistributed across storage locations within the system . We will All code to reproduce the results will be made available on GitHub . refer to conserved inputs as mass , but note that this can be any type of conserved quantity . We argue that for modeling such systems , specialized mechanisms should be used to represent locations & whereabouts , objects , or storage & placing locations and thus enable conservation . Conservation laws should pervade machine learning models in the physical world . Since a large part of machine learning models are developed to be deployed in the real world , in which conservation laws are omnipresent rather than the exception , these models should adhere to them automatically and benefit from them . However , standard deep learning approaches struggle at conserving quantities across layers or timesteps ( Beucler et al. , 2019b ; Greydanus et al. , 2019 ; Song & Hopke , 1996 ; Yitian & Gu , 2003 ) , and often solve a task by exploiting spurious correlations ( Szegedy et al. , 2014 ; Lapuschkin et al. , 2019 ) . Thus , an inductive bias of deep learning approaches via mass conservation over time in an open system , where mass can be added and removed , could lead to a higher generalization performance than standard deep learning for the above-mentioned tasks . A mass-conserving LSTM . In this work , we introduce Mass-Conserving LSTM ( MC-LSTM ) , a variant of LSTM that enforces mass conservation by design . MC-LSTM is a recurrent neural network with an architecture inspired by the gating mechanism in LSTMs . MC-LSTM has a strong inductive bias to guarantee the conservation of mass . This conservation is implemented by means of left-stochastic matrices , which ensure the sum of the memory cells in the network represents the current mass in the system . These left-stochastic matrices also enforce the mass to be conserved through time . The MC-LSTM gates operate as control units on mass flux . Inputs are divided into a subset of mass inputs , which are propagated through time and are conserved , and a subset of auxiliary inputs , which serve as inputs to the gates for controlling mass fluxes . We demonstrate that MC-LSTMs excel at tasks where conservation of mass is required and that it is highly apt at solving real-world problems in the physical domain . Contributions . We propose a novel neural network architecture based on LSTM that conserves quantities , such as mass , energy , or count , of a specified set of inputs . We show properties of this novel architecture , called MC-LSTM , and demonstrate that these properties render it a powerful neural arithmetic unit . Further , we show its applicability in real-world areas of traffic forecasting and modeling the pendulum . In hydrology , large-scale benchmark experiments reveal that MC-LSTM has powerful predictive quality and can supply interpretable representations . 2 MASS-CONSERVING LSTM . The original LSTM introduced memory cells to Recurrent Neural Networks ( RNNs ) , which alleviate the vanishing gradient problem ( Hochreiter , 1991 ) . This is achieved by means of a fixed recurrent self-connection of the memory cells . If we denote the values in the memory cells at time t by ct , this recurrence can be formulated as ct = ct−1 + f ( xt , ht−1 ) , ( 1 ) where x and h are , respectively , the forward inputs and recurrent inputs , and f is some function that computes the increment for the memory cells . Here , we used the original formulation of LSTM without forget gate ( Hochreiter & Schmidhuber , 1997 ) , but in all experiments we also consider LSTM with forget gate ( Gers et al. , 2000 ) . MC-LSTMs modify this recurrence to guarantee the conservation of the mass input.The key idea is to use the memory cells from LSTMs as mass accumulators , or mass storage . The conservation law is implemented by three architectural changes . First , the increment , computed by f in Eq . ( 1 ) , has to distribute mass from inputs into accumulators . Second , the mass that leaves MC-LSTM must also disappear from the accumulators . Third , mass has to be redistributed between mass accumulators . These changes mean that all gates explicitly represent mass fluxes . Since , in general , not all inputs must be conserved , we distinguish between mass inputs , x , and auxiliary inputs , a . The former represents the quantity to be conserved and will fill the mass accumulators in MC-LSTM . The auxiliary inputs are used to control the gates . To keep the notation uncluttered , and without loss of generality , we use a single mass input at each timestep , xt , to introduce the architecture . The forward pass of MC-LSTM at timestep t can be specified as follows : mttot = R t · ct−1 + it · xt ( 2 ) ct = ( 1− ot ) mttot ( 3 ) ht = ot mttot . ( 4 ) where it and ot are the input- and output gates , respectively , andR is a positive left-stochastic matrix , i.e. , 1T ·R = 1 , for redistributing mass in the accumulators . The total mass mtot is the redistributed mass , Rt · ct−1 , plus the mass influx , or new mass , it · xt . The current mass in the system is stored in ct . Note the differences between Eq . ( 1 ) and Eq . ( 3 ) . First , the increment of the memory cells no longer depends on ht . Instead , mass inputs are distributed by means of the normalized i ( see Eq . 5 ) . Furthermore , Rt replaces the implicit identity matrix of LSTM to redistribute mass among memory cells . Finally , Eq . ( 3 ) introduces 1 − ot as a forget gate on the total mass , mtot . Together with Eq . ( 4 ) , this assures that no outgoing mass is stored in the accumulators . This formulation has some similarity to Gated Recurrent Units ( GRU ) ( Cho et al. , 2014 ) , however the gates are not used for mixing the old and new cell state , but for splitting off the output . Basic gating and redistribution . The MC-LSTM gates at timestep t are computed as follows : it = softmax ( W i · at +U i · ct−1 ‖ct−1‖1 + bi ) ( 5 ) ot = σ ( W o · at +Uo · ct−1 ‖ct−1‖1 + bo ) ( 6 ) Rt = softmax ( Br ) , ( 7 ) where the softmax operator is applied column-wise , σ is the logistic sigmoid function , andW i , bi , W o , bo , and Br are learnable model parameters . Note that for the input gate and redistribution matrix , the requirement is that they are column normalized . This can also be achieved by other means than using the softmax function . For example , an alternative way to ensure a column-normalized matrixRt is to use a normalized logistic , σ̃ ( rkj ) = σ ( rkj ) ∑ n σ ( rkn ) . Also note that MC-LSTMs compute the gates from the memory cells , directly . This is in contrast with the original LSTM , which uses the activations from the previous time step . The accumulated values from the memory cells , ct , are normalized to counter saturation of the sigmoids and to supply probability vectors that represent the current distribution of the mass across cell states We use this variation e.g . in our experiments with neural arithmetics ( see Sec . 5.1 ) . Time-dependent redistribution . It can also be useful to predict a redistribution matrix for each sample and timestep , similar to how the gates are computed : Rt = softmax ( Wr · at + Ur · ct−1 ‖ct−1‖1 +Br ) , ( 8 ) where the parameters Wr and Ur are weight tensors and their multiplications result inK×K matrices . Again , the softmax function is applied column-wise . This version collapses to a time-independent redistribution matrix if Wr and Ur are equal to 0 . Thus , there exists the option to initialize Wr and Ur with weights that are small in absolute value compared to the weights ofBr , to favour learning time-independent redistribution matrices . We use this variant in the hydrology experiments ( see Sec . 5.4 ) . Redistribution via a hypernetwork . Even more general , a hypernetwork ( Schmidhuber , 1992 ; Ha et al. , 2017 ) that we denote with g can be used to procure R. The hypernetwork has to produce a column-normalized , square matrix Rt = g ( a0 , . . . , at , c0 , . . . , ct−1 ) . Notably , a hypernetwork can be used to design an autoregressive version of MC-LSTMs , if the network additionally predicts auxiliary inputs for the next time step . We use this variant in the pendulum experiments ( see Sec . 5.3 ) .

Summary: In this paper the authors propose a novel architecture, called Mass-Conserving LSTM (MC-LSTM) based on LSTM. The authors base their work over the hypothesis that the real world is based over conservation laws related to mass, energy, etc. Thus, they propose that also the quantities involved in deep learning models should be conserved. To do so, they aim at exploiting the memory cells of the LSTM as mass accumulators and then force the conservation laws via the model equations. The authors finally show successfully the potential of this novel network into three experimental settings where several types of “conservation” are required (e.g. mass conservation, energy conservation, etc).