MC-LSTM: Mass-conserving LSTM

1 INTRODUCTION . Inductive biases enabled the success of CNNs and LSTMs . One of the greatest success stories of deep learning is Convolutional Neural Networks ( CNNs ) ( Fukushima , 1980 ; LeCun & Bengio , 1998 ; Schmidhuber , 2015 ; LeCun et al. , 2015 ) whose proficiency can be attributed to their strong inductive bias towards visual tasks ( Cohen & Shashua , 2017 ; Gaier & Ha , 2019 ) . The effect of this inductive bias has been demonstrated by CNNs that solve vision-related tasks with random weights , meaning without learning ( He et al. , 2016 ; Gaier & Ha , 2019 ; Ulyanov et al. , 2020 ) . Another success story is Long Short-Term Memory ( LSTM ) ( Hochreiter , 1991 ; Hochreiter & Schmidhuber , 1997 ) , which has a strong inductive bias toward storing information through its memory cells . This inductive bias allows LSTM to excel at speech , text , and language tasks ( Sutskever et al. , 2014 ; Bohnet et al. , 2018 ; Kochkina et al. , 2017 ; Liu & Guo , 2019 ) , as well as timeseries prediction . Even with random weights and only a learned linear output layer LSTM is better at predicting timeseries than reservoir methods ( Schmidhuber et al. , 2007 ) . In a seminal paper on biases in machine learning , Mitchell ( 1980 ) stated that “ biases and initial knowledge are at the heart of the ability to generalize beyond observed data ” . Therefore , choosing an appropriate architecture and inductive bias for deep neural networks is key to generalization . Mechanisms beyond storing are required for real-world applications . While LSTM can store information over time , real-world applications require mechanisms that go beyond storing . Many real-world systems are governed by conservation laws related to mass , energy , momentum , charge , or particle counts , which are often expressed through continuity equations . In physical systems , different types of energies , mass or particles have to be conserved ( Evans & Hanney , 2005 ; Rabitz et al. , 1999 ; van der Schaft et al. , 1996 ) , in hydrology it is the amount of water ( Freeze & Harlan , 1969 ; Beven , 2011 ) , in traffic and transportation the number of vehicles ( Vanajakshi & Rilett , 2004 ; Xiao & Duan , 2020 ; Zhao et al. , 2017 ) , and in logistics the amount of goods , money or products . A real-world task could be to predict outgoing goods from a warehouse based on a general state of the warehouse , i.e. , how many goods are in storage , and incoming supplies . If the predictions are not precise , then they do not lead to an optimal control of the production process . For modeling such systems , certain inputs must be conserved but also redistributed across storage locations within the system . We will All code to reproduce the results will be made available on GitHub . refer to conserved inputs as mass , but note that this can be any type of conserved quantity . We argue that for modeling such systems , specialized mechanisms should be used to represent locations & whereabouts , objects , or storage & placing locations and thus enable conservation . Conservation laws should pervade machine learning models in the physical world . Since a large part of machine learning models are developed to be deployed in the real world , in which conservation laws are omnipresent rather than the exception , these models should adhere to them automatically and benefit from them . However , standard deep learning approaches struggle at conserving quantities across layers or timesteps ( Beucler et al. , 2019b ; Greydanus et al. , 2019 ; Song & Hopke , 1996 ; Yitian & Gu , 2003 ) , and often solve a task by exploiting spurious correlations ( Szegedy et al. , 2014 ; Lapuschkin et al. , 2019 ) . Thus , an inductive bias of deep learning approaches via mass conservation over time in an open system , where mass can be added and removed , could lead to a higher generalization performance than standard deep learning for the above-mentioned tasks . A mass-conserving LSTM . In this work , we introduce Mass-Conserving LSTM ( MC-LSTM ) , a variant of LSTM that enforces mass conservation by design . MC-LSTM is a recurrent neural network with an architecture inspired by the gating mechanism in LSTMs . MC-LSTM has a strong inductive bias to guarantee the conservation of mass . This conservation is implemented by means of left-stochastic matrices , which ensure the sum of the memory cells in the network represents the current mass in the system . These left-stochastic matrices also enforce the mass to be conserved through time . The MC-LSTM gates operate as control units on mass flux . Inputs are divided into a subset of mass inputs , which are propagated through time and are conserved , and a subset of auxiliary inputs , which serve as inputs to the gates for controlling mass fluxes . We demonstrate that MC-LSTMs excel at tasks where conservation of mass is required and that it is highly apt at solving real-world problems in the physical domain . Contributions . We propose a novel neural network architecture based on LSTM that conserves quantities , such as mass , energy , or count , of a specified set of inputs . We show properties of this novel architecture , called MC-LSTM , and demonstrate that these properties render it a powerful neural arithmetic unit . Further , we show its applicability in real-world areas of traffic forecasting and modeling the pendulum . In hydrology , large-scale benchmark experiments reveal that MC-LSTM has powerful predictive quality and can supply interpretable representations . 2 MASS-CONSERVING LSTM . The original LSTM introduced memory cells to Recurrent Neural Networks ( RNNs ) , which alleviate the vanishing gradient problem ( Hochreiter , 1991 ) . This is achieved by means of a fixed recurrent self-connection of the memory cells . If we denote the values in the memory cells at time t by ct , this recurrence can be formulated as ct = ct−1 + f ( xt , ht−1 ) , ( 1 ) where x and h are , respectively , the forward inputs and recurrent inputs , and f is some function that computes the increment for the memory cells . Here , we used the original formulation of LSTM without forget gate ( Hochreiter & Schmidhuber , 1997 ) , but in all experiments we also consider LSTM with forget gate ( Gers et al. , 2000 ) . MC-LSTMs modify this recurrence to guarantee the conservation of the mass input.The key idea is to use the memory cells from LSTMs as mass accumulators , or mass storage . The conservation law is implemented by three architectural changes . First , the increment , computed by f in Eq . ( 1 ) , has to distribute mass from inputs into accumulators . Second , the mass that leaves MC-LSTM must also disappear from the accumulators . Third , mass has to be redistributed between mass accumulators . These changes mean that all gates explicitly represent mass fluxes . Since , in general , not all inputs must be conserved , we distinguish between mass inputs , x , and auxiliary inputs , a . The former represents the quantity to be conserved and will fill the mass accumulators in MC-LSTM . The auxiliary inputs are used to control the gates . To keep the notation uncluttered , and without loss of generality , we use a single mass input at each timestep , xt , to introduce the architecture . The forward pass of MC-LSTM at timestep t can be specified as follows : mttot = R t · ct−1 + it · xt ( 2 ) ct = ( 1− ot ) mttot ( 3 ) ht = ot mttot . ( 4 ) where it and ot are the input- and output gates , respectively , andR is a positive left-stochastic matrix , i.e. , 1T ·R = 1 , for redistributing mass in the accumulators . The total mass mtot is the redistributed mass , Rt · ct−1 , plus the mass influx , or new mass , it · xt . The current mass in the system is stored in ct . Note the differences between Eq . ( 1 ) and Eq . ( 3 ) . First , the increment of the memory cells no longer depends on ht . Instead , mass inputs are distributed by means of the normalized i ( see Eq . 5 ) . Furthermore , Rt replaces the implicit identity matrix of LSTM to redistribute mass among memory cells . Finally , Eq . ( 3 ) introduces 1 − ot as a forget gate on the total mass , mtot . Together with Eq . ( 4 ) , this assures that no outgoing mass is stored in the accumulators . This formulation has some similarity to Gated Recurrent Units ( GRU ) ( Cho et al. , 2014 ) , however the gates are not used for mixing the old and new cell state , but for splitting off the output . Basic gating and redistribution . The MC-LSTM gates at timestep t are computed as follows : it = softmax ( W i · at +U i · ct−1 ‖ct−1‖1 + bi ) ( 5 ) ot = σ ( W o · at +Uo · ct−1 ‖ct−1‖1 + bo ) ( 6 ) Rt = softmax ( Br ) , ( 7 ) where the softmax operator is applied column-wise , σ is the logistic sigmoid function , andW i , bi , W o , bo , and Br are learnable model parameters . Note that for the input gate and redistribution matrix , the requirement is that they are column normalized . This can also be achieved by other means than using the softmax function . For example , an alternative way to ensure a column-normalized matrixRt is to use a normalized logistic , σ̃ ( rkj ) = σ ( rkj ) ∑ n σ ( rkn ) . Also note that MC-LSTMs compute the gates from the memory cells , directly . This is in contrast with the original LSTM , which uses the activations from the previous time step . The accumulated values from the memory cells , ct , are normalized to counter saturation of the sigmoids and to supply probability vectors that represent the current distribution of the mass across cell states We use this variation e.g . in our experiments with neural arithmetics ( see Sec . 5.1 ) . Time-dependent redistribution . It can also be useful to predict a redistribution matrix for each sample and timestep , similar to how the gates are computed : Rt = softmax ( Wr · at + Ur · ct−1 ‖ct−1‖1 +Br ) , ( 8 ) where the parameters Wr and Ur are weight tensors and their multiplications result inK×K matrices . Again , the softmax function is applied column-wise . This version collapses to a time-independent redistribution matrix if Wr and Ur are equal to 0 . Thus , there exists the option to initialize Wr and Ur with weights that are small in absolute value compared to the weights ofBr , to favour learning time-independent redistribution matrices . We use this variant in the hydrology experiments ( see Sec . 5.4 ) . Redistribution via a hypernetwork . Even more general , a hypernetwork ( Schmidhuber , 1992 ; Ha et al. , 2017 ) that we denote with g can be used to procure R. The hypernetwork has to produce a column-normalized , square matrix Rt = g ( a0 , . . . , at , c0 , . . . , ct−1 ) . Notably , a hypernetwork can be used to design an autoregressive version of MC-LSTMs , if the network additionally predicts auxiliary inputs for the next time step . We use this variant in the pendulum experiments ( see Sec . 5.3 ) .

Summary: The paper provides an interesting and novel LSTM structure named MC-LSTM, which extends the inductive bias of LSTM to deal with some real-world problems limited by conservation laws. The authors do some experiments related to traffic forecasting and hydrology to illustrate the effectiveness of MC-LSTM.  The new architecture is well-suited for predicting some physical systems, which is valuable.