Investigation of Latent Time-Scales in Neural ODE Surrogate Models

Authors: ashish s nair
shivam  barwey
pinaki  pal
jonathan f macart
romit  maulik
Venue: ICLR_cc_2024_Workshop_AI4DiffEqtnsInSci
Published: 2024-02-09 02:24:37

Abstract: This work explores autoencoder-based neural ordinary differential equation (neural ODE) surrogate models for advection-dominated dynamical systems. Alongside predictive demonstrations, physical insight into the sources of model acceleration (i.e., how the neural ODE achieves its acceleration) is the scope of the current study. Such investigations are performed by quantifying the effect of neural ODE components on latent system time-scales using eigenvalue analysis of dynamical system Jacobians. This work uncovers the key role played by the training trajectory length on the latent system time-scales: larger trajectory lengths correlate with an increase in limiting neural ODE time-scales, and optimal neural ODEs are found to recover the largest time-scales of the full-order (ground-truth) system. Demonstration studies are performed using datasets sourced from numerical solutions of the Kuramoto-Sivashinsky equation and hydrogen-air channel detonations (compressible reacting Navier-Stokes equations).

Main paper: 
 introduction: 
 Since its introduction in Chen et al. (2018) , the neural ODE strategy has cemented itself as a powerful scientific machine learning tool to model the evolution of dynamical systems using neural networks, and has found a vast number of applications across several fields (Finlay et al., 2020; Lee & Parish, 2020; Owoyele & Pal, 2022; Kumar et al., 2023; Poli et al., 2019) . Instead of directly enforcing discrete temporal representations (e.g., as used in residual networks described in Jiang et al. (2021) , or recurrent neural networks described in Reddy et al. ( 2019)), neural ODEs learn a continuous representation of nonlinear system dynamics using discretely sampled trajectory data. The critical advantage is that the instantaneous right-hand-side is modeled as a nonlinear function via a neural network, allowing the framework to leverage (a) existing time-integration schemes to execute the forecasting step, and (b) adjoint methods to enable memory-efficient backpropagation.When equipped with autoencoder-provided latent spaces, the neural ODE conveniently outputs a functional form for the instantaneous rate-of-change of low-dimensional yet expressive latent variables, a highly useful asset for reduced-order modeling (ROM), and in turn, real-time physics simulation applications. As a result, the combined autoencoder-based neural ODE strategy has been used to develop accelerated surrogate models in a variety of scenarios (Dutta et al., 2021; Linot et al., 2023) .Of particular interest to this work is the application of neural ODE ROMs to advection-dominated systems (i.e., systems in which the governing equations are heavily influenced by advection terms) that typically describe turbulent or shock-containing fluid flows over aircrafts and in power/propulsion devices (Linot et al., 2023) . If available, such ROMs could drastically accelerate simulation-augmented design loops for deploying resilient next-generation aerospace concepts (Brenner et al., 2019; Raman & Hassanaly, 2019) . To this end, recent work has observed the effect of smoothed latent trajectories for advection-dominated systems produced by reduced neural ODE simulations, pointing to a relationship between model acceleration and intrinsic time-scale elimination provided by the latent space (Wan et al., 2023) . Similar trends have been shown for neural ODE based surrogate models for stiff chemical kinetics (Vijayarangan et al., 2023) . Despite these advances, however, the source of acceleration in the overall modeling strategy remains unclear: the goal of achieving model accuracy from both forecasting and autoencoding perspectives often overshadows the need to identify the contribution of each of these components to the empirically observed model acceleration (i.e., the need to interpret the dynamical properties of the learned neural ODE). Quantitative insight into the sources of acceleration -particularly, the degree of time-scale elimination -provided by the neural ODE based ROM can lead to valuable physical and model-oriented insights, and is the scope of the current study.To understand and interpret sources of neural ODE acceleration, a more rigorous and quantitative analysis of time-scales produced by the neural ODE based ROM is warranted. This is accomplished here through a direct study of the effect of critical neural ODE training parameters -such as the overall integration time (training trajectory length) used to optimize models -on the accuracy and degree of time-scale elimination produced in the latent space. To execute this analysis, neural ODE Jacobians are used to directly quantify the fastest and slowest time-scales in the learned autoencodergenerated latent space. Ultimately, this study demonstrates the critical role of training trajectory length as a controlling parameter for time-scale elimination and model acceleration in latent spaces of advection-dominated systems. Demonstration studies are performed using ground-truth data from numerical simulations of the Kuramoto-Sivashinsky (KS) equation, and further tests are conducted for a hydrogen-air channel detonation configuration (Appendix A). The study concludes by demonstrating how these latent time-scales relate to the underlying physics of the full-order system. Specifically, for the KS equation, the largest (slowest) time-scales of the best performing neural ODE are found to recover the largest time-scales of the full-order system. 
methodology: 
 Dataset: The full-order system dataset comes from numerical solutions of the KS partial differential equation. The KS equation is solved using jax-cfd (Kochkov et al., 2021) , which generates ground-truth solution trajectories on a uniform one-dimensional spatial domain using a pseudospectral discretization.1 Time integration is performed with an explicit Euler method. Training data is generated from 25 distinct trajectories of 1500 instantaneous full-order solution snapshots separated by a fixed time-step of ∆t = 0.01. Each snapshot is 512-dimensional and resides in a one-dimensional physical space. During training, trajectories are broken into sub-trajectories of length specified by the training trajectory length n t . Trained models are then tested using new initial conditions. The reader is referred to the recent work of Wan et al. (2023) for details of the KSE formulation and initial condition generation.Neural ODE for Latent Dynamics: Ground-truth physical space trajectories are projected to a reduced latent space using an autoencoder. The neural ODE utilizes these latent trajectories to learn a continuous-time model of the latent dynamical system given by d w(t)EQUATIONwhere N is a neural network and w(t) represents the predicted latent variable. The parameters of the neural network are trained by minimizing the objective functionEQUATIONwhere the target latent variable w(t) ∈ R Nw is obtained by applying an encoder ϕ to the corresponding physical space snapshot u(t) ∈ R Nu , such that w(t) = ϕ(u(t)) and N w ≪ N u . The neural ODE formulation in Eq. 2 is advantageous as it is dynamics-informed, training the latent dynamical system to minimize error across all n t time steps. As such, the training trajectory length, n t , is a key parameter, selected a-priori, that influences the amount of dynamical information used to construct the latent model. Varying n t values in training is therefore expected to lead to neural ODEs with differing predictive and stability characteristics based on the intrinsic time-scales of the full-order system.Latent Time-Scales: Jacobian eigenvalue analysis is incorporated here to extract dynamical system time-scales (Maas & Pope, 1992; Mease et al., 2003; Valorani & Goussis, 2001) . Specifically, this analysis is employed to evaluate the impact of n t and neural network architecture hyperparameters on latent time-scales. Given Eq. 1, the fastest (limiting) and slowest time-scales in the latent space can be computed asEQUATIONrespectively. t lim is indicative of the largest feasible explicit time-step for evolution in the latent space, and t max is a measure of the slowest-evolving dynamical features (as discussed below, this work aims to correlate t max with neural ODE predictive accuracy). Additionally, since jax-cfd is used to solve the KS equation, differentiation through the full-order (physical space) dynamical system du/dt = F(u) is also possible, which facilitates Jacobian computation ∂F ∂u for extracting the corresponding true physical space time-scales in the same manner.The neural ODE architecture comprises a feed-forward neural network with four hidden layers (120 neurons in each hidden layer) utilizing ELU activation functions. The encoder architecture features a series of 1D convolutional layers with batch normalization and ELU activation function, designed to gradually reduce the spatial dimension of the physical space input. This down-sampling process halves the spatial component while doubling the channel count, leading to a flatten operation and a linear layer that creates the latent space representation w(t). The decoder mirrors the encoder design, substituting convolution layers with transpose convolution layers. Architectures are implemented in PyTorch (Paszke et al., 2019) , and torchdiffeq (Chen, 2018) is leveraged for neural ODE training routines. The ADAM optimizer was used for optimization with initial learning rate set to 10 -3 . Two distinct training methodologies to train the autoencoder and neural ODE were compared: a decoupled approach, where the autoencoder and neural ODE are trained separately, and a coupled approach, where the two are trained simultaneously in an end-to-end manner. For rollout predictions, the decoupled approach was found to give models with better predictive accuracy. To compare different models' predictive accuracy, mean squared (L RO ) and relative absolute (R RO ) rollout errors are used. L RO is the same as Eq. 2, but with different rollout n t values. R RO is similar, but measures the absolute difference between the ground truth and predicted values, normalized by the ground truth value, and is commonly used in the neural ODE literature. 
results: 
 Figure 1 (left) shows limiting time-scales t lim (t) in the latent space for autoencoders with different latent dimensions N w , convolutional layer counts, and training trajectory lengths (n t ) under both coupled and decoupled training methods. It can be seen that the number of convolutional layers and latent space dimensionality do not have a significant impact on the limiting time-scales. As for the effect of the training trajectory length n t , the figure demonstrates that increasing the training trajectory length increases the limiting time-scale in latent space for both decoupled and coupled training methods, resulting in smoother trajectories and larger permissible explicit time-steps. Specifically, in both coupled and decoupled training strategies, raising n t from 100 to 4000 markedly amplifies limiting time-scale in latent space by roughly two orders of magnitude. It is evident that among all the studied hyperparameters, the training trajectory length (n t ) is the sole parameter with a substantial impact on t lim .Figure 1 (e) and (f) compare, respectively, the ground truth spatio-temporal evolution of the KSE velocity field u with rollout predictions by the autoencoder neural ODE framework, which was trained using n t = 500 in a decoupled manner. Corresponding target (w) and predicted ( w) latent space trajectories for n t = 500 and 4000 are provided in Fig. 1(g ). It can be seen that the latent space trajectories predicted by the neural ODE with n t = 500 more closely matches the ground truth trajectories as compared to the neural ODE with n t = 4000. These noticeable differences in accuracy suggest that the relationship between increasing n t and predictive strength is not straightforward, as discussed later. Additionally, latent space trajectories are smoother than state space fields, with n t = 4000 trajectories being smoother than n t = 500, reflecting an increase in t lim with larger n t . < l a t e x i t s h a 1 _ b a s e 6 4 = " Z w K k a J c d U 3 h H l n h M y V D 4 t d z 5 5 Q M = " > A A A B 6 H i c b V D L T g J B E O z F F + I L 9 e h l I j H x R H a N r y P R i 0 d I 5 J H A h s w O D Y z M z m 5 m Z o 1 k w x d 4 8 a A x X v 0 k b / 6 N A + x B w U o 6 q V R 1 p 7 s r i A X X x n W / n d z K 6 t r 6 R n 6 z s L W 9 s 7 t X 3 D 9 o 6 C h R D O s s E p F q B V S j 4 B L r h h u B r V g h D Q O B z W B 0 O / W b j 6 g 0 j + S 9 G c f o h 3 Q g e Z 8 z a q x U e + o W S 2 7 Z n Y E s E y 8 j J c h Q 7 R a / O r 2 I J S F K w w T V u u 2 5 s f F T q g x n A i e F T q I x p m x E B 9 i 2 V N I Q t Z / O D p 2 Q E 6 v 0 S D 9 S t q Q h M / X 3 R E p D r c d h Y D t D a o Z 6 0 Z u K / 3 n t x P S v / Z T L O D E o 2 X x R P x H E R G T 6 N e l x h c y I s S W U K W 5 v J W x I F W X G Z l O w I X i L L y + T x l n Z u y x f 1 M 5 L l Z s s j j w c w T G c g g d X U I E 7 q E I d G C A 8 w y u 8 O Q / O i / P u f M x b c 0 4 2 c w h / 4 H z + A O l 5 j Q Y = < /Y 7 L H E Z r L 2 T G d r K + E O g = " > A A A B 7 n i c b V D L S g N B E O y N r x h f U Y 9 e B o P g K e y K r 2 P Q i 8 c I 5 g H J E m Y n s 8 m Q 2 d l l p l c J S z 7 C i w d F v P o 9 3 v w b J 8 k e N L G g o a j q p r s r S K Q w 6 L r f T m F l d W 1 9 o 7 h Z 2 t r e 2 d 0 r 7 x 8 0 T Z x q x h s s l r F u B 9 R w K R R v o E D J 2 4 n m N A o k b w W j 2 6 n f e u T a i F g 9 4 D j h f k Q H S o S C U b R S K + s G I X m a 9 M o V t + r O Q J a J l 5 M K 5 K j 3 y l / d f s z S i C t k k h r T 8 d w E / Y x q F E z y S a m b G p 5 Q N q I D 3 r F U 0 Y g b P 5 u d O y E n V u m T M N a 2 F J K Z + n s i o 5 E x 4 y i w n R H F o V n 0 p u J / X i f F 8 N r P h E p S 5 I r N F 4 W p J B i T 6 e + k L z R n K M e W U K a F v Z W w I d W U o U 2 o Z E P w F l 9 e J s 2 z q n d Z v b g / r 9 R u 8 j i K c A T H cA o e X E E N 7 q A O D W A w g m d 4 h T c n c V 6 c d + d j 3 l p w 8 p l D + A P n 8 w c w x Y 9 9 < / l a t e x i t > w < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 u 0 l + m 9 h A h s g s 4 + j I o 5 0 3 z u X L s 4= " > A A A C D H i c b V C 7 S g N B F J 3 1 b X x F L W 0 G g 2 A h Y V d 8 N Y J o Y 6 l g H p A N Y X b 2 b j J k d n a Z u a u G J R 9 g 4 6 / Y W C h i 6 w f Y + T d O Y g p N P D B w O O d c 7 t w T p F I Y d N 0 v Z 2 p 6 Z n Z u f m G x s L S 8 s r p W X N + o m i T T H C o 8 k Y m u B 8 y A F A o q K F B C P d X A 4 k B C L e h e D P z a L W g j E n W D v R S a M W s r E Q n O 0 E q t Y s m / E y G g k C H k f h D R u / 4 e 9 R H u M a d 9 q l p 4 e u i 6 N u W W 3 S H o J P F G p E R G u G o V P / 0 w 4 V k M C r l k x j Q 8 N 8 V m z j Q K L q F f 8 D M D K e N d 1 o a G p Y r F Y J r 5 8 J g + 3 b F K S K N E 2 6 e Q D t X f E z m L j e n F g U 3 G D D t m 3 B u I / 3 m N D K O T Z i 5 U m i E o / r M o y i T F h A 6 a o a H Q w F H 2 L G F c C / t X y j t M M 4 6 2 v 4 I t w R s / e Z J U 9 8 v e U f n w + q B 0 d j 6 q Y 4 F s k W 2 y S z x y T M 7 I J b k i F c L J A 3 k i L + T V e X S e n T f n / S c 6 5 Y x m N s k f O B / f c u 2 a l Q = = < / l a t e x i t > e w, nt = 500 < l a t e x i t s h a 1 _ b a s e 6 4 = " D L Z f 8 p e R W K I C c K f r Q m S U 2 + h + 6 K E = " > A A A C D X i c b V A 9 S w N B E N 3 z M 8 a v q K X N Y h Q s J N x J / G i E o I 1 l B K N C L o S 9 v b l k c W / v 2 J 0 z h i N / w M a / Y m O h i K 2 9 n f / G T U z h 1 4 O B x 3 s z z M w L U i k M u u 6 H M z E 5 N T 0 z W 5 g r z i 8 s L i 2 X V l Y v T J J p D g 2 e y E R f B c y A F A o a K F D C V a q B x Y G E y + D 6 Z O h f 3 o A 2 I l H n 2 E + h F b O O E p H g D K 3 U L m 3 6 P R E C C h l C 7 g c R 7 Q 1 2 q I 9 w i z k d U N X G o 6 r r u u 1 S 2 a 2 4 I 9 C / x B u T M h m j 3 i 6 9 + 2 H C s x g U c s m M a X p u i q 2 c a R R c w q D o Z w Z S x q 9 Z B 5 q W K h a D a e W j b w Z 0 y y o h j R J t S y E d q d 8 n c h Y b 0 4 8 D 2 x k z 7 J r f 3 l D 8 z 2 t m G B 2 2 c q H S D E H x r 0 V R J i k m d B g N D Y U G j r J v C e N a 2 F s p 7 z L N O N o A i z Y E 7 / f L f 8 n F b s X b r + y d V c u 1 4 3 E c B b J O N s g 2 8 c g B q Z FT U i c N w s k d e S B P 5 N m 5 d x 6 d F + f 1 q 3 X C G c + s k R 9 w 3 j 4 B 6 + + a z g = = < / l a t e x i t > e w, nt = 4000 (target)< l a t e x i t s h a 1 _ b a s e 6 4 = " d Z s L L d U + Q G I g 8 8 m w X P 0 / J O t k P R 4 = " > A A A B 9 X i c b V D L T s M w E H T K q 5 R X g S M X i w q J U 5 U g X s c K L h y L R B 9 S G y r H d V q r t h P Z G 6 C K 8 h 9 c O I A Q V / 6 F G 3 + D 2 + Y A L S O t N J r Z 1 e 5 O E A t u w H W / n c L S 8 s r q W n G 9 t L G 5 t b 1 T 3 t 1 r m i j R l D V o J C L d D o h h g i v W A A 6 C t W P N i A w E a w W j 6 4 n f e m D a 8 E j d w T h m v i Q D x U N O C V j p H n p p F 9 g T p I L L L O u V K 2 7 V n Q I v E i 8 n F Z S j 3 i t / d f s R T S R T Q A U x p u O 5 M f g p 0 c C p Y F m p m x g W E z o i A 9 a x V B H J j J 9 O r 8 7 w k V X 6 O I y 0 L Q V 4 q v 6 e S I k 0 Z i w D 2 y k J D M 2 8 N x H / 8 z o J h J d + y l W c A F N 0 t i h M B I Y I T y L A f a 4 Z B T G 2 h F D N 7 a 2 Y D o k m F G x Q J R u C N / / y I m m e V L 3 z 6 t n t a a V 2 l c d R R A f o E B 0 j D 1 2 g G r p B d d R A F G n 0 j F 7 R m / P o v D j v z s e s t e D k M / v o D 5 z P H 4 0 k k z Y = < / l a t e x i t > t lim < l a t e x i t s h a 1 _ b a s e 6 4 = " g 6 p / e u q 3 j Q g n Z i c G 2 q O W j J 0 x d A Q = " > A A A B 8 n i c b V D L S g N B E J z 1 G e M r 6 t H L Y B A 8 h V 3 x d Q x 6 8 R g h L 0 i W M D v p T Y b M z i 4 z v W J Y 8 h l e P C j i 1 a / x 5 t 8 4 S f a g i Q U N R V U 3 3 V 1 B I o V B 1 / 1 2 V l b X 1 j c 2 C 1 v F 7 Z 3 d v f 3 S w W H T x K n m 0 O C x j H U 7 Y A a k U N B A g R L a i Q Y W B R J a w e h u 6 r c e Q R s R q z q O E / A j N l A i F J y h l T p d h C f M 6 i K C S a 9 U d i v u D H S Z e D k p k x y 1 X u m r 2 4 9 5 G o F C L p k x H c 9 N 0 M + Y R s E l T I r d 1 E D C + I g N o G O p Y h E Y P 5 u d P K G n V u n T M N a 2 F N K Z + n s i Y 5 E x 4 y i w n R H D o V n 0 p u J / X i f F 8 M b P h E p S B M X n i 8 J U U o z p 9 H / a F x o 4 y r E l j G t h b 6 V 8 y D T j a F M q 2 h C 8 x Z e X S f O 8 4 l 1 V L h 8 u y t X b P I 4 C O S Y n 5 I x 4 5 J p U y T 2 p k Q b h J C b P 5 J W 8 O e i 8 O O / O x 7 x 1 x c l n j s g f O J 8 / y v i R m g = = < / l a t e x i t >< l a t e x i t s h a 1 _ b a s e 6 4 = " g 6 p / e u q 3 j Qg n Z i c G 2 q O W j J 0 x d A Q = " > A A A B 8 n i c b V D L S g N B E J z 1 G e M r 6 t H L Y B A 8 h V 3 x d Q x 6 8 R g h L 0 i W M D v p T Y b M z i 4 z v W J Y 8 h l e P C j i 1 a / x 5 t 8 4 S f a g i Q U N R V U 3 3 V 1 B I o V B 1 / 1 2 V l b X 1 j c 2 C 1 v F 7 Z 3 d v f 3 S w W H T x K n m 0 O C x j H U 7 Y A a k U N B A g R L a i Q Y W B R J a w e h u 6 r c e Q R s R q z q O E / A j N l A i F J y h l T p d h C f M 6 i K C S a 9 U d i v u D H S Z e D k p k x y 1 X u m r 2 4 9 5 G o F C L p k x H c 9 N 0 M + Y R s E l T I r d 1 E D C + I g N o G O p Y h E Y P 5 u d P K G n V u n T M N a 2 F N K Z + n s i Y 5 E x 4 y i w n R H D o V n 0 p u J / X i f F 8 M b P h E p S B M X n i 8 J U U o z p 9 H / a F x o 4 y r E l j G t h b 6 V 8 y D T j a F M q 2 h C 8 x Z e X S f O 8 4 l 1 V L h 8 u y t X b P I 4 C O S Y n 5 I x 4 5 J p U y T 2 p k Q b h J C b P 5 J W 8 O e i 8 O O / O x 7 x 1 x c l n j s g f O J 8 / y v i R m g = = < / l a t e x i t >< l a t e x i t s h a 1 _ b a s e 6 4 = " g 6 p / e u q 3 j Qg n Z i c G 2 q O W j J 0 x d A Q = " > A A A B 8 n i c b V D L S g N B E J z 1 G e M r 6 t H L Y B A 8 h V 3 x d Q x 6 8 R g h L 0 i W M D v p T Y b M z i 4 z v W J Y 8 h l e P C j i 1 a / x 5 t 8 4 S f a g i Q U N R V U 3 3 V 1 B I o V B 1 / 1 2 V l b X 1 j c 2 C 1 v F 7 Z 3 d v f 3 S w W H T x K n m 0 O C x j H U 7 Y A a k U N B A g R L a i Q Y W B R J a w e h u 6 r c e Q R s R q z q O E / A j N l A i F J y h l T p d h C f M 6 i K C S a 9 U d i v u D H S Z e D k p k x y 1 X u m r 2 4 9 5 G o F C L p k x H c 9 N 0 M + Y R s E l T I r d 1 E D C + I g N o G O p Y h E Y P 5 u d P K G n V u n T M N a 2 F N K Z + n s i Y 5 E x 4 y i w n R H D o V n 0 p u J / X i f F 8 M b P h E p S B M X n i 8 J U U o z p 9 H / a F x o 4 y r E l j G t h b 6 V 8 y D T j a F M q 2 h C 8 x Z e X S f O 8 4 l 1 V L h 8 u y t X b P I 4 C O S Y n 5 I x 4 5 J p U y T 2 p k Q b h J C b P 5 J W 8 O e i 8 O O / O x 7 x 1 x c l n j s g f O J 8 / y v i R m g = = < / l a t e x i t >< l a t e x i t s h a 1 _ b a s e 6 4 = " g 6 p / e u q 3 j Q number of autoencoder convolutional layers (with n t = 500 and latent dimension of 25), (c) n t using a decoupled training approach, and (d) n t using a coupled training approach. Latent dimension 25 and 4 convolution layers are used in (c) and (d). (Right) Neural ODE (latent dimension of 25 and 4 convolution layers) predictions for an unseen initial condition, where (e) shows ground-truth trajectory in physical space, (f) shows decoded rollout predictions in physical space for an n t = 500 model using a decoupled training approach, and (g) shows corresponding latent space trajectories (solid line is target latent variable w, dashed is predicted w for n t = 500, and dotted is predicted w for n t = 4000).g n Z i c G 2 q O W j J 0 x d A Q = " > A A A B 8 n i c b V D L S g N B E J z 1 G e M r 6 t H L Y B A 8 h V 3 x d Q x 6 8 R g h L 0 i W M D v p T Y b M z i 4 z v W J Y 8 h l e P C j i 1 a / x 5 t 8 4 S f a g i Q U N R V U 3 3 V 1 B I o V B 1 / 1 2 V l b X 1 j c 2 C 1 v F 7 Z 3 d v f 3 S w W H T x K n m 0 O C x j H U 7 Y A a k U N B A g R L a i Q Y W B R J a w e h u 6 r c e Q R s R q z q O E / A j N l A i F J y h l T p d h C f M 6 i K C S a 9 U d i v u D H S Z e D k p k x y 1 X u m r 2 4 9 5 G o F C L p k x H c 9 N 0 M + Y R s E l T I r d 1 E D C + I g N o G O p Y h E Y P 5 u d P K G n V u n T M N a 2 F N K Z + n s i Y 5 E x 4 y i w n R H D o V n 0 p u J / X i f F 8 M b P h E p S B M X n i 8 J U U o z p 9 H / a F x o 4 y r E l j G t h b 6 V 8 y D T j a F M q 2 h C 8 x Z e X S f O 8 4 l 1 V L h 8 u y t X b P I 4 C O S Y n 5 I x 4 5 J p U y T 2 p k Q b h J C b P 5 J W 8 O e i 8 O O / O x 7 x 1 x c l n j s g f O J 8 / y v i R m g = = < / l a t e x i t > Time (a) (b) (c) (d)Figure 2 displays the mean-squared (L RO ) and relative absolute (R RO ) errors for neural ODEs trained using varying n t values in a decoupled training approach, across a 500 time-step predicted rollout trajectory. It can be seen that the rollout losses initially rise with increasing n t , and reach a minimum at an optimal n t of 500, indicating this value best represents the system's physics for enhanced predictive accuracy at this tested rollout length. Although this may seem expected, as the optimal n t coincides with the trajectory length used to evaluate the error, identification of this 'optimal' n t is crucial for achieving maximum prediction precision. Figure 2 also shows t max (t) for neural ODEs with different training trajectory lengths n t . Interestingly, neural ODEs with n t = 8 and n t = 4000 have t max values diverging from the full system, while n t = 500 aligns closely, correlating with the lowest rollout errors. This suggests neural ODEs with the best accuracy have t max values matching the full system's slowest physics. Although determining the exact optimal n t a-priori is still an open problem, analyzing t max can guide the selection of effective n t values for maximizing predictive accuracy. 
conclusion: 
 This study explores the impact of training parameters on latent time-scales within neural ODE-based surrogate models. Key insights include the pivotal role of training trajectory length in influencing the latent system time-scales and model acceleration. The eigenvalue analysis method employed here proves effective in quantifying the fastest and slowest time-scales in the latent space, further highlighting the critical importance of training trajectory length (n t ). This study also underscores the significance of the optimal n t in neural ODE-based models, emphasizing its correlation with the largest eigenvalues of the system's dynamics. The optimal n t is crucial for achieving predictive accuracy, as it aligns the neural ODE's time-scales with the full system's slowest evolving physics. This correlation between n t and the largest eigenvalues offers a novel perspective in neural ODE training, guiding future research and practical applications in dynamical system modeling.  
 
 Appendix: To better evaluate how the time-scale trends in the autoencoder-neural ODE framework generalize beyond the KS equation, this section serves as an extension of the approach to unsteady detonations, a more realistic and highly advection-dominated benchmark problem. In broad terms, detonations are being actively explored as driving modes for combustion in next-generation power and propulsion concepts (Raman et al., 2023; Shepherd, 2009) . The dynamical complexity of the detonation wave front motivates development of surrogate models that accelerate the solution of respective governing equations, which are the compressible reacting Navier-Stokes equations. Considering the analysis of the KS equations in the main text, the goal of this section is to highlight the consistent relationship between training trajectory length (n t ) and limiting latent timescale (t lim ) across fundamentally different PDEs.Training data is generated using resolved simulations of the compressible reacting Navier-Stokes equations in one spatial dimension in a standard channel detonation configuration. The resulting dataset can be described by the unsteady, self-sustained propagation of a hydrogen-air detonation wave through the spatial domain, where a detonation is characterized as a shock-wave coupled with a strong chemical reaction zone (see Fig. 3 ). Data is generated using a solver based on UMReact-ingFlow (Bielawski et al., 2023) built on the AMReX framework (Zhang et al., 2019) , with chemistry modeled using the detailed chemical kinetic mechanism from Mueller et al. (1999) . For a complete description of the simulation configuration and dataset, the reader is directed to Nair et al. (2024) . In summary, two detonation trajectories are simulated to populate the dataset, each described by the pressure ratio between the driver and ambient gas, P d /P amb , with testing set evaluations performed on the same pressure ratios, but extrapolated in time. Each temporal snapshot captures fluid density, pressure, temperature, velocity, and species mass fractions at each spatial point (6000 total spatial points), giving an input channel depth of 13, unlike the unity depth in the KS equation. The snapshots were recorded at intervals of ∆t = 10 -8 s.The predictive accuracy of the autoencoder-based neural ODE, trained with a decoupled approach using n t = 250, latent dimension of 10, and 4 encoder convolutional layers, is assessed by extrapolating beyond the training data until the detonation exits the domain. Figure 3 shows predicted and target profiles for pressure, temperature, and mass fractions of two intermediate species (H 2 O 2 and H 2 O), indicating that the model effectively extrapolates beyond the training set for different pressure ratios and can properly represent the self-sustained detonation propagation.Figure 4 (left) shows the limiting time-scale, t lim , in the latent space versus time for neural ODEs trained with various n t values (using latent dimension of 10 and 4 encoder convolution layers).Although the configuration is significantly different here, Fig. 4 reveals that higher n t generally increases t lim in a similar manner as the KS equation trends shown in Fig. 1 . Although not shown here, the latent variable evolution -which captures the propagation of detonation wave front (a shock coupled with a cheimcal reaction zone) -was also found to exhibit smoothness for this problem, which has significant implications for acceleration of detonation-containing flow simulations. Interestingly, Fig. 4 (right) -which shows the rollout error of predicted trajectories (evaluated using 1250 rollout steps) as a function of n t -also indicates an optimal n t value akin to observations in Figure 2 for the KS equation. Although this optimal value is present, it should be noted that comparing the neural ODE limiting time-scales (t lim (t)) to their full-system counterparts in this configuration (i.e., producing an analog to Fig. 2 (right) for the detonation) was not possible, since this requires full-system Jacobians ( ∂F ∂u ) of the compressible reacting Navier-Stokes equations and the solver used here was not differentiable. However, the general consistency in trends with respect to the training trajectory length is promising, and motivates further neural ODE analysis for complex advection-dominated configurations in future work. In this work, the domain length is 64, and is discretized into 512 points. The KS viscosity is set to 0.01. 


Decision: The reviewers both agree that the topic is a good fit for this workshop. Both mention that the results are only empirical and limited by showing only 1 ODE. For a workshop paper,  I think that is ok as preliminary work. I vote for acceptance and recommend for a full submission that the authors test their method on more systems.

Reviews: review: title: title: The paper is well-written and structured but its generalizability remains limited.
main_review: main_review: The paper explores the impact of training parameters on latent time-scales within a neural ODE-based surrogate model of the Kuramoto-Sivashinsky equation. It also investigates the significance of the trajectory length and relates it to the largest eigenvalues of the system. 

Pros 
- The paper is well-structured, the plots are well-documented and good to understand. 
- The focus on trajectory lengths makes it easy to follow along. The results show the sensitivity of the accuracy of results to trajectory length. 

Quality: The quality of the paper is good. \
Clarity: The paper is well-structured. \
Originality: good. \
Significance of this work: The results presented are promising. However, to determine the significance and expected generalizability--given the title and abstract--a more elaborate benchmark would be needed. 


Major issues 
- Generalizability: As only one ODE is investigated, the generalizability of results is unclear. 
- One of the main results of the authors is the correlation between training length and the largest eigenvalue of the system dynamics/t_lim. This is also visualized and described in Figure 1c/d. However, it is not clear how exactly they are correlated. For example: t_lim is very similar for decoupled approach with trajectory length 1000, 100 and 200 but higher for 500, i.e. there is neither a clear positive nor negative correlation. An investigation of average t_lim values over time (x-axis) and more training trajectory lengths could help investigate this correlation better. 


Minor issues 
- The paper title might be misleading. The authors only investigate the KS equation. In the abstract (+introduction) the authors specify that the subject of the paper are "surrogate models for advection-dominated dynamical systems". Generalizability is unclear. 
- Why did the authors choose n_t= 500 and n_t=4000 for Figure 1g? 
- Is it n_t=7 oder n_t=8 in Figure 2 (right)? 
- Acceleration is not mentioned in the discussion.
title: title: Interesting but limited experiments
main_review: main_review: ### Summary
This work studies the relation between the (fastest/slowest) time-scales of a dynamical system and the time scales of an autoencoder-based neural ODE surrogate in its latent space. The authors show empirically for the Kuramoto-Sivashinsky equation that the autoencoder architecture (number of layers and latent dimension) does not impact significantly the time scales of the latent space, while the length of the training trajectories significantly correlates with the slowest/fastest time.

### Comments
* The subject is interesting and well suited to the workshop.
* The goal of the paper is clearly presented and well introduced, although a bit verbose.
* The results are interesting but very limited and mainly empirical. In particular a single dynamical system is used for the experiments. It is hard to know if the results carry for other systems. The architecture (and training) of the neural ODE could also have an impact, which is not considered.