SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Decoder. We introduce an MLP  $\Phi^d (\cdot)$  to predict both the position and velocity vectors using corresponding node representations, i.e.,  $[\hat{\pmb{p}}_i^s,\hat{\pmb{q}}_i^s] = \Phi^d (\pmb {v}_i^s)$ .
Dynamic Graph Updating. We can estimate the instant distance between nodes using the encoder and then construct the graphs, which could suffer from a large computational burden. To improve the efficiency of graph construction during ODE propagation, we not only update the graph structure every  $\Delta s$ , and but also introduce a graph updating strategy that calculates the distance between first-order and second-order neighbors in the last graph. By doing so, we can delete edges between first-order neighbors and add edges between second-order neighbors, reducing quadratic complexity to linear complexity in sparse graphs. We will also validate this empirically.
Learning Objective. Given the ground truth, we first minimize the mean squared error (MSE) of the predicted trajectory. Moreover, we require both node and context representations to be robust to noise attacks to improve the robustness of the ODE system. The overall objective is written as:
$$
\mathcal {L} = \sum_ {s = t - k} ^ {t} \left| \left| \hat {\boldsymbol {Y}} ^ {s} - \boldsymbol {Y} ^ {s} \right| \right| + \eta \left(\left| \left| \tilde {\boldsymbol {V}} ^ {s} - \boldsymbol {V} ^ {s} \right| \right| + \left| \left| \tilde {\boldsymbol {c}} ^ {s} - \boldsymbol {c} ^ {s} \right| \right|\right), \tag {13}
$$
where  $\hat{Y}^s$  denotes the predictions from the encoder and  $\eta$  is a parameter set to 0.1 to balance two losses.  $\tilde{V}^s$  and  $\tilde{c}^s$  denote the perturbed representations under noise attack to the input.
We evaluate our proposed CARE on both particle-based and mesh-based physical systems. To ensure the accuracy of our results, we use a rigorous data split strategy, where first  $80\%$  of the samples are reserved for training purposes and the remaining  $10\%$  are set aside for testing and validating,
Table 1: The RMSE  $(\times 10^{-2})$  results of the compared methods with the prediction lengths 1, 5, 10 and 20.  $v_{x}, v_{y}$  and  $v_{z}$  represent the velocity in the direction of each coordinate axis.
Prediction Length	+1	+5	+10	+20
Variable	vx	vy	vz	vx	vy	vz	vx	vy	vz	vx	vy	vz
Lennard-Jones Potential
LSTM	3.95	3.92	3.68	9.12	9.21	9.15	10.84	10.87	10.76	14.82	14.94	14.67
GNS	3.28	3.75	3.39	7.97	8.05	7.68	10.09	10.15	10.13	13.65	13.62	13.59
STGCN	2.91	3.08	2.95	5.06	5.17	5.11	6.89	6.90	6.93	9.31	9.32	9.44
MeshGraphNet	2.89	3.13	2.94	5.29	5.53	5.28	7.03	7.09	7.11	9.12	9.21	9.24
CG-ODE	1.79	2.05	1.71	3.47	3.92	3.38	5.46	5.99	5.36	9.03	9.26	8.92
TIE	1.62	1.98	1.47	3.25	3.90	3.15	5.24	5.82	5.17	8.24	8.34	8.47
Ours	0.76	0.89	1.01	2.94	3.16	2.85	5.01	4.69	4.71	5.75	5.91	5.82
3-body Stillinger-Weber Potential
LSTM	17.11	17.14	17.18	23.64	23.69	23.60	25.46	25.42	25.48	28.44	28.45	28.44
GNS	15.39	15.27	15.33	22.14	22.19	22.17	25.29	25.36	25.31	27.18	27.15	27.14
STGCN	12.33	12.31	12.35	17.94	17.96	17.91	20.08	20.14	20.13	23.49	23.51	23.52
MeshGraphNet	12.16	12.10	12.13	18.33	18.38	18.34	20.65	20.62	20.71	23.62	23.54	23.61
CG-ODE	9.78	9.74	9.75	12.11	12.05	12.14	15.55	15.58	15.50	16.17	16.24	16.22
TIE	10.18	10.26	10.19	14.75	14.70	14.73	18.42	18.45	18.41	20.92	21.04	21.36
Ours	4.21	4.29	4.18	9.74	9.79	9.71	13.65	13.71	13.57	15.30	15.39	15.35
CARE
Ground Truth
Figure 2: Visualization of Lennard-Jones Potential with multiple timestamps. We render the 3D positions of each particle according to the historical positions and predicted velocities.
separately. During training, we split each trajectory sample into two parts, i.e., a conditional part and a prediction part. We initialize node representations and the context representation based on the first part and utilize the second part to supervise the model. The size of the two parts is represented as conditional length and prediction length, respectively. Our approach is compared with various baselines for interacting systems modeling, i.e., LSTM [15], STGCN [67], GNS [45], MeshGraphNet [42], TIE [46] and CG-ODE [19].
5.1 Performance on Particle-based Physical Simulations
Datasets. We evaluate our proposed CARE on two particle-based simulation datasets with temporal environmental variations, i.e., Lennard-Jones Potential [47] and 3-body Stillinger-Weber Potential [4]. Lennard-Jones Potential is popular in modeling electronically neutral atoms or molecules. 3-body Stillinger-Weber Potential provides more complex relationships in atom systems The temperature in two particle-based systems is continuously changed along with the time to model the environmental variations. The objective is to predict the future velocity values in all directions, i.e.,  $v_{x}, v_{y}$  and  $v_{z}$ . More details can be found in Appendix.
Performance Comparison. We evaluate the performance in terms of RMSE with different prediction lengths. Table 1 show the compared results on these two datasets. We can observe that our proposed CARE outperforms all the baselines on two datasets. In particular, compared with TIE, CARE accomplishes an error reduction of  $24.03\%$  and  $36.35\%$  on two datasets, respectively. The remarkable performance can be attributed to two factors: (1) Introduction of the context variable. Our CARE infers the context states during the evolution of the system, which can help the model understand environmental variations. (2) Introduction of robust learning. We add noise attack to both system and context states, which improves the model generalization to potential distribution changes.
Table 2: The RMSE results of the compared methods over different prediction lengths 1, 10, 20 and 50.  $v_{x}$ ,  $v_{y}$  and  $p$  represent the velocity in different directions and the pressure field, respectively.
Prediction Length	+1	+10	+20	+50
Variable	vx	vy	p	vx	vy	p	vx	vy	p	vx	vy	p
CylinderFlow
LSTM	3.35	29.4	12.5	7.06	44.8	17.8	9.47	49.5	19.9	14.3	73.6	42.3
GNS	3.12	28.8	11.9	7.18	44.3	17.3	9.01	49.6	19.2	13.5	73.2	41.6
STGCN	2.68	26.7	11.0	5.47	42.1	16.9	6.72	45.6	18.4	9.15	68.7	40.0
MeshGraphNet	1.75	22.4	10.6	4.09	39.7	15.7	5.38	44.5	17.2	7.92	64.3	37.7
CG-ODE	1.05	20.4	8.51	3.44	36.8	13.6	4.15	38.5	17.1	5.14	61.2	32.3
TIE	1.22	20.8	8.94	3.75	35.2	13.0	4.62	40.6	16.0	5.87	59.5	32.1
Ours	0.87	19.1	7.21	3.02	32.9	11.8	3.95	37.8	13.9	4.97	55.8	29.4
Airfoil
LSTM	7.49	7.73	1.92	8.86	9.02	3.78	10.8	11.0	4.71	14.9	15.7	4.96
GNS	6.95	7.14	1.69	8.20	8.34	3.34	10.2	10.5	3.98	14.2	14.1	4.11
STGCN	6.24	5.35	1.07	6.57	6.51	2.33	7.88	8.01	3.16	11.6	11.8	3.17
MeshGraphNet	4.72	4.68	0.50	5.89	5.74	1.23	6.32	6.48	1.85	9.03	9.12	2.08
CG-ODE	4.26	4.32	0.35	4.78	4.70	0.46	5.81	5.66	1.04	7.39	7.85	1.69
TIE	4.17	4.39	0.33	4.99	4.86	0.51	5.75	5.62	0.95	7.25	7.63	1.44
Ours	3.51	4.11	0.19	3.86	3.75	0.34	4.16	4.12	0.45	6.74	6.82	0.81
TIE
CARE
Ground Truth
Figure 3: Visualization of the CylinderFlow Dataset with multiple timestamps. We render the velocity in the  $x$ -axis in the fluid field of our CARE and the ground truth.
Visualization. Figure 2 visualizes the prediction of positions in comparison to the ground truth on Lennard-Jones Potential. Here, we sample six timestamps in every trajectory to validate the performance of both short-term and long-term predictions. From the qualitative results, we can observe that in the first three frames, the particle motion is not strenuous due to low temperature in the system. Surprisingly, our proposed CARE can always make faithful physical simulations close to the ground truth even though the system environment is highly variable.
5.2 Performance on Mesh-based Physical Simulations
Datasets. We employ two popular mesh-based simulation datasets, i.e., CylinderFlow, and Airfoil. CylinderFlow consists of simulation data from modeling an incompressible flow governed by the Navier-Stokes equations. Notably, the initial flow velocity of the incoming water flow to the cylinder varies cyclically over time, meaning the Reynolds number of the flow field also changes periodically. Airfoil is generated in a similar manner through simulations of a compressible flow, wherein the inlet velocity over the wing varies cyclically over time. We aim to forecast the velocity values  $v_{x}$  and  $v_{y}$ , as well as the pressure  $p$ . More details can be found in Appendix.
Performance Comparison. The performance with respect to different variables is recorded in Table 2. From the results, we can observe that the average performance of the proposed CARE is over the best baseline by  $12.99\%$  and  $22.78\%$  on two datasets, respectively. Note that unsteady flow [11, 16] is a crucial problem in recent fluid dynamics, our proposed CARE can benefit abundant complex mesh-based simulations under environmental variations.
Visualization. Moreover, we show the qualitative results of the best baseline and our CARE in comparison to the ground truth. From the results, we can observe that our CARE can capture more accurate signals in unsteady fluid dynamics. In particular, in the last two frames with complicated
Table 3: Ablation study on four datasets.
Datasets	Lennard-Jones	3-body Stillinger-Weber	CylinderFlow	Airfoil
Variable	vx	vy	vz	vx	vy	vz	vx	vy	p	vx	vy	p
CARE V1	6.98	7.12	7.06	18.2	18.3	18.3	6.13	60.4	32.2	7.13	7.21	1.43
CARE V2	6.03	6.35	6.30	16.8	16.5	16.6	5.21	57.2	29.8	6.94	6.99	1.15
Ours	5.75	5.91	5.82	15.3	15.4	15.4	4.97	55.8	29.4	6.74	6.82	0.81
(b)
Figure 4: (a), (b) The performance with respect to different condition and prediction lengths on CylinderFlow and Airfoil. (c) The sensitivity of interval on Lennard-Jones Potential (LJP) and 3-body Stillinger-Weber Potential (SWP) datasets. (d) The comparison of running time for our dynamic graph updating and full pairwise calculation on two particle-based datasets.
structures, our CARE still can generate superior simulations in both scenarios under potential environmental variation while the baseline fails, which shows the superiority of our proposed CARE.
5.3 Further Analysis
Ablation Study. To analyze the effectiveness of different components in our CARE, we introduce two different variants: (1) CARE V1, which removes the context variable in Eqn. 9; (2) CARE V2, which removes the robust learning term in Eqn. 13. The compared performance is recorded in Figure 3 and we have two observations. First, our full model outperforms CARE V1, which indicates the incorporation of the context variable would benefit interacting system modeling under temporal environmental variation. Second, without the robust learning term, the performance would get worse, implying that improving the robustness can also benefit tackling the distribution changes.
Parameter Sensitivity. We begin by analyzing the performance with respect to different condition lengths and prediction lengths. Here the condition length and prediction length vary from  $\{10,15,20,25,30\}$ ,  $\{20,50\}$ , respectively. From the results in Figure 4 (a) and (b), we can observe that our proposed CARE can always achieve superior performance compared with CG-ODE. Moreover, we can observe that a longer condition length would benefit the performance in most cases due to more provided information. It can also be seen that a smaller interval for graph updating would improve the performance before saturation from Figure 4 (c).
Efficiency. To show the efficiency of our proposed dynamic graph updating, we propose a model variant named CARE E, which calculates all pairwise distances to update graph structure during the evolution. The computational cost is recorded in Figure 4 (d) and we can observe that our strategy can reduce a large number of computational costs, which validates the complexity analysis before.
This paper studies the problem of modeling interacting dynamics under temporal environmental variation and we propose a probabilistic framework to depict the dynamical system. Then, a novel approach named CARE is proposed. CARE first constructs an encoder to initialize the context variable indicating the environment and then utilizes a coupled ODE system, which combines both the context variable and node representation to drive the evolution of the system. Finally, we introduce both efficient dynamical graph updating and robust learning strategies to enhance our framework. Extensive experiments on four datasets validate the superiority of our CARE.
Broader Impacts and Limitations. This work presents an effective learning-based model for interacting dynamical systems under temporal environmental variation, which can benefit complex physical simulations such as unsteady flow. Moreover, our study provides a new perspective on
modeling environmental variations for fluid dynamics and intermolecular interactions. One potential limitation is that our CARE cannot directly fit more physical scenarios requiring abundant external knowledge. In future work, we would extend our CARE to more complicated applications such as rigid dynamics.
This work was partially supported by NSF 2211557, NSF 1937599, NSF 2119643, NSF 2303037, NSF 2312501, NASA, SRC Jump 2.0, Okawa Foundation Grant, Amazon Research Awards, Cisco research grant, Picsart Gifts, and Snapchat Gifts.
[1] Kelsey R Allen, Yulia Rubanova, Tatiana Lopez-Guevara, William Whitney, Alvaro Sanchez-Gonzalez, Peter Battaglia, and Tobias Pfaff. Learning rigid dynamics with face interaction graph networks. arXiv preprint arXiv:2212.03574, 2022.
[2] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In NeurIPS, 2016.
[3] Suresh Bishnoi, Ravinder Bhattoo, Jayadeva Jayadeva, Sayan Ranu, and NM Anoop Krishnan. Enhancing the inductive biases of graph neural ode for modeling physical systems. In ICLR, 2023.
[4] W Michael Brown and Masako Yamada. Implementing molecular dynamics on hybrid high performance computers—three-body potentials. Computer Physics Communications, 184(12):2785-2793, 2013.
[5] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In NeurIPS, 2018.
[6] Yongqiang Chen, Yonggang Zhang, Yatao Bian, Han Yang, MA Kaili, Binghui Xie, Tongliang Liu, Bo Han, and James Cheng. Learning causally invariant representations for out-of-distribution generalization on graphs. In NeurIPS, pages 22131-22148, 2022.
[7] Yitong Deng, Hong-Xing Yu, Jiajun Wu, and Bo Zhu. Learning vortex dynamics for fluid inference and prediction. arXiv preprint arXiv:2301.11494, 2023.
[8] Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. In NeurIPS, 2019.
[9] Zheng Fang, Qingqing Long, Guojie Song, and Kunqing Xie. Spatial-temporal graph ode networks for traffic flow forecasting. In KDD, pages 364-373, 2021.
[10] Jayesh K Gupta, Sai Vemprala, and Ashish Kapoor. Learning modular simulations for homogeneous systems. In NeurIPS, 2022.
[11] Rohit Gupta and Phillip J Ansell. Unsteady flow physics of airfoil dynamic stall. AIAA journal, 57(1):165-175, 2019.
[12] Nesrine Hafiene, Wafa Karoui, and Lotfi Ben Romdhane. Influential nodes detection in dynamic social networks: A survey. Expert Systems with Applications, 159:113642, 2020.
[13] Mingguo He, Zhewei Wei, and Ji-Rong Wen. Convolutional neural networks on graphs with chebyshev approximation, revisited. In NeurIPS, 2022.
[14] Yue He, Zimu Wang, Peng Cui, Hao Zou, Yafeng Zhang, Qiang Cui, and Yong Jiang. Causpref: Causal preference learning for out-of-distribution recommendation. In WWW, pages 410-421, 2022.
[15] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.
A Proof of Lemma 4.1
$$
\mathrm {P} \left(\boldsymbol {Y} ^ {t} \mid G ^ {0: t - 1}\right) = \int \mathrm {P} \left(\boldsymbol {Y} ^ {t} \mid \boldsymbol {e} ^ {t - 1}, G ^ {t - 1}\right). \tag {14}
$$
Proof. We have:
$$
\begin{array}{l} \mathrm {P} \left(\boldsymbol {Y} ^ {t} \mid G ^ {0: t - 1}\right) \\ = \int \mathrm {P} \left(\boldsymbol {Y} ^ {t} \mid \boldsymbol {c} ^ {t - 1}, G ^ {0: t - 1}\right) \cdot \mathrm {P} \left(\boldsymbol {c} ^ {t - 1} \mid G ^ {0: t - 1}\right) d \boldsymbol {c} ^ {t - 1} \\ = \int \mathrm {P} \left(\boldsymbol {Y} ^ {t} \mid \boldsymbol {e} ^ {t - 1}, G ^ {t - 1}\right) \cdot \mathrm {P} \left(\boldsymbol {e} ^ {t - 1} \mid G ^ {0: t - 1}\right) d \boldsymbol {e} ^ {t - 1} \tag {15} \\ = \int \mathrm {P} \left(\boldsymbol {Y} ^ {t} \mid \boldsymbol {c} ^ {t - 1}, G ^ {t - 1}\right) \cdot \mathrm {P} \left(\boldsymbol {c} ^ {t - 1} \mid \boldsymbol {c} ^ {t - k}, G ^ {0: t - 1}\right) \cdot \mathrm {P} \left(\boldsymbol {c} ^ {t - k} \mid G ^ {0: t - k}\right) d \boldsymbol {c} ^ {t - 1} d \boldsymbol {c} ^ {t - k} \\ = \int \mathrm {P} \left(\boldsymbol {Y} ^ {t} \mid \boldsymbol {c} ^ {t - 1}, G ^ {t - 1}\right) \cdot \mathrm {P} \left(\boldsymbol {c} ^ {t - 1} \mid \boldsymbol {c} ^ {t - k}, G ^ {t - k: t - 1}\right) \cdot \mathrm {P} \left(\boldsymbol {c} ^ {t - k} \mid G ^ {0: t - k}\right) d \boldsymbol {c} ^ {t - 1} d \boldsymbol {c} ^ {t - k} \\ \end{array}
$$
B Proof of Lemma 4.2
For convenience, Eqn. 9 in the main paper is repeated as:
$$
\frac {d \boldsymbol {v} _ {i} ^ {s}}{d s} = \Phi \left(\left[ \boldsymbol {v} _ {1} ^ {s}, \dots , \boldsymbol {v} _ {N} ^ {s}, \boldsymbol {c} ^ {s} \right]\right) = \sigma \left(\sum_ {j \in \mathcal {N} ^ {s} (i)} \frac {\hat {\boldsymbol {A}} _ {i j} ^ {s}}{\sqrt {\hat {\boldsymbol {D}} _ {i} ^ {s} \cdot \hat {\boldsymbol {D}} _ {j} ^ {s}}} \boldsymbol {v} _ {j} ^ {s} \boldsymbol {W} _ {1} + \boldsymbol {c} ^ {s} \boldsymbol {W} _ {2}\right), \tag {16}
$$
Eqn. 12 is repeated as:
$$
\frac {d \boldsymbol {c} ^ {s}}{d s} = \sigma \left(\sum_ {i = 1} ^ {N} \left(\boldsymbol {v} _ {i} ^ {s} \boldsymbol {W} _ {3} + \frac {d \boldsymbol {v} _ {i} ^ {s}}{d s} \boldsymbol {W} _ {4}\right) + \boldsymbol {c} ^ {s} \boldsymbol {W} _ {5}\right). \tag {17}
$$
Lemma 4.2. Given the initial state  $(t_0, \pmb{v}_1^{t_0}, \dots, \pmb{v}_N^{t_0}, \pmb{c}^{t_0})$ , we claim that there exists  $\varepsilon > 0$ , s.t. the ODE system Eqn. 16 and Eqn. 17 has a unique solution in the interval  $[t_0 - \varepsilon, t_0 + \varepsilon]$ .
To begin, we introduce the Picard-Lindelöf Theorem as follows:
Theorem B.1. (Picard-Lindelöf Theorem) Let  $D \subseteq \mathbb{R} \times \mathbb{R}^n$  be a closed rectangle with  $(t_0, y_0) \in D$ . Let  $f: D \to \mathbb{R}^n$  be a function which is continuous with respect to  $t$  and Lipschitz continuous with respect to  $y$ . Then, there exists some  $\varepsilon > 0$  such that the initial value problem:
$$
y ^ {\prime} (t) = f (t, y (t)), \quad y \left(t _ {0}\right) = y _ {0}. \tag {18}
$$
has a unique solution  $y(t)$  in the interval  $[t_0 - \varepsilon, t_0 + \varepsilon]$ .
Proof. Let  $A_{ij}^{s} = 0$  if  $j \notin \mathcal{N}^{s}(i)$  and denote  $M_{ij}^{s} = \frac{A_{ij}^{s}}{\sqrt{\hat{D}_{i}^{s} \cdot \hat{D}_{j}^{s}}}$ . Then, we transpose them with Eqn. 16 and 17 becoming:
$$
\frac {d \left(\boldsymbol {v} _ {i} ^ {s}\right) ^ {T}}{d s} = \sigma \left(\sum_ {j = 1} ^ {N} M _ {i j} ^ {s} \boldsymbol {W} _ {1} ^ {T} \left(\boldsymbol {v} _ {j} ^ {s}\right) ^ {T} + \boldsymbol {W} _ {2} ^ {T} \left(\boldsymbol {c} ^ {s}\right) ^ {T}\right), \tag {19}
$$
$$
\frac {d (\boldsymbol {c} ^ {s}) ^ {T}}{d s} = \sigma \left(\sum_ {i = 1} ^ {N} (\boldsymbol {W} _ {3} ^ {T} (\boldsymbol {v} _ {i} ^ {s}) ^ {T} + \boldsymbol {W} _ {4} ^ {T} \frac {d (\boldsymbol {v} _ {i} ^ {s}) ^ {T}}{d s}) + \boldsymbol {W} _ {5} ^ {T} (\boldsymbol {c} ^ {s}) ^ {T}\right).
$$
$$
\text {L e t} \boldsymbol {Y} ^ {s} = \left( \begin{array}{c} (\boldsymbol {v} _ {1} ^ {s}) ^ {T} \\ \vdots \\ (\boldsymbol {v} _ {N} ^ {s}) ^ {T} \\ (\boldsymbol {c} ^ {s}) ^ {T} \end{array} \right) \in \mathbf {R} ^ {(N \times d _ {v} + d _ {c}) \times 1}, \text {w h e r e} \boldsymbol {v} _ {i} \in \mathbf {R} ^ {d _ {v}}, \boldsymbol {c} \in \mathbf {R} ^ {d _ {c}}.
$$
From the Eqn. 19 we get the ODE system  $\frac{d\mathbf{Y}^s}{ds} = f(\mathbf{Y}^s;\theta)$  with fixed parameters  $\theta$  for the ODE solver. Here the function  $f(\mathbf{Y}^t;\theta)$  is continuous w.r.t  $t$  since all components in the vector  $\mathbf{Y}^t$  are continuous w.r.t  $t$  and  $\theta$  does not depend on  $t$ .
Now consider activation functions  $\sigma$ , such as ReLU, that satisfy the following inequality for all  $x$  and  $y$ :
$$
\| \sigma (x) - \sigma (y) \| \leq \| x - y \|
$$
Then, for any two solutions  $\mathbf{Y}_1^s, \mathbf{Y}_2^s$ , we have:
$$
\| f (\mathbf {Y} _ {1} ^ {s}; \theta) - f (\mathbf {Y} _ {2} ^ {s}; \theta) \| _ {2} \leq \left\| \left( \begin{array}{c} \delta_ {1} \\ \vdots \\ \delta_ {N} \\ \delta_ {N + 1} \end{array} \right) \right\| _ {2},
$$
$$
\delta_ {i} = \sum_ {j = 1} ^ {N} \boldsymbol {M} _ {i j} ^ {s} \boldsymbol {W} _ {1} ^ {T} \left[ \left(\boldsymbol {v} _ {1 j} ^ {s}\right) ^ {T} - \left(\boldsymbol {v} _ {2 j} ^ {s}\right) ^ {T} \right] + \boldsymbol {W} _ {2} ^ {T} \left[ \left(\boldsymbol {c} _ {1} ^ {s}\right) ^ {T} - \left(\boldsymbol {c} _ {2} ^ {s}\right) ^ {T} \right], \quad i = 1, \dots , N
$$
$$
\delta_ {N + 1} = \sum_ {i = 1} ^ {N} \left(\boldsymbol {W} _ {3} ^ {T} \left[ (\boldsymbol {v} _ {1 i} ^ {s}) ^ {T} - (\boldsymbol {v} _ {2 i} ^ {s}) ^ {T} \right] + \boldsymbol {W} _ {4} ^ {T} \left[ \sigma \left(\frac {d (\boldsymbol {v} _ {1 i} ^ {s}) ^ {T}}{d s}\right) - \sigma \left(\frac {d (\boldsymbol {v} _ {2 i} ^ {s}) ^ {T}}{d s}\right) \right]\right) + \boldsymbol {W} _ {5} ^ {T} \left[ (\boldsymbol {c} _ {1} ^ {s}) ^ {T} - (\boldsymbol {c} _ {2} ^ {s}) ^ {T} \right].
$$
To simplify the representation, denote:
$$
\Delta \boldsymbol {v} _ {j} ^ {s} = \boldsymbol {v} _ {1 j} ^ {s} - \boldsymbol {v} _ {2 j} ^ {s}, \quad \Delta \boldsymbol {c} ^ {s} = \boldsymbol {c} _ {1} ^ {s} - \boldsymbol {c} _ {2} ^ {s}.
$$
Then by triangular inequality, we have  $\forall i\in \{1,\dots ,N\}$
$$
\begin{array}{l} \left\| \delta_ {i} ^ {T} \right\| _ {2} ^ {2} = \| \sum_ {j = 1} ^ {N} \Delta v _ {j} ^ {s} W _ {1} \left(M _ {i j} ^ {s}\right) ^ {T} + \Delta c ^ {s} W _ {2} \| _ {2} ^ {2} \\ \leq \left(\sum_ {j = 1} ^ {N} \| \Delta \boldsymbol {v} _ {j} ^ {s} \boldsymbol {W} _ {1} \left(\boldsymbol {M} _ {i j} ^ {s}\right) ^ {T} \| _ {2} + \| \Delta \boldsymbol {c} ^ {s} \boldsymbol {W} _ {2} \| _ {2}\right) ^ {2} \\ \leq \left(W M \sum_ {j = 1} ^ {N} \| \Delta \boldsymbol {v} _ {j} ^ {s} \| _ {2} + W \| \Delta \boldsymbol {c} ^ {s} \| _ {2}\right) ^ {2}, \\ \end{array}
$$
$$
\begin{array}{l} \left\| \delta_ {N + 1} ^ {T} \right\| _ {2} ^ {2} = \| \sum_ {i = 1} ^ {N} \left(\Delta \boldsymbol {v} _ {i} ^ {s} \boldsymbol {W} _ {3} + \left[ \sigma \left(\frac {d \boldsymbol {v} _ {1 i} ^ {s}}{d s}\right) - \sigma \left(\frac {d \boldsymbol {v} _ {2 i} ^ {s}}{d s}\right) \right] \boldsymbol {W} _ {4}\right) + \Delta \boldsymbol {c} ^ {s} \boldsymbol {W} _ {5} \| _ {2} ^ {2} \\ \leq \left(\sum_ {i = 1} ^ {N} \| \Delta \boldsymbol {v} _ {i} ^ {s} \boldsymbol {W} _ {3} \| _ {2} + \sum_ {i = 1} ^ {N} \| \left[ \sigma \left(\frac {d \boldsymbol {v} _ {1 i} ^ {s}}{d s}\right) - \sigma \left(\frac {d \boldsymbol {v} _ {2 i} ^ {s}}{d s}\right) \right] \boldsymbol {W} _ {4} \| _ {2} + \| \Delta \boldsymbol {c} ^ {s} \boldsymbol {W} _ {5} \| _ {2}\right) ^ {2} \\ \leq \left(W \sum_ {i = 1} ^ {N} \| \Delta \boldsymbol {v} _ {i} ^ {s} \| _ {2} + W \sum_ {i = 1} ^ {N} \| \frac {d \boldsymbol {v} _ {1 i} ^ {s}}{d s} - \frac {d \boldsymbol {v} _ {2 i} ^ {s}}{d s} \| _ {2} + W \| \Delta \boldsymbol {c} ^ {s} \| _ {2}\right) ^ {2} \\ = \left(W \sum_ {i = 1} ^ {N} \| \Delta \boldsymbol {v} _ {i} ^ {s} \| _ {2} + W \sum_ {i = 1} ^ {N} \| \delta_ {i} ^ {T} \| _ {2} + W \| \Delta \boldsymbol {c} ^ {s} \| _ {2}\right) ^ {2} \\ \leq \left(W \sum_ {i = 1} ^ {N} \| \Delta \boldsymbol {v} _ {i} ^ {s} \| _ {2} + W N \left(W M \sum_ {j = 1} ^ {N} \| \Delta \boldsymbol {v} _ {j} ^ {s} \| _ {2} + W \| \Delta \boldsymbol {c} ^ {s} \| _ {2}\right) + W \| \Delta \boldsymbol {c} ^ {s} \| _ {2}\right) ^ {2} \\ = \left(\left(W + M N W ^ {2}\right) \sum_ {i = 1} ^ {N} \| \Delta \boldsymbol {v} _ {i} ^ {s} \| _ {2} + \left(W + N W ^ {2}\right) \| \Delta \boldsymbol {c} ^ {s} \| _ {2}\right) ^ {2}. \\ \end{array}
$$
Without loss of generality, we assume constants  $M, W > 1$ . Thus, we have the following result: