Hydrodynamics-Informed Neural Network for Simulating Dense Crowd Motion Patterns

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 11:12:42

Abstract: With global occurrences of crowd crushes and stampedes, dense crowd simulation has been drawing great attention. In this research, our goal is to simulate dense crowd motions under six classic motion patterns, more specifically, to generate subsequent motions of dense crowds from the given initial states. Since dense crowds share similarities with fluids, such as continuity and fluidity, one common approach for dense crowd simulation is to construct hydrodynamics-based models, which consider dense crowds as fluids, guide crowd motions with Navier-Stokes equations, and conduct dense crowd simulation by solving governing equations. Despite the proposal of these models, dense crowd simulation faces multiple challenges, including the difficulty of directly solving Navier-Stokes equations due to their nonlinear nature, the ignorance of distinctive crowd characteristics which fluids lack, and the gaps in the evaluation and validation of crowd simulation models. To address the above challenges, we build a hydrodynamic model, which captures the crowd physical properties (continuity, fluidity, etc.) with Navier-Stokes equations and reflects the crowd social properties (sociality, personality, etc.) with operators that describe crowd interactions and crowd-environment interactions. To tackle the computational problem, we propose to solve the governing equation based on Navier-Stokes equations using neural networks, and introduce the Hydrodynamics-Informed Neural Network (HINN) which preserves the structure of the governing equation in its network architecture. To facilitate the evaluation, we construct a new dense crowd motion video dataset called Dense Crowd Flow Dataset (DCFD), containing six classic motion patterns (line, curve, circle, cross, cluster and scatter) and 457 video clips, which can serve as the groundtruths for various objective metrics. Numerous experiments are conducted using HINN to simulate dense crowd motions under six motion patterns with video clips from DCFD. Objective evaluation metrics that concerns authenticity, fidelity and diversity demonstrate the superior performance of our model in dense crowd simulation compared to other simulation models.

Main paper: 
 introduction: 
 Crowd simulation has long been popular within computer vision for its extensive applications in fields of urban planning [10, 30] , emergency evacuation [3, 14] , virtual reality [32, 45] , video games [48, 53] , and more. With crowd crushes and stampedes occurring frequently around the world, there is an increasingly urgent desire for dense crowd simulation in particular. Through dense crowd simulation, it becomes possible to predict crowd motions, prevent potential accidents, and protect public safety.Models for crowd simulation can be generally categorized into microscopic (rule-based or agent-based) models and macroscopic (hydrodynamics-based) models. Classic microscopic models, such as Boids [41] , Vicsek [51] and Social Force Model (SFM) [15] , require each crowd member or agent to perceive and react individually to the environment and others based on certain rules. Along this line of thought, follow-up researches extend these models by considering aspects of evacuation panic [5, 14, 34] , collision avoidance [35, 49] , human psychology [8, 9, 37] , etc. These models are normally applied for small-scale crowd simulation and are not suitable for largescale due to increased computational complexity and challenges in capturing individual behaviors.On the contrary, macroscopic models consider crowds as fluidlike entities, utilizing hydrodynamics to handle the continuity and fluidity of crowd motions, making them more suitable for largescale or dense crowd simulation. Proposed by Henderson [16] and supplemented by Bradley [4] , this approach to constructing hydrodynamic models for dense crowds involves regarding dense crowds as fluids, guiding crowd motions with Navier-Stokes equations, and conducting dense crowd simulation by solving the governing equations.Despite the advancement of hydrodynamic models, we are facing three challenges for dense crowd simulation. Firstly, due to the nonlinear nature of Navier-Stokes equations, the solving process remains complicated whether for crowds or fluids. Previous works [43, 50, 55] approximate the solution through Smoothed Particle Hydrodynamics (SPH) [12] , which is a computational method that considers crowds or fluids as assemblies of individuals or elements with associated properties and uses smoothing kernels to approach their interactions. However, SPH compromises simulation accuracy and introduces computational complexity as it is particle-based. Secondly, as shown in Fig. 1 , other than similarities shared with fluids (physical properties), crowds possess characteristics (social properties) such as sociality and personality, which makes it difficult to accomplish the simulation with pure hydrodynamics. Thirdly, there are gaps in the evaluation of crowd simulation, particularly for large-scale or dense crowds. Previous researches normally rely on metrics such as time consumption [7, 33] , computational complexity [39, 47] , and subjective judgment [11, 55] . However, these metrics are not convincing enough and fail to reflect authenticity, fidelity and diversity of the simulation.To address the above challenges, we construct a hydrodynamic model concerning both the physical and social properties of crowds. We guide crowd motions with the momentum equation of Navier-Stokes equations to capture physical properties, and incorporate additional forces consistent with crowd interactions and crowdenvironment interactions into the governing equation to reflect social properties. Then, inspired by Partial Differential Equation (PDE)-Preserved Neural Network (PPNN) [27] , we propose to solve the governing equation using neural networks and introduce the Hydrodynamics-Informed Neural Network (HINN) which preserves the structure of the governing equation in its network architecture. To facilitate the evaluation, we collect hundreds of video clips of six classic dense crowd motion patterns, including line, curve, circle, cross, cluster and scatter, to construct a real-world dense crowd motion video dataset called the Dense Crowd Flow Dataset (DCFD). The construction of DCFD enables us to employ objective metrics about authenticity, fidelity and diversity for evaluation, including Inception Score (IS) [42] , Fréchet Inception Distance (FID) [17] and Structural Similarity (SSIM) [52] , by serving as the groundtruths. We conduct numerous experiments using HINN to simulate dense crowd motions under the six classic motion patterns with video clips from DCFD, and evaluate the performances of our model and other simulation models with objective metrics. Our contributions are summarized as follows:• We propose a hydrodynamic model which considers both the similarities shared between crowds and fluids, as well as the unique characteristics possessed by crowds, namely physical and social properties. • We design three operators to describe interactions within dense crowds and between crowds and environments. These operators can be applied to simulate six classic dense crowd motion patterns. • We construct a new real-world dense crowd video dataset which contains six classic dense crowd motion patterns.Experiments demonstrate the effectiveness of our approach compared with other simulation models by using objective evaluation metrics. 
related work 2.1 hydrodynamics-based crowd simulation: 
 As crowds and fluids share continuity and fluidity, hydrodynamicsbased models aim to reflect crowd physical properties by drawing upon insights from fluid dynamics. This approach was proposed by Henderson [16] , regarding loose crowds as gas, dense crowds as liquid, and loose-dense transitions as phase transformation. To go one step further, Bradley [4] guided dense crowd motions with Navier-Stokes equations and accomplished dense crowd simulation by solving the governing equations. On this basis, Hughes [20] suggested to discard terms with little effect and introduce additional terms to the governing equations based on the specific situation. For the difficulty of solving Navier-Stokes equations directly, followup researchers, such as Yuan [55] and van Toll [50] , mostly used SPH [12] to approximate the solution. However, SPH on the one hand compromises accuracy, and on the other hand introduces computational complexity as it is particle-based. Hence, the main problem of such models lies in the absence of an accurate yet simple computational method for Navier-Stokes equations.In this research, we construct a hydrodynamic model to reflect physical properties of dense crowd motions under the guidance of Navier-Stokes equations. As a complement, we introduce additional forces into the governing equation to capture social properties of dense crowds. To tackle the aforementioned computational problem, we propose to solve the governing equation with PDE-solving neural networks.Methods of solving PDE with neural networks can be basically divided as data-driven and physics-informed. Data-driven neural networks, including Convolutional Neural Networks (CNN) [13, 23] and PDE-Net [28] , learn differential operators using convolutional kernels and fit the nonlinear response function by neural networks. Such networks rely heavily on large amounts of data, struggle with error accumulation and have poor generalizability. To tackle these problems, Physics-Informed Neural Networks (PINNs) [40] and its variants [21, 31] employ Deep Neural Networks (DNN) for their well-known capability as universal function approximators [18] , exploit automatic differentiation, and utilize the prior physics knowledge by incorporating PDE residuals into the loss function. Besides, physics-informed variants of neural operators, such as DeepONet [29] and Fourier Neural Operator (FNO) [26] , learn the mapping from functional parametric dependence to the solution, thus learn an entire family of PDE, instead of solving one instance. However, this method of embedding PDE residuals into the loss function depends largely on extensive adjustments of the hyperparameters to weigh each loss term. To this end, PDE-Preserved Neural Network (PPNN) [27] embeds PDE into the neural network architecture via connections between PDE operators and network structures, which improves the generalizability as well as long-term prediction accuracy at the same time.In this research, to address the aforementioned computational challenge faced by hydrodynamics-based models, inspired by PPNN, we propose HINN to solve our governing equation. HINN preserves the structure of our governing equation based on Navier-Stokes equations for dense crowd simulation within a plug-in module called Hydrodynamics-Informed Module (HIM). 
methodology 3.1 overview: 
 Our goal is to build a hydrodynamic model that considers both the physical and social properties for dense crowd simulation, more specifically, for generating subsequent motions of dense crowds from the given initial states. For a dense crowd video clip with 𝑇 + 1 frames, 𝐹 = [𝑓 0 , 𝑓 1 , . . . , 𝑓 𝑇 ], dense optical flows can be computed between consecutive frames. These optical flows approximately provide the velocity field U = [u 0 , u 1 , . . . , u 𝑇 -1 ], where u 𝑡 is the velocity between two consecutive frames 𝑓 𝑡 and 𝑓 𝑡 +1 . Taking the initial velocity u 0 as input, the hydrodynamic model can predict the velocity û1 for the next time-step. Subsequently, for any time-step 𝑡 > 0, the hydrodynamic model can predict the succeeding velocity û𝑡+1 based on the previously predicted velocity û𝑡 . Consequently, the predicted velocity field of this video clip can be denoted as Û = [ û0 , û1 , . . . , û𝑇 -1 ], where û0 = u 0 . Then, our target is to minimize the distance between groudtruths U and predictions Û, which can be reformulated as an optimization problem:EQUATIONwhere 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒 (•, •) can be various evaluation metrics that measure the distance between groudtruths U and predictions Û, including Huber loss [19] , IS, FID, SSIM, etc.To simulate dense crowd motions, we construct a hydrodynamic model that guides the crowd motions with a hydrodynamics-based governing equation and solves the governing equation using a nextstep prediction network called HINN. As shown in Fig. 2 , HINN contains two parts: a trainable network that can be any predictive network [13, 23, 46] , but we adopt ConvResNet here because of its effectiveness demonstrated in PPNN; and a plug-in module called HIM, which preserves the structure of the governing equation. At any time-step 𝑡, taking the previously predicted velocity û𝑡 as the input, HIM converts it to low resolution by using bilinear downsampling, extracts the hidden feature preserved by the governing equation in the low-resolution space, and uses bicubic up-sampling to transform extracted feature back into original high-resolution space as û′ 𝑡 . Then, the acceleration-related feature û′ 𝑡 is attached to the input velocity û𝑡 , serving as the combined input for the trainable network to predict the velocity û𝑡+1 for the next timestep. This predicted velocity û𝑡+1 can further serve as the input again to predict subsequent velocity. After 𝑇 steps, we obtain Û = [ û0 , û1 , . . . , û𝑇 -1 ] that denotes the dense crowd motion generated from the initial states.For the governing equation, we consider both physical and social properties of crowds. Since crowds share continuity and fluidity with fluids, we capture the crowd physical properties according to the following momentum equation of Navier-Stokes equations for incompressible fluids:𝜕u 𝜕𝑡 + (u • ∇)u = 𝜈∇ 2 u - 1 𝜌 ∇𝑝 + f (2)where u is the velocity, 𝜈 is the viscosity, 𝜌 is the density, 𝑝 is the pressure, and f is the external force. Generally, this equation can be decomposed into five key components: the acceleration 𝜕u 𝜕𝑡 , the convective acceleration (u • ∇)u, the viscosity force 𝜈∇ 2 u, the pressure force -1 𝜌 ∇𝑝 and the external forces f. In this research, we regard dense crowd simulation as a twodimensional problem, in which case the momentum equation of Navier-Stokes equations in horizontal and vertical directions can be expressed as follows:EQUATION𝜕𝑣 𝜕𝑡EQUATIONwhere 𝑢 and 𝑣 are the horizontal and vertical components of velocity, 𝑓 𝑥 and 𝑓 𝑦 are the horizontal and vertical components of external forces respectively. Meanwhile, to reflect social properties of dense crowd motions, we introduce additional forces into the equation concerning the characteristics that crowds possess while fluids lack. Combining these two aspects, we formulate our governing equation for dense crowd simulation considering the following terms, where the first two terms reflect the physical properties and the others reflect the social properties.Convection force. Similar to fluids, this term is caused by the uneven distribution of the velocity field. The relative velocity of crowds can lead to collisions or tendencies to avoid collisions, bringing about changes in the velocity field. Thus, we denote the convective force for dense crowds as:EQUATIONTo implement the convective force in HIM, we approximate the first-order differentiation with the Sobel operators:EQUATIONViscosity force. Originated from the uneven distribution of velocity field as well, we calculate the viscosity force for dense crowds by:EQUATIONTo implement the viscosity force in HIM, we approach the secondorder differentiation using the Laplacian operator:𝜕 2 𝜕𝑥 2 + 𝜕 2 𝜕𝑦 2 = ∇ 2 =       0 1 0 1 -4 1 0 1 0       (8)Alignment force. As mentioned in Boids, pressure from surrounding members can drive individual to move along with them, namely to match their velocity. In this way, we represent the alignment force for dense crowds as the difference to average velocity of the surrounding area:𝑓 𝑎𝑙𝑖 𝑥 = 1 𝑁 ∑︁ 𝑖 ∈ Ů 𝑢 𝑖 -𝑢 𝑓 𝑎𝑙𝑖 𝑦 = 1 𝑁 ∑︁ 𝑖 ∈ Ů 𝑣 𝑖 -𝑣 (9)where Ů is the surrounding area and 𝑁 is the number of grids contained in the area. The alignment force can be implemented in HIM by using the zero-centered mean filter.Navigation force. Referring to SFM, individuals have their desired velocities toward the destination, which manifests as the tendency for people to follow the steps of those ahead of them in dense crowds. To describe this tendency, the navigation force for dense crowds can be expressed as:EQUATIONwhere 𝑢 𝑒 and 𝑣 𝑒 are the horizontal and vertical components of the desired velocity, which is the velocity of the person ahead, and 𝜏 is the relaxation time. To calculate the navigation force, we design a forward discriminator in HIM that can determine the forward direction of each individual, identify the person ahead, and thus obtain the desired velocity for calculation.Cohesion force. Individuals located at crowd boundaries normally do not break away from the crowds easily, trying to resist dispersion and maintain the overall shape of crowds. To capture this property, we compute the cohesion force for dense crowds by:𝑓 𝑐𝑜ℎ 𝑥 =          1 𝑙𝑒 𝑓 𝑡 𝑏𝑜𝑟𝑑𝑒𝑟 0 𝑖𝑛𝑛𝑒𝑟 𝑎𝑟𝑒𝑎 -1 𝑟𝑖𝑔ℎ𝑡 𝑏𝑜𝑟𝑑𝑒𝑟 𝑓 𝑐𝑜ℎ 𝑦 =          1 𝑢𝑝𝑝𝑒𝑟 𝑏𝑜𝑟𝑑𝑒𝑟 0 𝑖𝑛𝑛𝑒𝑟 𝑎𝑟𝑒𝑎 -1 𝑙𝑜𝑤𝑒𝑟 𝑏𝑜𝑟𝑑𝑒𝑟 (11)To compute the cohesion force, in HIM, we employ the Canny edge detector [6] to recognize the crowd boundaries.To sum up, we propose our governing equation for dense crowd simulation as:EQUATION)EQUATIONwhere 𝜆 1 , 𝜆 2 , 𝜆 3 , 𝜆 4 and 𝜆 5 are parameters.The evaluation in crowd simulation faces considerable challenges. Previous works normally adopt metrics such as time consumption, computational complexity and subjective judgment, which fail to reflect the authenticity, fidelity and diversity of the simulation. The primary reason behind this issue is the lack of datasets that can act as groundtruths. To this end, we construct a new dense crowd motion video dataset called DCFD containing six different motion patterns. Based on DCFD, we propose to employ the following objective metrics to evaluate the performances of our hydrodynamic model and other simulation models. Inception Score. Originally proposed for Generative Adversarial Networks (GAN), IS evaluates the distinctiveness and variety of generated images based on a pretrained classifier called InceptionV3.To employ IS in dense crowd simulation, we modify the input and output layers of InceptionV3 and retrain the classifier on DCFD with a ratio of eight to two between the training set and the test set. Then, IS for dense crowd simulation can be denoted as:EQUATIONEQUATION)where 𝐾𝐿(•, •) is the Kullback-Leibler Divergence (KLD) [24] , Ĝ is the prediction set, and ŷ is the motion pattern of each velocity field Û given by the classifier.Fréchet Inception Distance. Based on InceptionV3 as well, FID takes the groundtruths into account. FID treats the groundtruth set and prediction set as two Gaussian distributions, calculates their mean and covariance respectively, and measures the distance between the groundtruths and predictions by:EQUATION)where 𝐺 is the groundtruth set and Ĝ is the prediction set, (𝑚, 𝐶) and ( m, Ĉ) are their mean and covariance respectively.Structural Similarity. Designed for image quality assessment, SSIM compares groundtruths and predictions in terms of three aspects including the luminance, contrast and structure. To apply SSIM to dense crowd simulation, we represent the velocities using heat maps and compute SSIM by: 𝑙 (ℎ, ĥ) = 2𝜇 ℎ 𝜇 ĥ 𝜇 2 ℎ + 𝜇 2 ĥ 𝑐 (ℎ, ĥ) = 2𝜎 ℎ 𝜎 ĥ 𝜎 2 ℎ + 𝜎 2 ĥ 𝑠 (ℎ, ĥ) = 𝜎 ℎ ĥ 𝜎 ℎ 𝜎 ĥ (17)𝑆𝑆𝐼 𝑀 (𝐺, Ĝ) = 1 𝑁 𝑀 𝑁 ∑︁ 𝑖=1 𝑀 -1 ∑︁ 𝑗=0𝑙 (ℎ 𝑖 𝑗 , ĥ 𝑖 𝑗 )𝑐 (ℎ 𝑖 𝑗 , ĥ 𝑖 𝑗 )𝑠 (ℎ 𝑖 𝑗 , ĥ 𝑖 𝑗 ) (18) where 𝑙 (•, •) is the luminance comparison, 𝑐 (•, •) is the contrast comparison, and 𝑠 (•, •) is the structure comparison; ℎ 𝑖 𝑗 and ĥ 𝑖 𝑗 are the heat maps corresponding to the velocity between the 𝑗 𝑡ℎ and ( 𝑗 + 1) 𝑡ℎ frames of the 𝑖 𝑡ℎ video in the groundtruth set and prediction set respectively. 
experiments 4.1 crowd dataset: 
 To facilitate dense crowd researches as well as the evaluation and validation of crowd simulation models, we construct a new dense crowd motion video dataset called DCFD. DCFD contains dense crowd motions of six classic patterns, including line, curve, circle, cross, cluster, and scatter. These crowd motions are sourced from various scenes, such as marathons, parades, intersections, etc. In total, DCFD consists of 457 video clips, which are all collected from Getty Images1 . More than ninety percent of these video clips are unique to DCFD and have not been included in previous datasets. Each video clip is cropped into MP4 format with a size of 360 × 480 pixels, spanning approximately 120 frames. To utilize these dense crowd videos in our research, we also calculate the velocity field of each video clip using the optical flows [1, 2] . As is shown in Tab.1, most existing crowd video datasets are either not dense enough or contain very few videos. Our DCFD surpasses these datasets considering the crowd density, number of video clips and crowd motion patterns, and notably contains velocity fields for all video clips. All the video clips and velocity fields are sorted and labeled manually by multiple annotators according to crowd motion patterns and are available in the dataset. We implement our HINN with PyTorch [36] and conduct all the experiments on a single Nvidia GeForce RTX 3090 GPU. The number of simulation time steps 𝑇 in training and testing phase is 100. The relaxation time 𝜏 in Eq.10 is set to 1. The parameters 𝜆 1 , 𝜆 2 , 𝜆 3 , 𝜆 4 , 𝜆 5 in 𝑓 𝑎𝑐𝑐 𝑥 and 𝑓 𝑎𝑐𝑐 𝑦 are 1, -0.2, 0.5, 1 and 1 respectively. Here, we select this group of parameters out of a simple comparison. Particularly, 𝜆 2 is negative since the convection acceleration and the viscosity force for fluids lie on either side of Navier-Stokes equations as shown in Eq.2. The training set and the test set are split from DCFD at a ratio of seven to three. All experiments are conducted with ADAM [22] as the optimizer, Huber loss as the loss function, and IS, FID and SSIM as the evaluation metrics. More details about the parameters can be seen in the supplementary.To examine the effectiveness of our proposed model, we compare it with two kinds of simulation methods. One method is to treat crowds as assemblies of individuals and constrain their behaviors with rules, which can effectively reflect the social properties of crowds. For this kind of method, we select traditional models such as Boids and SFM based on their established performance in crowd simulation. The other method is to guide crowd motions with the original Navier-Stokes equations and solve the governing equations with various PDE-solving networks, which focuses on the physical properties of crowds. For this kind of method, we explore different types of PDE-solving networks, including PDE-Net (data-driven), NSFnets [21] (physics-informed), and the advanced PPNN. In order to conduct the comparison, we apply our HINN along with these five simulation models to simulate dense crowd motions under six classic motion patterns and evaluate them with IS, FID and SSIM.According to the data presented in Tab.2, our proposed HINN outperforms all the other simulation models in general, achieving remarkable results in terms of IS, FID and SSIM. Considering classic crowd simulation models, namely Boids and SFM, we can see that they exhibit mediocre performance across all evaluated metrics, which again demonstrates that microscopic models are not suitable for simulating dense crowds. PDE-Net suffers from mode collapse that it fails to generate crowd motions under "cross" and "scatter" patterns, which contributes to its poor performance on groundtruthrelated metrics such as FID and SSIM. NSFnets performs slightly better than our HINN on SSIM, but shows disastrous performance on IS and FID, indicating its inadequacy in authenticity, fidelity and diversity. Closest to our HINN, PPNN shows better performance on IS but falls behind on FID and SSIM, suggesting that while it effectively captures the crowd physical properties like continuity and fluidity, it struggles to maintain authenticity and fidelity in the simulation. Compared to these models, our HINN excels in terms of FID and are able to achieve comparable performance on IS and SSIM relative to the best-performing models.In order to compare the performance of the simulation models more intuitively, we present the groundtruth and corresponding simulations using heat maps. As shown in Fig. 3 , the groundtruth is generated from a dense crowd video that contains crowd motion under a "curve" pattern. Given the initial state u 0 , Boids, SFM and PDE-Net can barely simulate the crowd motion after 50 time-steps. NSFnets merely restores the approximate overall shape, while overlooking finer details of dense crowd motions. The performance of PPNN at time-step 50 is actually acceptable. However, it appears that the simulation of PPNN remains relatively unchanged over time, resulting in a larger error at time-step 99. Compared to these simulation models, our HINN successfully simulates the entire process of this "curve" motion, pays attention to intricate details of crowd motions, and demonstrates its adaptability over time. The simulations of more crowd motions under different motion patterns can be found in the supplementary.In our governing equation for dense crowd simulation, we consider both the physical properties and social properties of crowds. To investigate the contribution of these two kinds of properties in the dense crowd simulation, ablation studies are conducted for physical and social operators. (1) w/o physical operators. We erase the physical operators (𝑓 𝑐𝑜𝑛 and 𝑓 𝑣𝑖𝑠 ) from our governing equation, relying solely on the operators that highlight crowd characteristics distinct from fluids. (2) w/o social operators. We exclude the social operators (𝑓 𝑎𝑙𝑖 , 𝑓 𝑛𝑎𝑣 and 𝑓 𝑐𝑜ℎ ) from our governing equation. In this circumstance, dense crowds are treated purely as fluids, neglecting the influence of crowd social behaviors. (3) w/o physical operators and social operators, i.e. w/o HIM. We remove the plug-in HIM which preserves our governing equation. In this scenario, our HINN reduces to a ConvResNet.From data in Tab.3, it can be seen that after removing the physical and social operators from our governing equation, the simulation performance degrades across all evaluated metrics. Among all these models, HINN incorporated with physical operators demonstrates the highest similarity to our HINN in terms of SSIM, indicating the superior effectiveness of physical properties in maintaining structural integrity of crowd motions. HINN utilizing the social operators exhibits the closest alignment with our HINN based on FID and the average loss, underscoring the superiority of social properties in enhancing the fidelity and generating more realistic simulations. Additionally, either the physical operators or social operators may individually introduce a decrease in IS, but when we combine these two kinds of operators together in our governing equation, HINN achieves the best performance. To show influence of physical and social operators on dense crowd simulation more clearly, we exhibit a comparison of simulating crowd motion under a "scatter" pattern using different models as shown in Fig. 4 . We can observe that with physical operators (third and fifth rows), HINN better restores the overall motion and continuity of crowd; while with social operators (fourth and fifth rows), HINN can capture more finer details of crowd motions. Containing both physical and social operators, our HINN delivers the most comprehensive simulation with minimal average loss. Numerous experiments have demonstrated that our proposed model outperforms other simulation models and have shown the reliability of generated dense crowd motions. In future, we will continue to construct a more comprehensive dense crowd dataset with more video clips and motion patterns. We will also explore more physical and social operators which can reflect crowd properties so that we can simulate more complicated crowd motions. https://www.gettyimages.com 
 
 Appendix:  


Decision: This paper originally received 2 BA and BR, which became WA and 2 BA after rebuttal.

Main concerns mainly regarded confusing methodological parts to be clarified, weak novelty, missing discussion/analysis of similar comparative approaches, lack of discussion about ethical issues, insufficient experimental analysis and comparisons, insufficient analysis of the limitations of the proposed approach, and, more in general, other issues regarding weak or lack of comments or discussion and justifications of some methodological choices. Moreover, since a new dataset is also introduced, some remarks addressed the quality of this new data in terms of control parameters of such data.

Despite the number of criticisms, reviewers appreciate this work and the rebuttal satisfactorily addressed the majority of the raised remarks. For these reasons, this paper is considered acceptable for publication to ACM MM 2024.

Reviews: review: paper_summary: paper_summary: This paper introduces a Hydrodynamics-Informed Neural Network (HINN) for simulating dense crowd motion patterns. It addresses challenges in dense crowd simulation: the nonlinear complexity of directly solving Navier-Stokes equations, the oversight of crowd characteristics distinct from fluids, and the lack of robust evaluation for crowd simulation models. To overcome these, it integrates physical and social properties in the Navier-Stokes equation and introduces corresponding operators into the PDE-Solving neural network HINN. The HINN preserves the governing equation structure in its architecture, enhancing computational efficiency. A new real-world dense crowd motion video dataset is created for evaluation. New objective metrics about authenticity, fidelity, and diversity are employed, including IS, FID, and SSIM. Experiments demonstrate the superior performance of HINN in simulating dense crowd motions compared to other models.
strength_weakness: strength_weakness: 1. This paper clearly describes the traditional Navier-Stokes equations-based modeling of crowd dynamics and highlights its limitations and challenges, explaining the necessity of incorporating individual and social factors for crowd movement into the hydrodynamics equation.

2. The introduction of calculation elements for the velocity field in the governing equation is innovative. By introducing meticulously designed forces for *Alignment, Navigation, and Cohesion* on top of the existing *Convection and Viscosity* forces, the social properties of crowd movement are effectively represented.

3. This work contributes a new crowd motion video dataset, providing rich video data and velocity field statistics for various dense scenarios.
limitations: limitations: 1. Subjectively, I find the methodological innovation of this paper to be somewhat lacking. The core HINN framework proposed in the text merely follows the PPNN scheme, with contributions focused on replacing the physical function with Navier-Stokes equations and introducing new social forces.

2. Equations 2, 3, and 4 in section 3.2 use pressure force, but it is not mentioned in subsequent parts. The paper should explain why this term from the Navier-Stokes equation was not applied.

3. There is a lack of discussion comparing the proposed method to deep learning crowd simulators. Although the paper compares with basic models like SFM, it should also discuss comparisons with state-of-the-art physics-informed deep learning models (e.g., [1][2]) to fully demonstrate the paper's opinion that microscopic models are not suitable for simulating dense crowds.

4. There are some textual errors in the paper: 1) In Figure 2, 'upsampling' should be 'bicubic'; 2) the definitions of N and M in equation 18 are not clearly provided.

[1] Social physics informed diffusion model for crowd simulation, AAAI24

[2] Human trajectory prediction via neural social physics, ECCV22

Also, I have remaining **questions** listed as below:
a. With a fixed equation form and a single given initial velocity, how does the predictive HINN generate diverse (i.e., different pattern) flows? What accounts for the good performance of IS?

b. What is the significance of evaluating IS? This metric is not related to the optimization objective in the problem formulation. It does not measure the distance between predicted and real samples. Shouldn't more general distribution metrics like MMD be considered?

c. Does the Canny edge detector operate on the velocity field or frames?

d. What does the 'average loss' in the ablation study refer to?

e. I do not find links to code and data. Would it be open-released?
paper_summary: paper_summary: This paper proposes a new hydrodynamic model based on Naviser-Stokes equations and introduces a new dense crowd motion video dataset for crowd simulation. The hydrodynamic model can effectively capture physical and social properties by incorporating additional forces and achieve promising results on the new dataset.
strength_weakness: strength_weakness: 1.	The authors contribute the largest dense crowd motion video dataset.
2.	The authors use more objective metrics to evaluate the performances of models. 
3.	The structure of this paper is quite comprehensive.
limitations: limitations: 1.	In the visualizations, when highlighting areas of improved performance in the color boxes, it would be helpful to understand.
2.	Do you have more detailed parameters of the dataset, in terms of the number of crowds, the diversity of the scenes, and so on?
3.	The paper does not explicitly discuss ethical considerations. While there may not be direct ethical implications, discussions on responsible data use and potential biases in crowd simulation algorithms would strengthen the paper.
4.	In Table 2, the proposed method performs slightly better than PPNN, but it is difficult to prove the effectiveness of operators that describe crowd interactions and crowd-environment interactions. Have you conducted more experiments on other existing datasets?
5.	Have you conducted simple cross-scene experiments on the proposed algorithms?
paper_summary: paper_summary: Dense crowd simulation is crucial in addressing the increasing frequency of crowd crushes. In this research, the authors aim to simulate dense crowd movements under six motion patterns using an initial state. Drawing parallels with fluid dynamics, they propose a hydrodynamic model that captures both the physical and social properties of crowds through Navier-Stokes equations and crowd-interaction operators. To address computational challenges, they introduce the Hydrodynamics-Informed Neural Network (HINN), which solves the governing equations. Furthermore, the authors construct a new dataset, DCFD, to facilitate the evaluation of their approach.
strength_weakness: strength_weakness: 1) Unique integration of hydrodynamics and neural networks for crowd simulation.
2) Solid mathematical basis using Navier-Stokes equations and social interaction operators.
3) The proposed HINN effectively solves the equations, addressing computational challenges.
4) New dataset, DCFD, enables comprehensive testing of the model.
5) Has the potential to improve crowd management, safety planning, and emergency response.
limitations: limitations: 1) While a new dataset, DCFD, is introduced, the evaluation may be insufficient in terms of variety of scenarios, crowd densities, and comparison to state-of-the-art methods.
2) The paper may fail to adequately discuss potential limitations of the proposed approach, such as its performance under extreme conditions or generalization to real-world scenarios.
3) The choice of 6 patterns lacks justification compared to the 8 patterns in [44].
4) The rationale for using CNN as the baseline is unclear, especially given the advantages of Transformers.
5) References for comparison methods in Table 2 are missing, making it difficult to locate original literature.
6) The paper does not provide visualizations to illustrate the unique challenges and characteristics of the proposed new dataset. Without such visualizations, it is difficult for readers to fully understand the motivation for developing a new dataset and how it differs from existing ones.