This work presents a physics-informed deep learning-based super-resolution framework to enhance the spatio-temporal resolution of the solution of time-dependent partial differen- tial equations (PDE). Prior works on deep learning-based super-resolution models have shown promise in accelerating engineering design by reducing the computational expense of tradi- tional numerical schemes. However, these models heavily rely on the availability of high- resolution (HR) labeled data needed during training. In this work, we propose a physics-informed deep learning-based framework to enhance the spatial and temporal resolution of coarse-scale (both in space and time) PDE solutions without requiring any HR data. The framework consists of two trainable modules independently super- resolving the PDE solution, first in spatial and then in temporal direction. We analyze the capability of the developed framework by investigating its performance on an elastodynamics problem. It is observed that the proposed framework can successfully super-resolve (both in space and time) the low-resolution PDE solutions while satisfying physics-based constraints and yielding high accuracy - all without requiring any HR data. Furthermore, the analysis and obtained speed-up show that the proposed framework is well-suited for integration with traditional numerical methods to reduce computational complexity during engineering design. 1 Introduction Accurate modeling of the dynamic behavior of nonlinear systems is crucial for many industrial applications ranging from microscale MEMS sensors to large-scale structural systems. Therefore, significant research is being done to understand and resolve the complex physical phenomena oc- curring within these dynamical systems at extremely small spatial and temporal scales. This scien- tific pursuit of capturing complex physical phenomena occurring at widely varying spatio-temporal scales has led to the ever-increasing sophistication of the physical system’s governing Partial Dif- ferential Equations (PDEs). For example, a PDE-based model [AA20a, AA20b, JABG20, AZA20, *Senior member of technical staff at Advanced Micro Devices, Inc. (AMD). †Post Doctoral associate at Sandia National Laboratories. Accepted at AAAI 2023: Workshop on AI to Accelerate Science and Engineering (AI2ASE) 1 Temporallysuper-resolved Low-resolution InputSpatially super-resolvedtƚнȴƚƚнȴƚt Spatial Resolution Enhancement module TemporalResolution Enhancement module ƚнȴƚtFigure 1: The figure shows the structure of the proposed framework. The input to the framework is a two-channel image of the coarse-scale solution, and the final output of the framework is a ( k+ 1) channel image of the super-resolved solution. The framework consists of a spatial and a temporal resolution enhancement module. The former performs super-resolution in space, whereas the latter super-resolves in the temporal direction. The modules are trained independently. Aro19, AAA22] capturing defects evolution in materials at the nanoscale has been shown supe- rior to conventional theories for a wide range of applications. However, the massive data storage and computational expense requirements to simulate such multi-physics coupled PDEs with high- fidelity bring traditional numerical solvers to their limits. Hence, fast and accurate techniques to perform these multi-physics simulations at multi-scale are of utmost importance. On the other hand, the recent advances in Machine Learning (ML) have led to the develop- ment of several data-driven and Physics Informed ML models to solve PDEs occurring in fluid [SGPW20, RSL20, JCLK21] and solid mechanics [FTJ20, AKDC22, SLDN22, ZLY21]. How- ever, issues ranging from its theoretical considerations (such as convergence, stability, accuracy, and generalizability) to issues related to boundary conditions, neural network architecture de- sign, or optimization aspects still need to be fully resolved [Mar21, CDCG+22]. Therefore, hy- brid strategies integrating physics-informed ML with traditional approaches are emerging as a promising option to tackle this computational challenge of solving complex multi-physics PDEs [Aro21, Aro22b, GSW21]. To this end, in this research, we aim to investigate a two-stage hybrid approach integrating ML and traditional approaches to obtain (reconstruct) solutions to spatio-temporal PDEs. 1) In the first stage, low-resolution (LR) PDE solutions are obtained by doing numerical simulations on a coarse scale both in space and time (using large grid size and timestep). This low-resolution solution with satisfactory accuracy can be generated with a huge reduction in computational expense compared to solving PDE on a fine scale. 2) In the second stage, the spatio-temporal resolution of this coarse- scale solution is enhanced using a physics-formed deep learning-based framework. A significant advantage of such ‘physics-guided resolution enhancement’ approach is the reduced computational expense and data storage requirements during the scientific exploration phase, which will signifi- cantly accelerate the process of scientific investigation and engineering design. This enhancement in resolution will also be referred to as the upsampling or super-resolution (SR) in this work. The recent works involving spatio-temporal super-resolution of physical systems [RRL+22, 2 EAK+20, FFT21] use labeled high-resolution (HR) ground truth data for model training. Fukami et. al [FFT21] presents a purely data-driven SR framework, and therefore the super-resolved fields may not satisfy the physics-based constraints accurately. The works of Ren et al. [RRL+22] and Jiang et al. [EAK+20] are ’almost data-driven’ in that the scaling coefficient of prediction/data loss is chosen to be 20times the coefficient of physics loss in the total loss for optimal accuracy. In fact, the errors are huge when HR-labeled data is not taken into account (purely physics-driven) [EAK+20, see Table 1]. Moreover, the HR-labeled data is computationally expensive to obtain. Therefore, its use during training completely negates the massive benefit of accelerating scientific computing that ML models aim to achieve. In this research, we present an end-to-end physics- informed deep learning-based framework to enhance the spatial and temporal resolution of coarse scale (both in space and time) PDE solutions without requiring any HR-labeled data. In summary, our main contributions are as follows: 1. We propose a novel and efficient physics-informed deep learning-based spatio-temporal res- olution enhancement framework. 2. The framework consists of two trainable deep learning modules independently responsible for spatial and temporal upscaling of the coarse-scale PDE solution. 3. Unlike other works [EAK+20, FFT21, RRL+22], the proposed framework does not rely on the availability of any high-resolution labeled data. 4. The effectiveness of the framework is tested by using the low-resolution coarse grid simula- tion data as input as opposed to using downsampled high-resolution labeled data. Organization. The remainder of this paper is organized as follows: Section 2 briefly recalls the governing equations of a general spatio-temporal PDE. Section 3 present the details of the frame- work architecture, data setup, and composite loss function. Towards the end, Section 4 presents the results that validate the developed framework and demonstrate its effectiveness in super-resolving the solution fields for the test problem discussed. Finally, conclusions and avenues for further re- search are briefly discussed in Section 5. 2 Background 2.1 Governing equations A typical spatio-temporal PDE governing the dynamical systems can be written in the following form: ˙z−F(z,x, t;x∈Ω) =0, (1) subjected to the initial and boundary conditions I(z;t= 0,x∈Ω) =0, B(z;x∈∂Ω) =0.(2) 3 In equations 1, 2, zdenotes the system solution comprised of mstate variables, and ˙zdenotes its time derivative. Fis the nonlinear functional of the polynomial and derivatives terms of its arguments. Ωand∂Ωdenotes the physical domain and its boundary, respectively. Any additional constraints which are inherently present in the system or required because of numerical methods, such as mixed finite element methods, can be assembled into C(z) =0. Given a solution zcwhich is obtained by solving the system of equations (1)-(2) at a coarse scale (large mesh size and timestep), our objective is to enhance the spatial and temporal resolution of the solution by using a physics-informed deep-learning based method. The schematic of our proposed framework is shown in Figure 1. 3 Methodology Section 3.1 briefly outlines the physics-informed composite loss (objective function) used during the framework’s training. We then present an overview of the “end-to-end” spatio-temporal reso- lution enhancement framework in Section 3.3. 3.1 Objective function We note that the framework proposed in this work is unsupervised and therefore the composite loss function is obtained only from the governing equations of the system - Initial conditions, boundary conditions, and PDEs. Following [Aro22b, Sec. IV], we impose the boundary conditions in the ‘hard’ manner (exactly), thus eliminating the boundary condition loss contribution from the composite loss. The physics-informed objective function is then written as follows: L=λ1 ||˙z−F(z,x,t )||1| {z } PDE + λ2||C(z)||1|{z} Constraints+λ3||I(z)||1|{z} I.Cs.,(3) where ||A||1denotes the mean absolute error (MAE) between each element in the quantity Aand target 0. We use a fourth-order finite difference scheme to evaluate the spatial derivatives of the solu- tion over the grid. For time discretization, we use the Crank-Nicholson algorithm, which has the virtues of being unconditionally stable and is also second-order accurate in both space and time dimensions. 3.2 Input and output for the framework The input to the framework consists of tuples of LR coarse-scale PDE solution It c={zt c,zt+∆t c} at the consecutive timesteps tandt+ ∆t, respectively. The outputs from the framework consist of the spatially upscaled PDE solution at the same timesteps {ˆzt,ˆzt+∆t}along with the synthesis of 4 Model 1 Model mz1 zmˆz1 ˆzmL</epsilon1 Stop NoYesInput stateOutput stateSpatial/Temporal module TrainFigure 2: The figure shows the structure of the spatial/temporal module. Each module consists of mdeep-learning models independently super-resolving the state variables {z1, z2, . . . , z m}. The models are, however, trained using a coupled loss function. the HR snapshots at (k−1)intermediate timesteps {ˆzt+∆t k,ˆzt+2∆t k, ...,ˆzt+(k−1)∆t k}. Therefore, the framework produces spatially upscaled PDE solution at (k+ 1) timesteps referred to Ot. We refer to the scalar kas the temporal upscaling factor and is given as the ratio of the coarse-scale timestep to the fine-scale timestep i.e. k=∆tc ∆tf. Similarly, the upscaling factor in the spatial direction sis given as the ratio of the coarse to fine grid resolution, i.e. s=∆xc ∆xf. 3.3 Framework Architecture The framework is composed of two trainable modules: the Spatial resolution enhancement mod- ule and the Temporal resolution enhancement module, as shown in Figure 1. These two modules independently perform the super-resolution in space and time, respectively. We observed that this dual module approach which first performs super-resolution in space and subsequently increases temporal resolution leads to better convergence and accuracy of the super-resolved fields as is also observed in the data-driven approach of Fukami et. al [FFT21]. A typical solution zof any dynamical system consists of mstate variables. Therefore, each module in the framework consists of mdeep learning models (with the same architecture) that individually reconstruct each state variable; refer to figure 2. These models, however, are coupled during the training through the objective function (loss). Sections 3.3.1 and 3.3.2 discuss these spatial and temporal super-resolution modules in greater detail, respectively. 3.3.1 Spatial resolution enhancement module Given a tuple of LR snapshots at two consecutive time steps It={zt c,zt+∆t c}for the state vector z, the spatial upscaling module outputs the corresponding HR frames {ˆzt,ˆzt+∆t}representing the enhanced spatial resolution of the input state at the original time steps. Therefore, the input (output) to this module consists of a two-channel image representing the values of the LR (HR) state variables at time steps tandt+ ∆t. 5 Spatial Module!"!!"!"#!!"!$#!!"!Spatially Resolved OutputsLR Inputs"!$#!"!"!"!"#!Figure 3: The two dashed vertical (horizontal) boxes show the use of state variables from different (same) outputs to calculate the time derivative in PDE loss. Temporal Module"!$#!"!$%.'#!"!Spatio-Temporally Resolved OutputsSpatially ResolvedInputs"!"!"#!"!$#!"!"!$#!"!"%.'#!"! Figure 4: The four horizontal dashed boxes show the state variables within the same outputs while calculating the time derivative. The two inclined boxes highlight the states from different outputs used to calculate the time derivative. During training, we observe that for the successful evolution of solution from initial conditions in both the output channels, the PDE loss in (3) has to be implemented both within an output and across outputs as highlighted in Figure 3. This coupling in the loss helps in mitigating the propagation failure mode [DBW+22] for this module. Each model in the spatial upscaling module is built upon the Residual Dense Network (RDN) proposed in [ZTK+18], which has unique advantages for image SR over other networks [ZTK+18, Sec. 4]. We use 4residual blocks with 8layers in each block and a feature channel size of 32. The kernel size for convolution is set to be 3. 3.3.2 Temporal resolution enhancement module The temporal module enhances the resolution of the state variables in time. The output of the spatial module serves as an input to the temporal module. Therefore, the input is a two-channel image representing the values of the HR state variables at time steps, t, and t+ ∆t. The module outputs an image with (k+1) channels, representing the state variables at time steps {t,t+∆t k, . . . , t+ ∆t}. We note here that the inputs to the temporal module still have the temporal discretization error of the order O( ∆t2). Therefore, the temporal module also reconstructs the solution at the initial and the final times tandt+ ∆t, respectively, to reduce this error to O∆t k2 . We make 6 the following changes to the composite loss (3) for the training of this module: • Similar to the implementation of PDE loss in the spatial module, the PDE loss for the tem- poral module is also implemented both within an output and across outputs, as highlighted in Figure 4. • Since we reconstruct the outputs at initial and final timesteps ( tandt+ ∆t), we add a constraint loss between these inputs and outputs, which helps in faster convergence of the module. For the sake of simplicity, the model architecture is similar to the spatial module except for the number of channels in the output and the elimination of upsampling layer. 4 Experiment In this section, we briefly discuss the problem setup, dataset generation, and evaluation metric. We then evaluate the performance of the developed framework by investigating its effectiveness in enhancing the spatio-temporal resolution of the coarse-scale solutions to an elastodynamics problem. The results presented herein demonstrate the remarkable accuracy of the network without requiring any HR-labeled data, unlike all previous works. 4.1 Setup We consider a mixed-variable elastodynamics system widely used in structural engineering and seismologic applications. The governing equations of the system (in the absence of inertia) under 2-d antiplane strain conditions are given as follows: v= ˙u ∂σxz ∂x+∂σyz ∂y=ρ µ˙v σxz=µ∂u ∂x;σyz=µ∂u ∂y,(4) where ρis the material density, bis the body force per unit volume, µdenotes the shear modulus of the material, and uandvdenote the displacement and material velocity, respectively. σxzandσyz denote the nonzero components of the (symmetric) stress tensor σ. The equations 4, along with the boundary and initial conditions in the equations 5, define the governing equations of the system. u(t) =ubcon∂Ω, u(t= 0) = u0andv(t= 0) = v0.(5) In the above, u0andv0denote the known initial conditions on uandv, respectively. ubcdenote the known displacement on the domain boundary ∂Ω. Without loss of generality, we take v0=ubc= 0 in this work. The initial condition for uisu0= sin( πx) sin(πy). 7 4.2 Generation of low-resolution input data To evaluate the performance of the framework, we generate the low-resolution input data for the problem setup presented above in Section 4.1. We solve the governing system of equations (4) for the non-dimensional values of stress, displacement, and velocity, which amounts to setting ρ µ= 1in (4). The equations are solved on a coarse mesh of 64triangular elements ( 41nodes and ∆xc≈0.176) and coarse-scale timestep ∆tc= 0.005with finite element method (FEM) for 48 timesteps. The obtained coarse grid solution is then interpolated to a structured 32×32grid using the FEM interpolation and is used as an input to the super-resolution framework. For comparison, the HR ground truth data is obtained by solving the same equations on a fine mesh of 64×64 nodes and ∆xf=.0158 with∆tf=∆tc 2. This sets the spatial and temporal upscaling factors to be s= 11 andk= 2, respectively, for the test case discussed. 4.3 Evaluation metric For any state variable α(one of u, v, σ xz,orσyz), we define a full field error measure eto quan- titatively measure the discrepancy between the HR ground truth data αHRand the framework predictions ˆαas follows: e=||αHR−ˆα||L2 ||αHR||L2×100. (6) We note here that the HR ground truth data is only used for comparison with the predicted outputs of the framework. 4.4 Training The framework is implemented and trained using PyTorch. The training strategy used in this work consists of two stages: a) The spatial module is trained in the first stage, which amounts to en- hancing spatial resolution. b) The outputs from the trained spatial module are then used as input to the temporal module during its training. The weights for the spatial module are frozen during this stage. In both stages, we use Adam optimizer with a learning rate of 4×10−4for around 2000 epochs with a batch size of 8samples. As the training progresses, the learning rate is adjusted using ReduceLROnPlateau scheduler with patience set to 40. We use a sequential training data sampler to respect the causal structure inherently present in the spatio-temporal PDEs, which has been shown to improve the accuracy of the physics-informed neural networks significantly, refer [WSP22]. We use the scaling coefficients λ1= 5,λ2= 1, and λ3= 10 in this work. 4.5 Performance Evaluation This section focuses on the performance evaluation of the proposed framework. The low-resolution coarse-scale PDE solution generated in Section 4.2 is used as an input to the framework. In what 8 (a) Proposed framework. (b) Bi-linear interpolation. Figure 5: The figure shows the error for the solution fields of elastodynamics at different time steps after super-resolution using the proposed framework, Figure 5a, and using bi-linear interpolation, Figure 5b. The super-resolution framework is highly accurate as compared to simple bi-linear interpolation. The framework’s accuracy suffers beyond time, t= 0.2438 (extrapolation region), as the framework is trained till t= 0.2438 . The high error in the extrapolation region suggests that the framework is capable of producing high-accuracy results for super-resolution within the convex hull of training set. follows, we compare the error measure efort≥0.02because the initial velocity condition is zero in this work. Figure 5a shows the error measure efor all the HR state variables obtained from the framework. We note that the %error eis less than 4%for all the state variables at all times t≤0.24. We also note that the framework exhibits poor extrapolation capabilities as the errors become quite large aftert >0.24. Figure 5b shows the error measure efor the state variables when a simple bi-linear interpolation of the low-resolution data is performed to upscale the solution. We can see that the errors are relatively large for all of the state variables. Figure 6 shows the super-resolved state variables obtained from the framework at a particular time t= 0.14. We can notice that the super-resolved fields are indistinguishable from the HR ground truth data. At the same time, the simple bi-linear upscaling of the LR input data can be seen to significantly differ from the ground truth reference data. Therefore, we conclude that the proposed framework successfully enhanced the spatial and temporal resolution of the solution by a factor of 11and2, respectively. 4.6 Speed-up Next, we calculate the speed-up obtained using the proposed framework to upsample the coarse- scale solution compared to obtaining the fine-scale solution using FEM. The FEM calculations are performed on a single-core AMD EPYC 7742 Processor. For the same simulation end time, the coarse-scale simulation ( ∆xc≈0.176and∆tc= 0.005) takes 0.033seconds, whereas it takes 4.27 seconds to run the fine-scale simulation ( ∆xf≈0.0158 and∆tf= 0.0025 ). The above numbers are averaged over 10runs of the entire simulation. They do not include the time taken for mesh 9 LR input HR Ground Truth Framework outputs Bi-linear interpolation Figure 6: The figure shows the snapshot of the elastodynamics’ solution fields at a time step. The LR input and HR Ground-truth are the coarse-scale and fine-scale solution fields obtained using the finite element method. For a given LR input, the reconstructed solution from the framework is highly accurate compared to the bi-linear interpolation method. generation, node numbering, memory allocation, data I/O, or any other bookkeeping required by FEM. On the other hand, after training, the inference time (averaged over 100inferences) for the spa- tial and temporal modules are 0.015and0.017seconds, respectively. Therefore, using the proposed framework, it takes around 1.569seconds (including time taken for the coarse-scale simulation) to obtain the fine-scale PDE solution for the same simulation end time. Thus, we obtain a speed- up factor of around 2.72for the (relatively) simple test case discussed here. These inferences are performed on an NVIDIA Tesla V100-SXM2 GPU with 32GB RAM. We emphasize here that the speed-up factor strongly depends on the complexity of the problem (linear vs. nonlinear) and the framework’s spatial and temporal upscaling factors. 10 5 Conclusion In this work, we presented a novel unsupervised physics-informed machine learning framework, a first in the literature, that: • enables spatial and temporal upscaling (resolution enhancement) of coarse-grained solutions to spatio-temporal PDEs while ensuring that the (upscaled) outputs satisfy the governing laws of the system. • easily allows imposition of any additional constraints (PDE or algebraic) in the framework. • is generalizable to non-rectangular domains by using elliptic coordinate transformation as outlined in [GSW20]. • is amenable to scalability to clusters with multi-gpu nodes using DistributedDataParallel functionality in PyTorch [LZV+20] functionality for workloads that require substantial com- putational resources. We demonstrated the framework’s application to an elastodynamics problem under anti-plane strain conditions. The framework successfully enhanced the spatial and temporal resolutions of the coarse-scale input fields by a factor of s= 11 andk= 2 in the space and time directions, respectively, while satisfying the physics-based constraints and yielding great accuracy (error ≤ 4%). In the future, we aim to study the framework’s application on a wide range of physical ap- plications in fluid mechanics and compare the performance of the current framework with other super-resolution works (although supervised) in the literature [RRL+22, EAK+20]. Another inter- esting line of research to pursue would be the addition of ConvLSTM [SCW+15] in the temporal module to improve the predicting capabilities of the framework beyond the convex hull of the traning set. Research Data The low-resolution simulation data used in this work has been generated using Fenics [ABH+15]. The source code for the proposed framework and the dataset used in this research can be found at [Aro22a] upon acceptance of this paper. Acknowledgment The work was conceptualized during the authors’ time at Carnegie Mellon University (CMU). We gratefully acknowledge the Pittsburgh Supercomputing Center (PSC) for providing the computing resources that contributed to the research results reported in this paper. 11