To mitigate global warming, greenhouse gas sources need to be resolved at a high spatial resolution and monitored in time to ensure the reduction and ultimately elimination of the pollution source. However, the complexity of computation in resolving high-resolution wind fields left the simulations impractical to test different time lengths and model configurations. This study presents a preliminary development of a physics-informed super-resolution (SR) generative adversarial network (GAN) that super-resolves the three-dimensional (3D) low-resolution wind fields by upscaling x9 times. We develop a pixel-wise self-attention (PWA) module that learns 3D weather dynamics via a self-attention computation followed by a 2D convolution. We also employ a loss term that regularizes the self-attention map during pretraining, capturing the vertical convection process from input wind data. The new PWA SR-GAN shows the high-fidelity super-resolved 3D wind data, learns a wind structure at the high-frequency domain, and reduces the computational cost of a high-resolution wind simulation by x 89.7 times. Accurate tracing and monitoring of greenhouse gas (GHG) sources is a key measurement to take action for tackling the mitigation of global warming issues [3]. The higher spatial resolution of wind simulation enables precise tracking of GHG emissions from potential sources, and helps decision-making in various fields such as policymakers, agriculture, and renewable energy. However, high-resolution simulation is not always available due to the demands of computational resources. Artificial Intelligence (AI), including deep neural networks (DNNs), can reduce the computational cost by upscaling the wind fields from low-resolution (LR) to high-resolution (HR) data. Superresolution (SR) is one of the solutions to achieve the goal: conventional SR techniques rely on convolutional neural networks (CNNs) to reproduce realistic HR wind fields [12,14,7,9,15,1]. However, these CNN-based approaches either perform on 2D data or fall short of capturing the 3D dynamics in weather systems because a convolutional kernel truncates the association between vertical layers, limiting to learning of convection and diurnal cycles induced by incoming solar radiations. Another issue is that widely used weather simulations employ a non-uniform grid in a vertical axis for computational stability. The difference in height between a pair of two adjacent vertical layers becomes larger at higher altitudes. Thus, CNN is not sufficient to capture physics at the same vertical scaling. To address these issues, we develop a prototype of a novel architecture of physics-informed neural networks that learn multi-scale spatial dynamics. Our goal is to develop a physics-informed neural network that super-resolves 900 m resolution of three-dimensional wind data into 100 m scale high-resolution wind data and captures three-dimensional dynamics in weather systems. We develop a pixel-wise self-attention network (PWA) to learn the three-dimensional dynamics of weather simulations. Using the neural network as a generator, we train a generative network for super-resolving wind velocity fields. The Weather Research & Forecasting Model (WRF) [11] has been widely used for simulating weather systems from mesoscale to turbulence scales. Large-eddy simulation (LES) is often nested into the WRF framework to allow simulating a convective process to capture a more realistic turbulence structure [2]. Outputs from WRF-LES used for this study are nested in 900 m, 300 m, and 100 m horizontal resolution. The outer 900 m and inner 100 m simulations differ in their model physics and our ultimate goal is to super-resolve 900 m LR data to 100 m HR data. This study, however, focuses on synthesizing the 900 m LR data by spatially averaging 100 m HR data as a preliminary step. Our training and testing data are produced by a WRF nested LES at 100 m horizontal scale and 59 vertical layers, generating outputs every five minutes from 00 UTC to 23:59 UTC. The initialization time window is set for two hours starting at 22:00 UTC a day before. We sample training and testing data randomly from one-month simulation outputs from September 2019. We select three wind velocity fields (U, V, W), all at 100 m spatial resolution over 40 km x 40 km simulation domain, and use only 8 layers from the surface level. We then extract 22 500 training and 2500 testing data, as the smaller geographical images, each of which is 126 pixels x 126 pixels ( 12.6 km x 12.6 km) for high-resolution (HR) data. To synthesize low-resolution (LR) data, we average every 9 x 9 pixels, giving 900 m LR data. A standard convolutional filter may truncate these signals and be suboptimal to the non-uniform vertical grid system in numerical models for capturing information at the same scale. We introduce a self-attention computation [13] at each grid column to better embed nonlinear association between vertical layers into networks: self-attention inherently computes attention scores for each element in sequences, enabling the representation of signal associations between a given vertical layer and others. This versatility extends beyond the constrains of a convolutional kernel that limits associations solely to neighboring layers above and below. where e v is a dimension of intermediate representation and we depict 64 for the dimension. The learned self-attention map M shown in Figure 1b shows strong signals in diagonal and near-diagonal elements, indicating that adjacent layers have a stronger effect on each other. In particular, near-surface layers (i.e., left top corner of M ) exhibit strong attention in wider elements, and this is associated with daytime convection within the ABL. Effective learning is achieved through the sparsity of weak signals as well as emphasizing key elements in the selfattention map M depicted in Figure 1b. To do so, we add the regularization of M as one of the loss terms as follows: where M ij = softmax(QK T /  e v ) and R depicts a regularization term. The minimization of R(M ) in Equation 2indicates the elements of strong attention get larger scores as well as weaker attention reduces the scores. One limitation of this approach is that the range is bounded in [0, 1] at each row. To have more attention to strong signals, and in opposite, suppress the weak ones, we introduce a re-scaling scheme to the self-attention map.
Simply, we have a trainable matrix , adjusting the value of self-attention map M by M rescale =   M . Note that we apply the self-attention map regularization scheme to pre-rescaled M . where L content represents mean squared error x LR X LR ,x HR X HR (G(x LR )  x HR ) 2 ; G() depicts the outputs of our neural network. We have three different weight coefficients:  R ,  G , and  Gv balances self-attention regularization, horizontal gradient, and vertical gradient terms. We use ( R ,  G ,  Gv ) = (0.01, 50, 100) in this work. The training uses Adam optimizer [6] with the learning rate at 1e  3 on an A-100 GPU for 200 epochs. CNN-based neural networks are known for filtering low-frequency data in input images and removing high-frequency information [10]. Generative Adversarial Networks (GANs) [4] are our solution to incorporate them into super-resolved images because our training input data x LR has already smoothed out high-frequency information in high-resolution image x HR through averaging the data over 9 x 9 pixels. Thus, it is challenging to learn the high-frequency data only from low-resolution training inputs. The generator loss in a GAN is typically defined as the negative log-likelihood of the discriminator's output when the generator tries to generate realistic data. The loss term L adversarial is often represented as: where x  is a set of low-resolution wind data sampled from a distribution of low-resolution images; G(x  ) is a super-resolved data by the generator; D(G(x)) is a discriminator's output for highresolution data x; and p(x  ) is a prior distribution of x  . After pre-training PWA-SR network with Equation 4, we modify the loss function for training a generator by adding an adversarial term into the combined loss function; For discriminator, we follow the same architecture based on Ledig et al. [8]. See Ledig et al. [8] for further details on the training of GAN. 3 Super-resolution of 3D wind data We first investigate whether PWA SR-GAN (hereafter SR-GAN) can generate wind filed data that closely approximates the ground-truth HR image in frequency space. We apply the fast Fourier transform to our 2500 test samples for calculating the power spectrum of images instead of mean squared errors (MSEs) because MSEs do not always evaluate if SR models restore high-frequency mode. That is, the test measures how well the model learns the high-frequency mode of wind data through GAN training. Figure 3 shows the results of the power spectrum analysis as a function of wave numbers for U, V, and W wind data respectively. The results compare the power spectrum among different approaches: HR data, bicubic interpolation, SR-PWA, and SR-GAN. The fidelity of capturing realistic wind structures improves as their results of the power spectrum align more closely with HR data (black line). Note that a conventional residual CNN yields a result similar to that of SR-PWA (not reported in this paper). We see that the SR-GAN approach (blue line) significantly outperforms the SR-PWA (green line) for learning high-frequency domains, especially in vertical wind W. However, there is a performance trade-off in low to medium-wave numbers when the learning high-frequency domain achieves well. As part of a qualitative evaluation, we present a snapshot example of wind data from the first layer in Figure 4, illustrating the fidelity of super-resolved wind images as a comparison between LR data and HR data. SR-PWA restores major wind structures and the velocity intensities. SR-GAN excels in generating finer wind structures, as evidenced by the results depicted in In summary, we show our preliminary investigation of the super-resolved 3D wind structures based on a newly developing SR network that utilizes a self-attention network and a generative model. It enables to incorporating of 3D dynamics of weather systems that are essential to reconstructing physically representative 3D wind fields, and then achieve to generate high-fidelity outputs. This work is a preliminary step toward building a tracing methodology that is capable of simulating the trajectory of GHGs combined with limited observation and reduces the computational overhead. Further algorithmic advancements should improve the accuracy of the methodology.