# NSGA-PINN: A Multi-Objective Optimization Method for Physics-Informed Neural Network Training
## 4 Numerical Experiments
### Inverse Pendulum Problem  
The algorithm was first used to train PINN on the inverse pendulum problem without noise. The pendulum dynamics are described by the following initial value problem (IVP):  
\[\dot{\theta}(t)=\omega(t) \tag{7}\] \[\dot{\omega}(t)=-k\sin\theta(t)\]  
where the initial condition is sampled as follows \((\theta(0),\omega(0))=(\theta_{0},\omega_{0})\in[-\pi,\pi]\times[0,\pi]\) and the true parameter unknown parameter \(k=1.0\).  
Our goal is to approximate the mapping using a surrogate physics-informed neural network: \(\theta_{0},\omega_{0},t\mapsto\theta(t),\omega(t)\). For this example, we used a neural network for PINN consisting of 3 hidden layers and 100 neurons in each layer. The PINN training loss for the neural network is defined as follows:  
\[\mathcal{L}=\mathcal{L}_{res}+\mathcal{L}_{ics}+\mathcal{L}_{data}. \tag{8}\]  
To determine the total loss in this problem, we add the residual loss, initial loss, and data loss. We calculate the data loss using the mesh data \(t_{s}\), which ranges from 0 to 1 (seconds) with a step size of 0.01. We fit these data onto the ODE to determine the data loss value accurately. For this problem, we set the parent population to 20 and the maximum number of generations to 20 in our NSGA-PINN.  
Figure 2: NSGA-PINN structure diagram.  
In the course of our experiment, we tested various methods, which are illustrated in Figure 3.  
Based on our observations, we found that the ADAM optimizer did not yield better results after 400 epochs, as the loss value remained in the scale of \(1\times 10^{-2}\). We also tried the NSGA-II algorithm for PINN, which introduced some diversity to prevent the algorithm from getting stuck at local minima, but the loss value was still around 4.0. Ultimately, we implemented our proposed NSGA-PINN algorithm to train PINN on this problem, resulting in a significant improvement with a loss value to the scale of \(1\times 10^{-5}\).  
To gain a clear understanding of the differences in loss values between optimization methods, we collected numerical loss values from our experiment. For the NSGA and NSGA-PINN methods, loss values were calculated as the average since they are obtained using ensemble methods through multiple runs. Our observations presented in Table 1 revealed that the total loss value of PINN trained with the traditional ADAM optimizer decreased to \(1.935\times 10^{-4}\). However, by training with the NSGA-PINN method, the loss value decreased even further to \(6.55\times 10^{-5}\), indicating improved satisfaction with the initial condition constraints.  
In Figure 4, we compare the predicted angle and velocity state values to the true values to analyze the behavior of the proposed NSGA-PINN method. The top figure shows how accurately the predicted values match the true values, illustrating the successful performance of our algorithm. At the bottom of the figure, we observe the predicted value of the parameter \(k\), which agrees with the true value of \(k=1\). This result was obtained after running our NSGA-PINN algorithm for three generations.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Methods** & **Residual Loss** & **Initial Loss** & **Data Loss** & **Total Loss** \\ \hline ADAM & 0.00013 & \(3.24\times 10^{-5}\) & \(3.11\times 10^{-5}\) & \(1.935\times 10^{-4}\) \\ NSGA & 0.12 & 2.67 & 4.10 & 6.89 \\ NSGA-PINN & \(3.94\times 10^{-5}\) & \(1.21\times 10^{-5}\) & \(1.41\times 10^{-5}\) & \(6.55\times 10^{-5}\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Inverse pendulum problem: Each loss value from NN trained by using different training methods.  
Figure 3: Inverse pendulum problem: Group of figures from left to right columns show each loss value by using different training method.