# Welding Image Data Augmentation Method Based on LRGAN Model
## 2 LRGAN Model
### The Discriminator Part of the LRGAN Model
#### 2.1.4 Spectral Normalization and Lipschitz Constant  
Although the least squares loss function can accelerate convergence and, to some extent, ensure the stability of training, the issue of gradient vanishing caused by drastic changes in the discriminator's parameters has not been fully resolved. Therefore, this paper introduces spectral normalization during the training of the discriminator model to constrain the parameter matrix, ensuring that the gradients during backpropagation are limited within a certain range. This prevents excessive changes in the discriminator's parameters while preserving the structure of the convolutional kernels. Spectral normalization strictly constrains the spectral norm of the weight matrix at each layer, thereby controlling the Lipschitz constant of the discriminator, which enhances the stability of the generative adversarial network during training. Compared to other normalization techniques, spectral normalization only requires adjusting the Lipschitz constant, a conclusion that has been confirmed in [23]. According to the definition of the Lipschitz constant, we can obtain Equation (15):  
\[\frac{\|f(x)-f(x^{\prime})\|}{\|x-x^{\prime}\|}\leq M \tag{15}\]  
Here, M is a constant greater than 0, f(x) is a function defined on the real number set, and let \(||f||_{Lip}=\parallel f(x)-f(x^{\prime})\parallel/||x-x^{\prime}||\); then, \(\parallel f\parallel_{Lip}=\sup_{x}\sigma(\
abla f(x))\), \(\sigma(A)\) is the spectral norm of the matrix A, as shown in Equation (16):  
\[\sigma(\textbf{A})=\max\frac{\parallel\textbf{A}x\parallel_{2}}{\parallel x \parallel_{2}}=\max_{\|x\|_{2}\leqslant 1}\parallel\textbf{A}x\parallel_{2} \tag{16}\]  
For each layer g of the network, with h as the input, the bias term is not discussed in this paper, and g(h) = Wh; then:  
\[\parallel g\parallel_{Lip}=\sup_{k}\sigma(\
abla g(h))=\sup_{h}\sigma(W)= \sigma(W) \tag{17}\]  
Figure 4: Discriminator activation function. In the entire network, \(f(x)=W^{L+1}a_{L}\big{(}W^{L}\big{(}a_{L-1}(W^{L-1}(\cdots_{a_{1}}(W^{1}x)\cdots) )\big{)}\big{)}\). Because of \(\parallel g_{1}\circ g_{2}\parallel\leq\leq\parallel g_{1}\parallel_{\text{ Lip}}\bullet\parallel g_{2}\parallel_{\text{Lip}}\), there is:  
\[\parallel f\parallel_{Lip}\leq\prod_{l=1}^{L+1}\sigma\Big{(}W^{l}\Big{)} \tag{18}\]  
Here, L is the number of network layers, and WI is the weight of the L-th layer. To limit the Lipschitz constant of each layer to 1, \(\parallel g\parallel_{\text{Lip}}=1\), \(\sigma(W)=1\) is required. Therefore, by applying spectral normalization to the weight matrix W as shown in Equation (19), \(\sigma(W)=1\) is obtained. \[W\text{sn}=\frac{W}{\sigma(W)} \tag{19}\]  
In the LRGAN model, the ReLU activation function, the sigmoid activation function, and the spectral normalization operation in the discriminator all satisfy the Lipschitz constraint. To ensure that the entire discriminator meets the Lipschitz constraint, it is sufficient to divide the parameters of each convolutional kernel by the spectral norm of the corresponding convolutional kernel matrix. As shown in Figure 5, the discriminator in this paper is a simple five-layer nested convolutional neural network, where all layers use the ReLU activation function except for the output layer, which uses the sigmoid activation function. Using bounded activations allows the network to quickly learn to reach saturation, especially for high-resolution images, where using the ReLU activation function in the discriminator works very well. To ensure that the five-layer discriminator network satisfies the Lipschitz condition, the maximum singular value of each layer's weight matrix must be equal to 1. The discriminator can use spectral normalization to make the maximum singular value of each layer's weight matrix equal to 1, thereby limiting the gradient size of the discriminator to within 1 without altering the structure of the weight matrix. This helps make the adversarial training process more stable and easier to converge.