—As the scaling down of semiconductor manufacturing nodes, optical proximity correction (OPC) has become more and more crucial , where the OPC using machine learning to construct models, or so -called machine learning optical proximity correction (MLOPC), is flourishing. The forward and reverse networks in MLOPC are two deciding parts. Training the image -to-image OPC dataset using various advanced ML models can lead to less fitted results. The tests on diverse mainstream image -to-image ML models, including Unet, Vnet, ResUnet, U2net, and attention Unet, are conducted in our studies. We propose a meta -learning method based on reinforcement learning (RL) with the forward and reverse networks cascaded to boost the training efficiency and accuracy for MLOPC. The meta -learning is performed so that the selected setting of the next training step is based on the Q -model in RL, and the baseline trai ning method is the direct utilization of adaptive moment estimation (Adam). In the setting of the cascaded network with six different models and three types of datasets, the model using the meta -learning approach takes advantage of the cascade and RL -controlled training path to boost the model accuracy, and the results using meta -learning surpass the baseline results in most of the cases, especially the instanc es with highly complex data or network architecture. Among them, the networks trained via the proposed meta -learning RL method receive the averaged increments of 19.5%, 1.2%, and 4.8% in the forward models and the averaged increments of 1 2%, 3.1%, and 13% in the reverse models. I. INTRODUCTION riven by the demand for higher performance, shrinking technology nodes is essential, but they have encountered various challenges nowadays. Photolithography is pivotal in manufacturing semiconductors by transferring photomask patterns onto silicon wafers, achieving the desired chip designs. However, the reduction of feature sizes causes the diffraction limit of light, increasing the complexity of printing precise patterns, such as line end pinching, line bridging, and corner rounding. Optical Proximity Corr ection (OPC) has been one of the most popular resolution enhancement techniques (RETs) to correct the variations in image intensity caused by diffraction and other optical effects, optimize mask printability, and im prove the edge -of-corner placement accura cy of pattern features. Over the years, different OPC methods have been proposed to address the inevitable geometric distortion by shrunk feature siz es. The conventional OPC methods can be categorized as rule -based OPC, model -based OPC, and inverse lithogr aphy technology - based (ILT) OPC. Rule -based OPC utilizes the pre -established rules, and the OPC process can be performed quickly, but this method is typically limited to non -systematic effects before 90 nm nodes [1, 2]. Model -based OPC is a more sophisticated approach with physical or empirical model parameters to predict the behavior of light during the lithography process. The o ptimized mask layouts were achieved using simulation models and iteratively adjusting the feature shapes and sizes [3-6]. The reverse engineering (RE) technique enables the analysis and dissection of existing products or systems to understand thei r design, structure, and function [7, 8]. Its application has proven significant across various industries. In semiconductor manufacturing, inverse lithography t echnology (ILT) is critical in reverse engineering operations. Conventionally, by developing a model for reverse mapping in ILT, optimized mask shapes can be obtained using pixel -wise optimization techniques without predefined rules or models while solving the OPC problem as an inverse problem. This approach enables finding inverse solutions for lithography that cover resist development and etching processes in addition to optics [9-12]. However, ILT requires significant computational resources due to the complex optimization algorithms involved. Since every pixel mu st be calculated in conventional ILT optimization processes, especially for complete IC design layouts or high -resolution patterns, a large number of iterations c an be time -consuming. MLOPC is a novel OPC technique that has emerged with machine learning ne tworks. Instead of relying on manual rule definition, model calibration, and time -consuming iterative computing, it involves training a machine learning model to predict OPC solutions for new patterns based on input dataset features. Various techniques hav e been proposed in this regard. Gu et al. developed a linear regression model to modify the mask with fewer required iteration times than the original model -based OPC [13]. Jia et al. applied the stochastic gradient descent method to inverse mask synthesis as an MLOPC problem [14]. Luo generated the optimal mask pixel values using a multilayer perceptron (MLP) neural network (NN) [15]. With the evolution of deep learning, Ma et al. [16] and Cecil et al. [17] proposed a convolutional neural networ k (CNN) with an inverse lithography ML model, which boosted iteration efficiency and improved the quality of the optimized mask output. Borisov et al. presented a CNN network with small training parameters for lithography hotspot detection, which would con sume less computation time and maintain a high level of D 2 accuracy [18]. Yang et al. replaced shallow learning models with deep convolutional neural networks without a pooling process, detecting the hotspot with better accuracy and fewer false alarm penalties [19]. Yan g et al. also extracted high -dimensional features using CNN with biased learning algorithms. The results showed a 5.9% improvement in hotspot detection for large -scale layouts [20]. Tung et al. showed that the modified attention layer embedded in Unet structure could achieve higher accuracy with self - attention mechanism [21]. Shao et al. used CNN models to build two pre -trained model s, LithoNet and OPCNet. The first one was responsible for predicting mask s pattern to Scanning Electron Microscope (SEM) images, and the other one predicted the optimized mask from LithoNet's output SEM images. They were connected together to achieve a fully data -driven framework of OPC [22]. U-Net, one of the robust Deep Neural Networks (DNNs), was initially proposed for the biomedical image segmentation problem [23]. More and more OPC research has adopted U -Net due to its flexibility and simplicity, especially when training with limited sample data [24]. Yu et al. [25] compared single -channel U -Net (SCU -Net) and multi -channel U -Net (MCU -Net) and demonstrated that MCU -Net can improve accuracy and significantly reduce computation time. Cao et al. combined GA N (Generative Adversarial Network) and OPC with the U -Net architecture, increasing speed and lithographic performance [26]. Furthermore, to further reduce GPU computations, Xiao et al. proposed a novel OPC solution by concatenating a forward and reverse model composed of U -Net, allowing for the direct transfer of the original mask to the optimized mask within a single model [27]. Tejender et al. combined OPC and etch proximity effects (EPC) with MLP neural networks and dimensionality reduction (DR) algo rithms, overcoming the problem of the lack of physics EPC model and redu cing the long computation time [28]. These approaches are expected to enhance lithography printability a nd accuracy in the field of semiconductor manufacturing. The semiconductor industry faces challenges in deep learning applications due to the limited availability of relevant dataset s, specifically in the context of the "few -shot learning problem." Meta -learning, also known as "learn to learn," has emerged as a promising approach to address the "few -shot learning problem" arising from the scarcity of data samples. Meta -learning involves two concepts: the base learner learns the task, while the meta -learner learns how to optimize the training of the base learner. The meta -learner can be implemented using evolutionary algorithms or supervised learning algorithms. With the optimized meta - learner, the learning strategies of the base learners can be improved. Sev eral meta -learning frameworks have been proposed for various IC technology processes in recent years, showcasing promising results in resolving few -shot learning problems with limited data samples. Chen et al. combined meta -learning models with tree -based ensemble models and focused on virtual physical vapor deposition metrology to overcome the physical metrology delay of the wafer acceptance test [29]. Lee et al. applied a meta -learning approach, the Reptile algorithm [30], to resolve the few -shot regression problem of the sputtering process. The base learner adapted the variable fabrication parameters of titanium/nitride compounds, while the meta -learner optimized hyperparameters such as batch size, epochs, and others [31]. Tejender et al. proposed a meta -learned sampling method assisted by technology computer -aided design (TCAD), training the meta -learner to adjust the way of sampling [32]. The work he re aims to contribute significantly to advancing the OPC within the Industry 4.0 framework by employing novel network architectures and innovative reverse engineering techniques integrated with meta -learning. II. METHOD A. Dataset From the previous work [33], the gds.py module 1.6.13 [34] of Python 3. 9.5 [35] is employed to gener ate these patterns, which are then exported to K -layout 0.28.7 [36] for further processing. N -type silicon wafers are chosen as the substrate material, and KrF positive photoresist is utilized in the lithography process. The lithography sys tem, Canon FPA -6300ES6a, operates at a DUV wavelength of 248 nm (resolution ≤90 nm). Subsequently, the developed mask layout is captured using the Hitachi SU -8010 Cold Cathode Field Emission SEM. A comprehensive dataset is typically prepared for model trai ning in conventional machine learning training processes. However, in semiconductor manufacturing, producing a large number of samples before obtaining experimenta l results is time -consuming and resource -exhausted. The number of each one -bar pattern, two -bar pattern, tri -bar pattern, and polygonal rectangle pattern varies between about 90 to 210, along with their SEM images after the lithography process. Fig. 1. Illustration of (a) K -layout mask pattern, (b) binary mask pattern, (c) lithography SEM image, (d) binary SEM image. In the preprocessing stage of the training data, we first converted the pre -drawn K -Layout mask into a binary mask, as shown in Fig. 1, representing a tri-bar pattern. The edges were converted into white pixels (1), while the rest of the background was set to black pixels (0). The resulting binary mask had an image size of 256×256, with each pixel corresponding to approximately 24 n m. The filled graph i s more helpful for model training because the number of 0s and the number of 1s in the pixels of the original edge -only pictures differ too much, making it challenging for the model to learn effectively. Yet, the SEM images exhibited evitab le noise in the backgrounds. Due to noise in the original SEM images, some regions' boundaries can become faint or disconnected after image processing. Filling these areas using an algorithm can lead to distortion, creating inaccuracies during data prep rocessing. (a) (b) (c) (d) 4 μm 4 μm 4 μm 3 To avoid propagating these errors and maintain accuracy, we utilized only the edge information in the final training data. Therefore, after image processing, the SEM photos were aligned onto a 256×256 black background and registered with the binary mask, resulting in the binary SEM image. B. Algorithm The forward and reverse models are both constructed using an Unet [23, 37] in Fig. 2. In order to test whether the meta -learner can adapt to various tasks, i.e., different types of networks, we replace the Unet with other variations of the Unet architec ture. There were Attent ion Unet [37, 38] in Fig. 3, ResUnet -a [37, 39] in Fig. 4, R2Unet [37, 40] in Fig. 5, Self Attention Unet [37, 38] in Fig. 6, and Vnet [37, 41] in Fig. 7. Fig. 2. Unet [23, 37] Fig. 3. Attention Unet [37, 38] Fig. 4. ResUnet -a [37, 39] Fig. 5. R2Unet [37, 40] Unet 256×256 128×128 64×64 32×32 16×16164 128 256 512 1024 512256128164 FiltersCopy and ConcatenateConv2D, 1 ×1, Softmax Conv2D, 3 ×3, GELU, BN, padding MaxPooling2D UpSampling2D 1024512256128Olaf Ronneberger et al. in Medical Image Computing and Computer -Assisted Intervention –MICCAI, Proceedings, Part III 18 , 2015, pp. 234 -241. Y . Sha. Keras -unet-collection, GitHub repository, accessed 4 September 2021 Attention Unet 256×256 128×128 64×64 32×32164 128 256 5121 64 Conv2D, 1 ×1, Softmax Conv2D, 3 ×3, ReLU , BN, padding Stride Convolution, 2 ×2, stride = 2 UpSampling2D Filters Attention and Concatenate256128768384192X. Wang et al. The Visual Computer, pp. 1 -18, 2022. Y . Sha. Keras -unet-collection. GitHub repository, accessed 4 September 2021 256×256 128×128 64×64 32×32132 64 128 256 512ResUnet -a 16×16 8×810241792 5121 32 Conv2D, 1 ×1, Softmax Conv2D, 3 ×3, ReLU , BN, padding MaxPooling2D UpSampling2D Filters ResBlock -a Copy and Concatenate PSP PoolingF. I. Diakogiannis et al. ISPRS Journal of Photogrammetry and Remote Sensing, vol. 162, pp. 94 -114, 2020. Y . Sha. Keras -unet-collection. GitHub repository, accessed 4 September 2021 256×256 128×128 64×64 32×32164 128 256 512 512R2Unet 256128164 Recurrent (Conv2D, 3 ×3, ReLU , BN, padding) ×2Conv2D, 1 ×1, Softmax Conv2D, 3 ×3, GELU, BN, padding MaxPooling2D UpSampling2D Filters Copy and Concatenate512256128M. Z. Alomet al. in NAECON 2018 -IEEE National Aerospace and Electronics Conference , 2018, pp. 228 -233. Y . Sha. Keras -unet-collection. GitHub repository, accessed 4 September 2021 4 Fig. 6. Self Attention Unet [37, 38] Fig. 7. Vnet [37, 41] The baselines of the six networks in Fig. 2, Fig. 3, Fig. 4, Fig. 5, Fig. 6, and Fig. 7 are responsible for the primary synthesis and output of the dataset. We divide it into two parts: the forward model and the reverse model. In previous research, the focus has been on forward propagation, which means that when the input is a mask, the outpu t generates SEM pattern images by (1). ff Y F(X : W , b )= (1) Y is the predicted SEM image, X is the actual mask pattern, and W and b represent the weights and biases in the forward neural network. F expresses th e network that converts the mask patterns into SEM images. Next, we aim to incorporate a reverse propagation model after this stage, which is intended to transfer the SEM patterns back into masks. R expresses the function that transfers the SEM images back to the mask pattern by (2). bb X R(Y : W , b )= (2) X is the predicted mask pattern, Y is the actual SEM image, and W and b represent the weights and biases in the reverse neural network. Both forward and reverse model s split 20% for validation data, and they are trained by their validation loss for 20000 epochs with the early stop patience of 30 epochs. The optimizer is Adam, with a 0.001 learning rate in the baseline. The vali dation loss is calculated by binary cross -entropy (BCE), and the test loss is decided by customized BCE loss, the BCE loss with edge enhancement process. In this research, each forward and reverse model is composed of the same kind of network. The forward and reverse models above were connected as the primary model, as shown in (3). In our research, there were two types of models: one is a forward -to- reverse cascaded model involving the reverse model connected to the forward model, in which the input is mask layout, and the output is the predicte d mask. ( ( : , )) ( ( : , ))ff bbX R F X W b Y F R Y W b= =  (3) The other one is the reverse -to-forward cascaded model, including the forward model followed by the reverse model in (4). ( ( : , )) ( ( : , ))bb ffY F R Y W b X R F X W b= =  (4) The train data is split by 20% for the validation data . The forward and the reverse model s are trained with patience=1 in an early stop callback. Afterward, the cascaded model is trained for two epochs with an initial pre -trained learning rate of 0.001 to obtain the previous validation loss and the current validation loss. Then , the cascaded model is trained by meta -learning RL. The validation loss is calculated by BCE loss. The test loss is determined by customized BCE loss. , , , _ _ ( , , , ) _ f f b bf f b b W b w bF R Cascade Losses W b W b Argmin (5a) 256×256 128×128 64×64 32×32164 128 256 5121 64Self Attention Unet 256128 768384192 Filters Attention Copy and ConcatenateConv2D, 1 ×1, Softmax Conv2D, 3 ×3, ReLU , BN, padding UpSampling2DStride Convolution, 2 ×2, stride = 2X. Wang et al. The Visual Computer, pp. 1 -18, 2022. Y . Sha. Keras -unet-collection. GitHub repository, accessed 4 September 2021 Vnet 256×256 128×128 64×64 32×321 3216 12864 16×16 256256 1281286432 116 Conv2D, 1 ×1, Softmax Conv2D, 3 ×3, PReLU , BN, padding Filters Copy and Add Copy and ConcatenateConv2DTranspose, 3 ×3, ReLU , BN, padding Stride Convolution, 2 ×2, stride = 2F. Milletari et al. in 2016 fourth international conference on 3D vision (3DV) , 2016, pp. 565 -571. Y . Sha. Keras -unet-collection. GitHub repository, accessed 4 September 2021 5 , , , _ _ ( , , , ) _ f f b bf f b b W b w bR F Cascade Losses W b W b Argmin (5b) As shown in Fig. 8, the architecture involves cascading the forward and the reserves MLOPC model. In the forward -to-reverse cascaded model, the objective is to en hance the training of the initial cascaded model, with the aim of minimizing the validation loss as defined in (5a), where the F_R_Cascade_Losses are the BCE loss value of the predicted SEMs of the forward model and the predicted mask s of the reverse model . Conversely, in the reverse -to-forward cascaded model, the training is conducted based on (5b) , where the R_F_Cascade_Losses are the bce loss of the predicted masks of the reverse model and the predicted SEMs of the forward model . Training the cascaded models in this approach , together with the RL described in the next paragraph, aims to improve accuracy in the first model. The process is reversed if the input is a mask and the same procedure is followed in the opposite direction. The meta -learner plays a role in adjusting the hyperparameters of the base learner based on the current performance of the base learner, which is cascaded in Fig. 8. We employ the DQN (Deep Q -Network) approach from RL (reinforcement learning) to establish a q -model as the meta -learner. In this framework, the state is defined as a combination of the validation losses at the first model in the cascaded model. We define four actions in our approach, which represent two sets of different learning rates (0.001 and 0.0001) and the weighted combination of losses from the forward and cascaded models in the cascade model. The weights for these combinat ions are [0.2, 0.8] and [0.8, 0.2], respectively. Consequently, there are four strategies of action resulting from the permutations and combinations, numbered [0,0], [0,1], [1,0], and [1,1]. Finally, the reward is defined as 1/loss of the first model in the cascaded model achieved by the action, where a minor loss corresponds to a greater reward for that action by (6). ( ) , selected currnet actionaction argmax r argmaxQ state action == (6) r is the reward in RL, and Q is the Q model. The q -model is composed of three dense layers. The first two layers each consist of 10 and 15 neurons with the ReLU activation function, while the final output layer comprises a single neuron with the swish activation function. The input t o the q -model consists of four neurons. Specifically, the first two neurons receive inputs for the previous and current losses corresponding to the state of RL. The remaining two neurons receive the action codes as inputs. O nce the q -model receives these i nputs, it selects the action that maximizes the reward, thereby minimizing the validation data loss among the four possible actions in (7). The chosen action is then executed in the next iteration, and the q -model is repeatedly trained based on the newly o btained state, action, and reward combinations, as shown in Fig. 8. The goal is to teach the q -model the optimal action to select in different combina tions of (previous state, current state) to yield the most significant reduction in the validation data loss. If the first model of the cascaded model fails to improve the validation loss within 30 epochs, the RL training process will be terminated. 2 ,([ , ], [ , ]) qqprev current prev current true Wb meanQ state stateargmin raction action r = − (7) The value of rtrue was the state after the action. Meta -learner employs the Epsilon Greedy [42] application method from Q -learning. When no values have been accumul ated in the Q -table, we want the model to perform exploration in order to explore unknown territories to understand the feedback it brings. Initially, we set epsilon (ε) to 0, where there is a probability of 1 -ε to execute actions randomly without consider ing the Q -table's results and a probability of ε to select the best action displayed in the Q -table Fig. 8. The model of meta -learner and base -learner. to achiev e a maximized reward. Furthermore, ε increases over time because, in the beginning, we are uncertain about which actions are desired. We initially searched within the ε range of 0.005 to 0.03 with an increment of 0.005. We found that when ε = 0.0 15, Predicted Mask/SEM fitted to True Mask/SEM with Cross Entropy loss Forward/Reverse Model ( Unet , etc)Reverse/Forward Model ( Unet , etc)Predicted SEM/Mask fitted to True SEM/Mask with Cross Entropy loss SEM/Mask Meta -learner: QmodelCascaded model (Two outputs, with weightings in training) Step -by-step training control by RL (learning rate, output weights, etc)SEM/Mask 2 meat -RL outperformed most of the baselines. Therefore, following the same approach, we narrowed down the search range to ε = 0.01 to 0.0 2 with an increment of 0.001. We found that when a = 0.013, meta -RL exhibited the best performance. Consequently, we cho se this value as our hyperparameter. As time progresses, ε increases by 0.013 after each epoch. This means that initially, there is more random behavior, but as time passes, decisions will be made more based on the Q -table, i.e., exploitation. Adjusting ε enables the model to transition between exploration and exploitation, allowing the model to achieve better results during tra ining. To ensure that the Q -model maintains a certain level of flexibility in its choices until reaching the early stop condition o f 30 times patience iterations, we set ε not to exceed 0.8. Additionally, suppose the patience is more than 20, regardless of the epoch. In that case, there is a 0.1 probability of selecting the worst action to prevent RL from getting stuck in local minima during t raining . Once trained, the q -model serves as a meta -learner capable of self -training the cascaded model across different neural networks. This approach saves considerable time by automating the process of manually adjusting model parameters. Addit ionally, the meta - learner can adapt to various neural network architectures, thereby achieving the "learn to learn" objective in meta -learning. TensorFlow 2.4.1 [43], NumPy 1.22.3 [44], OpenCV 4.7.0 [45], and GPU GeForce RTX 2080Ti in a Linux environment were used in this work. Algorithm: Reinforce Learning for Meta Learner For epochs action = argmax (Qmodel(state, action)) if action == 1 learning_rate = a1 output_wt = b1 elif learning_rate = a2 output_wt = b2 elif action == 1 learning_rate = a3 output_wt = b3 else learning_rate = a4 output_wt = b4 Reward, Next_State = Cascade_model.fit(Xtrain, Ytrain) Append/Update Rward, Next_State, action in Rarry, Sarry, Aarry Qmodel.fit([S arry, Aarry], R arry) end for III. RESULTS AND DISCUSSIONS In our setting, the first model of the cascaded model can be improved in the meta -learned RL method, but the second one can be worse. Hence, the first model is used to compare with the model using the baseline training method. Fig. 9 to Fig. 14 show the test set predictions of actual SEMs in the three datasets for va rious models using the baseline Adam training or meta -learned RL training. Fig. 9 and Fig. 10 are the predictions of the two -bar data from the baseline and the meta -learning RL training method. Among them, the results of Unet, R2Unet, and the two kinds of Attention Unet are similar, and the predicti ons have no significant defects. Nevertheless, the baseline and meta -learning results of Vnet and ResUnet -a are very different. In Fig. 9, the border of Vnet's prediction graph is dim and bumpy, the outer frame of ResUnet -a's prediction is vague and unclear, and some regions on 2 the edges are disconnected. In contrast, using meta -learned RL, the edges of Vnet's predicted graph become regular , the corner sides of the outlines are improved, and the lines of ResUnet -a's predicted graph are seldom disconnected. Fig. 11 and Fig. 12 are the predictions of the tri -bar dataset based on the baseline and meta -learned RL training method. The results of Unet and the two kinds of Attention Unet are similar; only the corners of the predicted graphs in these three models in the baseline are distorted and convex outward, while the results are slightly better when using meta -learned RL. R2Unet has similar results in the prediction of the two methods. Th e graphics in the baseline are less able to predict the changes in the corners and edges of the original SEM image. At the same time, the meta -learned RL method is more predictable, but the color of the corners will become lighter. The results of Vnet and ResUnet -a are significantly worse in the baseline. Vnet predicts graphs with rugged edges, unclear inner boundaries, and broken corners, and ResUnet -a can hardly predict complete pattern outlines. When meta - learned RL is used, although the corners of the p rediction of Vnet are distorted as well, the edges become smoother. The prediction of ResUnet -a is more visible on its outline. Fig. 13 and Fig. 14 are predictions of the mixed dataset based on the baseline and meta - learned RL training. In this case, only the results of Vnet and ResUnet -a are worse t han the other models. When using the meta - learned RL method, the gnarly edges of the Vnet prediction graphics are improved. Still, there will be some defects in the ou tlines. The disconnected regions on the border of the prediction from ResUnet -a become fe wer. Fig. 15 to Fig. 20 show the test set predictions of actual masks in the three datasets for var ious models using the baseline Adam training or meta -learned RL training. Fig. 15 and Fig. 16 are the predictions of the two -bar data from the baseline and the meta - learning RL training method. Among them, Unet, R2Unet, and the two kinds of Attention Unet reveal similar consequences in both training methods, whereas the predictions of Vnet and ResUnet -a from the two training are relatively distinct. When using baseline training, unet, R2Unet, and the two kinds of Attention Unet can predict the corners of the pattern outline well. Sti ll, the outline edges are easily distorted, or the color of lines in some regions is faded. The outlines of Vnet and ResUnet -a predictions are uneven, and the contour edges are prone to noise. These problems are partially corrected when using meta -learning RL training. The flaws of the pattern outline can be fixed when using meta -learning RL training, but the prediction of the corners will be slightly worse. Overall, in terms of the customized BCE loss, the patterns with meta -learning RL training are closer to the actual SEM images . Fig. 17 and Fig. 18 are the predictions of actual masks of the tri -bar dataset based on the baseline and meta -learned RL training method. In Fig. 17, the results of model s with the baseline training show faded color in most regions on the edges, and deformations occur at the lower edges. Furthermore, Vnet and ResUnet -a, using the baseline training, can barely predict the complete pattern. On the contrary, as shown in Fig. 18, the outlines of the results become smooth with fewer disconnections if the meta -learning training is used, and the predictions from Vnet and ResUn et-a are more visible. Nevertheless, the lines of the patterns are slightly thicker in some cases. In Fig. 19 and Fig. 20, the results of Unet, R2Unet, and the two kinds of Attention Unet are satisfied, which are generally in line with the actual graphics, but some edges tend to become curved or tra nslated when using the baseline training method. In contrast, these defects can be improved using the meta -learning RL training, but the angle at the corner will be slightly distorted. When Vnet uses the meta -learning RL training, the prediction is greatly enhanced. In addition to smoother edges, borders are no longer broken. The prediction of ResUnet -a improves the broken outline and uneven color, but the boundary of the internal test is still convex. Fig. 9. The predicted SEMs and real SEMs of the two -bar patterns for the baseline training method. 4 μm(a) Mask (b) SEM (c) Unet (d) Att-Unet (e) Satt-Unet (f) Vnet (g) R2Unet (h) ResUnet -a 3 Fig. 10. The predicted SEMs and real SEMs of two -bar patterns for the meta -learning RL training. Fig. 11. The predicted SEMs and real SEMs of tri -line patterns or the baseline training method. Fig. 12. The predicted SEMs and real SEMs of tri -line patterns for the meta -learning RL training. Fig. 13. The predicted SEMs and re al SEMs of the mixed patterns for the baseline training method. 4 μm(a) Mask (b) SEM (c) Unet (d) Att-Unet (e) Satt-Unet (f) Vnet (g) R2Unet (h) ResUnet -a 4 μm(a) Mask (b) SEM (c) Unet (d) Att-Unet (e) Satt-Unet (f) Vnet (g) R2U -net (h) ResUnet -a 4 μm(a) Mask (b) SEM (c) Unet (d) Att-Unet (e) Satt-Unet (f) Vnet (g) R2U -net (h) ResUnet -a 4 μm(a) Mask (b) SEM (c) Unet (d) Att-Unet (e) Satt-Unet (f) Vnet (g) R2Unet (h) ResUnet -a 4 Fig. 14. The predicted SEMs and real SEMs of the mixed patterns for the meta -learning RL training. Fig. 15. The predicted Masks and real Masks of the two -bar patterns for the baseline training method. Fig. 16. The predicted Masks and real Masks of two -bar patterns for the meta -learning RL training. Fig. 17. The predicted Masks and real Masks of the tri -line patterns for the baseline training method. 4 μm(a) Mask (b) SEM (c) Unet (d) Att-Unet (e) Satt-Unet (f) Vnet (g) R2Unet (h) ResUnet -a 4 μm(c) Unet (d) Att-Unet (e) Satt-Unet (f) Vnet (g) R2Unet (h) ResUnet -a (a) SEM (b) Mask 4 μm(c) Unet (d) Att-Unet (e) Satt-Unet (f) Vnet (g) R2Unet (h) ResUnet -a (a) SEM (b) Mask 4 μm(a) SEM (b) Mask (c) Unet (d) Att-Unet (e) Satt-Unet (f) Vnet (g) R2Unet (h) ResUnet -a 5 Fig. 18. The predicted Masks and real Masks of tri -line patterns for the meta -learning RL training. Fig. 19. The predicted Masks and real Masks of the mixed patterns for the baseline training method. Fig. 20. The predicted Masks and real Masks of the mixed patterns for the meta -learning RL training. Table 1 shows the results of the customized BCE loss value of the test set in six different models and three kinds of datasets from the baseline and the meta -learning RL with the forward -to-reverse c ascade. Each model from the baseline and the meta - learning RL method has been trained five times, respectively, and the test set's maximum, minimum, and average customized BCE loss values are calculated. From the performance on either the BCE loss value of the validation set or the customized BCE loss values of the test set, the meta -learned RL is better than the baseline , where the models trained by the meta -learning RL can win 15 out of 18 cases in terms of the average values . In the tri -bar dataset, the customized BCE loss values of the six cases in the baseline show difficult training compared to the two -bar and mixed datasets. This observation shows the complexity of the tri -line dataset and the challenges imposed by a small data amount. Express ly, in the baseline results of tri -line patterns, the Vnet and ResUnet -a exhibit poor test set performance. It can be seen from the customized BCE loss score that the models are hardly trained well, and they also have many undesirable effects that can be observed in the test set predicted graphs, such as breaks or even distortions in the pattern outlines. On the other hand, if meta -learning RL is used, the results will have the opportunity to be significantly enhanced, especially for the models with more complex ar chitectures such as ResUnet -a. In the rest of the other four models, an enhancement can also be observed in the customized BCE loss values of the test set. This indicate s the meta -learning RL's effect in coping with these complicated situations. As for the two-bar and mixed datasets, although most of the baseline performance is already satisfactory, it can also be seen that Vnet and ResUnet -a models can be improved using meta -learning RL. In addition, Unet, two types of Attention Unets, and R2 -Unet are also slightly enhanced by the meta -learning RL. 4 μm(a) SEM (b) Mask (c) Unet (d) Att-Unet (e) Satt-Unet (f) Vnet (g) R2Unet (h) ResUnet -a 4 μm(a)SEM (b) Mask (c) Unet (d) Att-Unet (e) Satt-Unet (f) Vnet (g) R2Unet (h) ResUnet -a 4 μm(a)SEM (b) Mask (c) Unet (d) Att-Unet (e) Satt-Unet (f) Vnet (g) R2Unet (h) ResUnet -a 6 Table 2 shows the results of the customized BCE loss value of the test set in six different models and three kinds of datasets from meta -learned RL with reverse -to-forward cascading. Each model from the baseline and the meta -learning RL method has been trained five times, respectively, and the test set's maximum, minimum, and average customized BCE loss values are calcul ated. The models trained by the meta -learning RL can win 14 out of 18 cases in the se cases. Among them, the Vnet and ResUnet -a models can be significantly improved in the three data sets due to the complexity of training. In the cases of Unet, the two types of attention Unets and R2Unet of the two -bar dataset, even if the results of the meta -learning method are similar to the baseline, the models can still be further improved by the meta -learning RL, which is reflected in corrections on the graphic outlines of the predictions. The results in Table 1 and Table 2 manifest the effectivity of the meta -learning RL in either the forward -to-reverse or the reverse -to-forward cascades. In the part of the two -bar dataset, the graph is more straightforward , and the amount of data is more than the other two, so the baseline result is relatively good, and the meta -learned RL method is not so helpful. As for the part of the mixed dataset , we think that although there are many types of patterns, most are relatively simple, so it is easier to train the model. This is also why the meta -learned RL can only be slightly helpful for some instances while using the mixed dataset. When the patterns are more complex, such as the tri -bar dataset, more complex error surfaces te nd to exist under this situation. According to observations, it is more likely to be trapped in some saddle regions. It is necessary to use a flexible learning rate chosen to train to some extent for a better training result, especially for the more compli cated models. As a result, the enhancements of Vnet and ResUnet -a, or sometimes Unet, are more prominent when using meta -learned RL. The forward and reverse networks in MLOPC are two critical components. The forward network training is usually more diffic ult in our cases evaluated by the customized BCE loss , i.e., datasets, based on our test on the various mainstream image -to-image prediction ML models, including Unet, Vnet, ResUnet -a, R2Unet, and two types of Attention Unet. The goal is to utilize the cascaded model network and the meta -learning method to pull up the first model in the cascaded model , though this requires some considerations. Firstly, the successive substitution, where the repeated feeding, starting from the forward or the reverse network, is carried out for several runs. The doubt is whether the predicted SEM or mask will be more and more similar to the actual solutions. The test based on DUV -248 lithography in our study concludes that successive substit ution is not a possible way to pull up the accuracy of the forward or the reverse network. The reason is that the error accumulates instead of diminishes , which seems to be understandable since the accuracy of the network is not perfect, and the error at e ach feeding can add up. Indeed , in theory, it is also possible that the forward and reverse network correct each other and leads to converged iteration in the successiv e substitution. The actual outcome, i.e., improved or degraded test set prediction, depe nds on the problem's nature and cannot be known beforehand. With the fitting only conducted at the final output after the successive substitution, we often observe the significantly degraded test set prediction. Additionally, after multiple substitutions, the bulk model leads to optimizer burdens. It should be emphasized that the training setting is critical to boosting the mutual correction nature. The training should b e implemented with auto -encoder -like architecture, and the fitting is established at the encoded vector and the endpoint for one iteration to make the mutual correction more pronounced. Furthermore, if successive substitutions are executed several times, more and more outputs are generated, and all of them should be fitted to the true SEMs or masks. The increased number of network outputs and the increased fitting loading result in unmanageable, poor convergence, which challenges the optimizers' training capability based on our test. In many test cases, four to six successive substations alrea dy lead to training problems where the binary cross -entropy loss minimization can easily be trapped, and the prediction accuracy of the forward and reverse model is not increased at all from the first or second epoch. Therefore, we finally choose only one substitution for enhancing the forward model, where the mask goes into the forward model first, and the predicted SEM goes into the reverse model, thus completing one iteration. More iteration only leads to degraded results. Table 1. The customized BCE loss value of the test set in the six models using the baseline and the meta -learning RL training with the forward -to-reverse cascade. Forward Model ( patience = 30 ) F -to-R Tri-bar Baseline Meta -learned RL Model val. BCE test set customized BCE val. BCE test set customized BCE avg max min avg avg max min avg Unet 0.03757 0.37668 0.36181 0.36590 0.03197 0.31737 0.29591 0.30553 Att. Unet 0.03831 0.37904 0.30536 0.34001 0.03277 0.31543 0.30088 0.30781 Self Att. Unet 0.04167 0.45026 0.33929 0.38461 0.03267 0.32904 0.29079 0.30601 Vnet 0.05401 0.49967 0.47175 0.48415 0.04047 0.41299 0.34563 0.37438 2 R2Unet 0.03235 0.32980 0.30499 0.31840 0.03357 0.32302 0.29928 0.31011 ResUnet -a 0.08736 0.92171 0.64350 0.76581 0.03754 0.43563 0.40354 0.41968 Two-bar Baseline Meta -learned RL Model val. BCE test set customized BCE val. BCE test set customized BCE avg max min avg avg max min avg Unet 0.01662 0.20069 0.17742 0.18848 0.01588 0.19075 0.18272 0.18598 Att. Unet 0.01613 0.19085 0.18180 0.18610 0.01568 0.19463 0.18581 0.18932 Self Att. Unet 0.01607 0.18372 0.17397 0.17748 0.01624 0.18588 0.17967 0.18287 Vnet 0.02105 0.23293 0.20109 0.21710 0.01799 0.21644 0.20285 0.20961 R2Unet 0.01598 0.19352 0.18022 0.18497 0.01581 0.18013 0.17408 0.17627 ResUnet -a 0.01960 0.24555 0.21619 0.22963 0.01849 0.23817 0.20839 0.22400 Mixed -bar Baseline Meta -learned RL Model val. BCE test set customized BCE val. BCE test set customized BCE avg max min avg avg max min avg Unet 0.01134 0.13226 0.12587 0.12940 0.01014 0.12796 0.11464 0.12199 Att. Unet 0.01260 0.16001 0.14585 0.15168 0.01086 0.14538 0.13444 0.14062 Self Att. Unet 0.01257 0.15839 0.14276 0.15306 0.01108 0.14174 0.13665 0.13970 Vnet 0.01662 0.21637 0.18958 0.20518 0.01439 0.20618 0.18824 0.19753 R2Unet 0.01119 0.15859 0.12315 0.13306 0.01117 0.14933 0.12824 0.13879 ResUnet -a 0.01667 0.36609 0.17035 0.24346 0.01382 0.40830 0.16467 0.22509 Table 2. The customized BCE loss value of the test set in the six models using baseline and meta -learned RL training with the reverse -to-forward cascade. Backward Model ( patience = 30 ) R -to-F Tri-bar Baseline Meta -learned RL Model val. BCE test set customized BCE val. BCE test set customized BCE avg max min avg avg max min avg Unet 0.02254 0.32139 0.24338 0.26959 0.02306 0.22957 0.22392 0.22608 Att. Unet 0.02362 0.26155 0.22938 0.24466 0.02456 0.23981 0.22633 0.23273 Self Att. Unet 0.02549 0.32270 0.26449 0.29216 0.02392 0.23546 0.21316 0.22542 Vnet 0.03200 0.36934 0.34330 0.35668 0.02571 0.29662 0.25490 0.27340 R2Unet 0.02185 0.23288 0.19543 0.21488 0.02475 0.23116 0.22245 0.22877 ResUnet -a 0.04186 0.44684 0.40937 0.42795 0.03693 0.39774 0.36766 0.37589 Two-bar Baseline Meta -learned RL Model val. BCE test set customized BCE val. BCE test set customized BCE avg max min avg avg max min avg Unet 0.01278 0.14194 0.13638 0.13840 0.01253 0.13956 0.13474 0.13684 Att. Unet 0.01293 0.14233 0.13414 0.13690 0.01289 0.14414 0.13734 0.14116 2 Self Att. Unet 0.01263 0.14739 0.13910 0.14428 0.01260 0.15243 0.13790 0.14130 Vnet 0.01655 0.16865 0.15842 0.16313 0.01357 0.15256 0.14770 0.14979 R2Unet 0.01293 0.14528 0.13659 0.14038 0.01265 0.14181 0.13627 0.13882 ResUnet -a 0.02918 0.37063 0.22398 0.28481 0.01910 0.28562 0.23995 0.25849 Mixed -bar Baseline Meta -learned RL Model val. BCE test set customized BCE val. BCE test set customized BCE avg max min avg avg max min avg Unet 0.00749 0.11485 0.08765 0.09856 0.00782 0.10627 0.09087 0.09985 Att. Unet 0.00886 0.11762 0.10182 0.11106 0.00815 0.10930 0.10364 0.10690 Self Att. Unet 0.00861 0.12131 0.10478 0.11301 0.00815 0.11392 0.10287 0.10734 Vnet 0.01472 0.17850 0.14857 0.15890 0.01012 0.13326 0.12079 0.12588 R2Unet 0.00744 0.11171 0.09232 0.10269 0.00856 0.11216 0.10942 0.11046 ResUnet -a 0.01028 0.58830 0.15950 0.32804 0.01033 0.15608 0.12766 0.14185 Moreover , the complex model trained by the baseline method or under complex data often gives results with large fluctuations owing to the GPU variability , indicating the instability of the baseline method . In contrast to the models using baseline training, the test set results of the model trained by the meta -learning RL with the cascaded model are relatively stable under the same setting . Thus, b esides outperforming the baseline training method in most cases, the predictions of the models trained with meta - learning RL training are similar in each run . In addition to the cascade order, the flow in meta -learning RL training can be in different settings and take some considerations. First, we can train the forward and the reverse network in advance using the e arly stop patience setting. After that, the trained forward and reverse network is cascaded together in the order described above and trained as a whole, with the weights set as trainable. The weights of the second network in the cascaded model can also be set as untrainable if their accuracy is high. The second way is training the cascaded model from scratch, which means the weights in the forward and reverse models start from tensorflow weight initializers. Finally, it is also possible to train both the f orward and reverse models slightly using small patience in the early stop, and the training of the cascaded model is established from that. Regarding the final cascaded model traini ng, it is also the users' choice to fit the final output, i.e., masks, or t he intermediate and final output, i.e., SEMs and masks. As discussed above, in theory, the mutual correction phenomenon should be more pronounced if we fit the intermediate output to drag back t he deviated network predictions. Regarding the network hype rparameter tuning, novel training scheme, and avoiding poor training, some investigation deserves attention in OPC network training. Based on our tests, the training is prone to settle on the highly local minimum in the for ward OPC problems. Therefore, it is desirable to have a way to avoid these pitfalls and boost the training automatically. The manual tuning of the training is not pleasant. Hyperparameter tuning means repeated training on the training set, which is time -consuming. Manual tuning becomes im possible if the training scheme is made even more complex. For example, if we decide to change the learning rate, output weights, and other settings, manual trials on the train path can be unmanageable. In this work, we use a meta - learning approach using r einforcement learning. Actually, many different algorithms can be used to realize m eta-learning besides RL. This includes genetic algorithms (GA), artificial neural networks (ANN), any ML algorithms together with an optimization algorithm, and all optimiza tion algorithms alone, especially those that require no gradient. We choose RL here since the effect of using RL is satisfactory , though the same meta -learning flow presented in this work can be accomplished with any other algorithms. When constructing the meta -learning framework here, the first consideration is how many training setting possibilities are used in the procedure and how many training possibilities are enough to boost the accuracy. Second, we should assess if the meta -learning algorithm is eff icient with so many training combinations. Here , it is decided that the learning rate, the fitting output weights, and the network connection in training, i.e., cascaded or isolated, are essential factors based on the test before meta -learning. Adding more training possibilities can overload the meta -learning algorithms and increase the GPU runtime. A key, reduced set of training possibilities should be used unless necessary. On the other hand, RL is a versatile framework that can tackle large -scale problem s and is thus suitable for our purpose. Regarding the RL implementation, the decision of the approximate state can be a crucial consideration. Due to overly large state space, it is generally impossible to use precise states in many real RL problems, incl uding ours . The approximate state should reflect the true state in the environment and facilitate action -state-reward mapping. The reward is the validation set binary cross - entropy of the first model in the cascaded model, which can also be changed to cust omized loss as described in section II further to highlight the critical part of SEMs or masks. Actions can be incremental type, i.e., +1/0/ -1, or non -incremental type, i.e., directly transition to any state. Incremental type action leads to smooth converg ence but is less efficient, while the non -incremental type 2 action gives faster convergence but can lead to convergence problems in some cases. In this work, we use a non -incremental action type since the learning rate and fitting output weights do not need to be changed smoothly in theory to prevent convergence problems. In addition, since the training is conducted at each step in RL, using incremental type action limits the search sp ace unless an additional state variable is added to the total state to ind icate the potential pause of training at the current step. IV. CONCLUSION This study can enhance the MLOPC model accuracy through the proposed meta -learning RL training method. The training path controlled by a meta -learner is shown to be effective in getting out of a predicament when a model is prone to get stuck in a local minimum, whic h can be crucial in many practical fields where a dataset is complicated or the model used is difficult to train. Various mainstream image -to-image ML models have been tested, including Unet, Vnet, ResUnet -a, R2Unet, and two types of Attention -Unet, to ver ify the usability of the proposed meta -learning training. The selection of RL actions at each training step adjusts the training settings, and exploitation is tuned using the Epsilon Greedy algorithm. With the combination of the casc aded forward and revers e models and meta -learning, the optimal model training can achieve improved results in comparison to the baseline Adam optimizers. After training, the lower customized BCE values of the test set predictions reflect ed that the models trained by a meta -learn ing RL have enhanced comprehension of the SEM image outlines, such as smoother outlines or improved corner regions closer to real SEMs. The average d increments of 19.5%, 1.2%, and 4.8% can be seen in the forward models , and the average d increments of 1 2%, 3.1%, and 13% in the reverse models can be seen in the three datasets, where the enhancements of ResUnet -a in the more complicated dataset are particularly evident.