# An Integration Framework for the Inpainting and Colorization of Arbitrary Masked Grayscale Images
## 4 Experiments
### Implementation Details  
Our framework integrates multiple pre-trained models and fine-tuned components. For mask generation, we utilize the pre-trained ViT-Large model from SAM. Since SAM is designed for general-purpose segmentation and has been extensively trained on large-scale datasets, we use it as-is, without additional fine-tuning. The MAT component employs a model pre-trained on the Places365 dataset. Given that MAT is already optimized for large-hole inpainting and high-resolution image restoration, we leverage the pre-trained weights without further fine-tuning. For image colorization, we fine-tune the DDColor model to better adapt to our specific task. The model is trained on both DACON and Places365 datasets, where input images are first converted to grayscale before training. This allows the model to learn robust colorization strategies tailored to real-world image distributions. We also optimize DDColor using a multi-objective loss that balances perceptual, adversarial, and colorfulness constraints, ensuring high-fidelity colorization with minimal artifacts. The model is trained using eight NVIDIA A100 graphic processing units (GPUs) (80 GB video random access memory (VRAM) per GPU) with a batch size of four per GPU. Training is conducted in a Python 3.11 environment with PyTorch 2.5 and compute unified device architecture (CUDA) 11.8. To ensure reproducibility, we set the global random seed to 36 across all experiments, covering NumPy, PyTorch, and CUDA operations. We process input images at 512 \(\times\) 512 resolution and employ data augmentation strategies including horizontal flipping and mixing techniques with mixing probabilities \(p_{\text{cutmix}}=0.5\) and \(p_{\text{fmix}}=0.5\). For the network architecture, we implement DDColor using ConvNeXt-Large as the encoder with pre-trained weights. The decoder follows a multi-scale design with \(N_{q}=100\) queries across \(N_{s}=3\) scales and \(N_{d}=9\) decoder layers. The final layer incorporates spectral normalization, producing a 2-channel output. The discriminator follows a Dynamic UNet architecture with 64 base channels. The optimization process uses AdamW for the generator with learning rate \(\eta_{g}=10^{-4}\), weight decay \(\lambda=0.01\), and momentum parameters \(\beta_{1}=0.9\), \(\beta_{2}=0.99\). The discriminator uses Adam optimizer with learning rate \(\eta_{d}=10^{-4}\) and identical momentum parameters. The learning rate follows a step decay schedule with \(\gamma=0.5\) at iteration 8000, training for a total of 10,000 iterations. The loss function combines the following multiple objectives:  
\[\mathcal{L}_{\text{total}}=\lambda_{1}\mathcal{L}_{\text{L1}}+\lambda_{p} \mathcal{L}_{\text{percep}}+\lambda_{g}\mathcal{L}_{\text{GAN}}+\lambda_{c} \mathcal{L}_{\text{color}} \tag{1}\]  
where \(\lambda_{1}=0.1\), \(\lambda_{p}=5.0\), \(\lambda_{g}=1.0\), and \(\lambda_{c}=0.5\) are the respective weights. The perceptual loss \(\mathcal{L}_{\text{percep}}\) utilizes VGG16-BN (Visual Geometry Group 16-layer with Batch Normalization) features with progressive weights across different layers, while \(\mathcal{L}_{\text{GAN}}\) implements the vanilla GAN objective. We perform validation and save model checkpoints every 5000 iterations. Evaluation Metrics  
We employ three different metrics to evaluate the quality of our image inpainting results, which are then combined into a single comprehensive metric, as follows:  
**Structural Similarity Index (SSIM)**[41]: The SSIM metric measures the perceptual difference between two images by considering structural information. For Red, Green, Blue (RGB) images, it is computed as follows:  
\[\text{SSIM}(x,y)=\frac{(2\mu_{x}\mu_{y}+c_{1})(2\sigma_{xy}+c_{2})}{(\mu_{x}^{ 2}+\mu_{y}^{2}+c_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+c_{2})} \tag{2}\]  
where \(\mu_{x}\) and \(\mu_{y}\) are the average pixel values, \(\sigma_{x}^{2}\) and \(\sigma_{y}^{2}\) are the variances, and \(\sigma_{xy}\) is the covariance of images \(x\) and \(y\).