# Deep Watermarking Based on Swin Transformer for Deep Model Protection
## 3 Materials and Methods
### Embedding Network Based on Swin-Unet
It is important to note that the Swin Transformer employs a hierarchical windowed attention mechanism, which must be used in pairs: the first Swin Transformer block partitions the feature map into non-overlapping fixed-size windows and computes local self-attention within each window. The subsequent Swin Transformer block adopts a shifted window strategy, where the window positions are shifted by 50% to create overlapping regions with the previous stage. This alternating window partitioning mechanism overcomes the global computation limitations of conventional Transformers, ensuring computational efficiency through local attention while enabling cross-region feature interaction via inter-layer window shifts. To extract multi-level features effectively, multiple Swin Transformer blocks are stacked, progressively establishing global feature relationships through a deep network. Subsequently, a feature downscaling module is employed, combining patch merging and Swin Transformer blocks. This module is represented by the blue block in Figure 4. Patch merging merges adjacent 2 \(\times\) 2 patches by concatenating their feature representations along the channel dimension, followed by a linear projection. This operation reduces the spatial resolution of the feature map by half while increasing the channel dimension, similar to downsampling in CNNs, but with a better preservation of structured information. Specifically, given an input feature map of size \(W\times H\times C\), where H denotes the height, W the width, and C the number of channels, the patch merging transformation follows this process:  
\[W\times H\times C\xrightarrow{\textit{Patch Merging}}\ \ \ \frac{W}{2}\times\frac{H}{2}\times 4C \xrightarrow{1\times 1\times\text{comp}}\ \ \frac{W}{2}\times\frac{H}{2}\times 2C, \tag{2}\]  
Here, the 1 \(\times\) 1 convolution is applied to reduce the merged 4C channels to 2C, a process analogous to channel compression in CNNs. This operation reduces computational  
Figure 4: Patch partition and linear embedding of embedding pipeline. complexity while retaining sufficient feature representation capacity. In this approach, three feature downscaling modules are used in total, progressively transforming the shape from \(\frac{W}{4}\times\frac{H}{4}\times 96\) to \(\frac{W}{32}\times\frac{H}{32}\times 768\). It is worth noting that, although progressive downsampling inevitably leads to some loss of low-level spatial information, the multi-scale features extracted at each encoder stage are transmitted to the decoder through skip connections. This design maintains the multi-scale representation of the feature pyramid while effectively compensating for information loss during downsampling through cross-layer feature fusion, which is crucial for subsequent watermark embedding and processed image reconstruction. Step 3. Decoder Stage  
The decoder adopts a symmetric architecture to progressively reconstruct image features, with the core operations being feature upsampling and cross-layer feature fusion. First, for each decoding layer, the input feature map undergoes a \(1\times 1\) convolution operation to expand the channel dimension from C to 2C. This process can be represented as:  
\[W\times H\times C\xrightarrow{1\times 1\times\text{conv}}\ W\times H\times 2C \xrightarrow{\text{Patch Expanding}}\ \ 2W\times 2H\times\frac{C}{2}, \tag{3}\]  
The purpose of channel expansion is to enhance the feature representation, providing richer features for subsequent upsampling operations. Next, a feature upscaling module is employed, combining patch expanding and Swin Transformer blocks. The expanded feature map undergoes spatial dimension reconstruction by rearranging the 2C channels of features in adjacent \(2\times 2\) patches, thereby generating a feature map with doubled resolution. This operation ensures that the output feature map's channel count remains consistent with the corresponding encoder layer's skip connection feature, facilitating subsequent feature fusion.