# Semantic-Aligned Cross-Modal Visual Grounding Network with Transformers
## 3 Method
### Multi-Scale Cross-Modal Fusion  
Multi-scale cross-modal fusion first encodes textual and visual features for a given image, respectively, and then fuses them at a multi-scale level to offer robust fusion features for visual grounding. Next, we will elaborate on them.  
**Textual and visual feature encoding.** Given an image \(\mathbf{I}\in\mathbb{R}^{3\times H\times W}\) and a corresponding referring text \(\mathbf{T}\in\mathbb{R}^{B}\), where \(H\) and \(W\) are the height and width of \(\mathbf{I}\), respectively, and \(B\) represents the word length of \(\mathbf{T}\), we first adopt the backbone in YOLOv7 [24] as the image encoder to extract multi-scale visual features \(\{\mathbf{F}_{\mathbf{v}_{l}}\}_{l=1}^{L}\in\mathbb{R}^{C\times\frac{H}{l}\times \frac{W}{l}}\), where \(C_{l}\) is the feature dimension in the \(l\)-th layer and \(s_{l}\) is the scale factor. YOLOv7 uses extended efficient layer aggregation networks (E-ELAN), RepConv [55], and auxiliary head modules to improve the accuracy of real-time visual grounding without increasing the reasoning cost. The multi-scale visual features could provide global abstract representation as well as local discriminative representation to support fine-grained classification. Meanwhile, the text sequence is encoded by lower-cased byte pair encoding (BPE), and is bracketed with [sos] and [eos] tokens, which represent the start and end of a sequence, respectively. We utilize the text encoder of CLIP [28] to extract the semantic information of \(\mathbf{T}\) to obtain a representative textual feature \(\mathbf{F}_{t}\in\mathbb{R}^{C_{t}\times B}\), which is further flattened \(\mathbf{F}_{\mathbf{s}}\in\mathbb{R}^{C_{s}}\). CLIP has an exceptional ability to align text and images. It is trained on a wide variety of images with abundant natural language supervision available on the internet. This training enables CLIP to effectively locate visual pixels through textual semantics during subsequent multi-modal fusion.  
**Multi-modal feature fusion.** In order to incorporate positional information into the fused features, we generate a coordinate map of the size \(\frac{H}{s_{l}}\times\frac{W}{s_{l}}\times D_{spatial}\) as the spatial features, where \(D_{spatial}\) indicates the channel of the spatial features. Following the modality fusion approach in [1], we first map the visual and text features to the same scale and then use the concatenate operation to fuse the visual, text, and spatial features. Finally, we supply \(\mathbf{F}_{\mathbf{v}_{l}}\), \(\mathbf{F}_{\mathbf{s}}\), and \(F_{coord}\) to generate an initial multi-modal feature \(\mathbf{F}_{\mathbf{m}_{l}}\) as follows:  
\[\mathbf{F}_{\mathbf{m}_{l}}=Conv([Relu(\mathbf{F}_{\mathbf{v}_{l}}W_{\mathbf{v}_{l}}),Relu(\mathbf{F} _{\mathbf{s}}W_{s}),F_{coord}]), \tag{1}\]  
where \([\cdot]\) is the concatenation operation, \(Conv()\) denotes a 2D \(1\times 1\) convolution, and \(W_{\mathbf{v}_{l}}\) and \(W_{\mathbf{s}}\) are the learnable matrices to eliminate the scale difference between \(\mathbf{F}_{\mathbf{v}_{l}}\) and \(\mathbf{F}_{\mathbf{s}}\). We refer readers to [56] for more details about the ReLu activation function. Since the initial multi-modal feature \(\mathbf{F}_{\mathbf{m}_{l}}\) is generated by concatenating both visual and textual features with the convolution operation, it cannot exceptionally align visual and textual features. Therefore, we pass it to the following modules for further refinement.