# Semantic-Aligned Cross-Modal Visual Grounding Network with Transformers
## 1 Introduction  
Visual grounding aims to locate the most relevant object or region in an image based on a natural language query. The recent advancements in deep neural networks have greatly contributed to the progress of visual grounding [1; 2; 3]. Technically, most 2D visual grounding approaches utilize visual representations to search for objects in an image, and they have shown promising performance in various applications such as automated driving systems (ADSs) [4], robotics [5], power transmission systems [6; 7], and remote sensing [8; 9; 10; 11], among others. The existing vision-based visual grounding approaches can be categorized into three paradigms: (1) the two-stage model, which sequentially extracts region proposals followed by object classification [12; 13; 14]; (2) the one-stage model, which simultaneously outputs the locations and categories of objects [15; 16; 17; 18]; and (3) the full end-to-end model, which directly generates results without manual parameter settings [19; 20]. These vision-based approaches leverage strong feature representations to achieve coarse-grained object discrimination, where visually distinct differences exist between inter-class objects. Coarse-grained visual grounding, in this context, refers to the differentiation and localization of objects between superior classes, such as dogs and cats. In contrast, fine-grained visual grounding aims to differentiate and locate subordinate classes within a common superior class, posing greater challenges in object discrimination. This is because distinguishing between extremely similar objects from two subordinate classes can sometimes exceed the capability of vision-based recognition, and even our eyes struggle to differentiate between them. For instance, the two planes shown in Figure 0(a) are so similar in visual appearancethat they cannot be distinguished, leading to detection failure by YOLO and deformable DETR [21; 22; 23]. As a result, vision-based approaches struggle to accurately capture the subtle differences between two similar objects, leading to incorrect predictions [21; 22; 23]. To tackle this problem, this paper explores the improvement of fine-grained visual grounding by integrating both textual and visual representations. As shown in Figure 1b, with the two passenger planes, they are difficult to distinguish based on visual appearances alone. However, their text descriptions clearly indicate their differences, such as \"narrowbody\" versus \"wide body\" and \"two jet engines\" versus \"four jet engines\". Textual descriptions provide more abstract semantic knowledge. Therefore, utilizing this textual information can effectively complement the limitations of visual representation and provide valuable cues for distinguishing between similar objects in fine-grained visual grounding (more details in Section 3). This paper introduces a novel approach called the semantic-aligned cross-modal visual grounding network with transformers (SAC-VGNet). SAC-VGNet is built upon the foundation of the state-of-the-art YOLOv7 [24] and combines textual and visual features [1; 25; 26] to enhance the discriminative capability of fine-grained visual grounding. In SAC-VGNet, given the textual and visual inputs, the network first employs a multi-scale cross-modal fusion module (MCMF) to effectively fuse both visual and textual descriptions. The MCMF consists of two sequential steps: textual and visual feature encoding, which generate the initial multi-modal feature, and multi-scale cross-modal feature decoding, which refines the multi-modal feature using a multi-head cross-attention operation. To further enhance the discrimination ability, SAC-VGNet adopts text-driven contrastive learning (TCL) to achieve an accurate feature alignment [27; 28; 29] (more details in Section 3.4). TCL uses the dot product to measure the similarity between the text projection and the related pixel-level projection while suppressing the response of the unrelated part.