# Globally Guided Confidence Enhancement Network for Image-Text Matching
## 3 Methodology
### Feature Alignment  
To capture a more fine-grained correspondence, inspired by [20], we use similarity vectors to capture fine-grained interactions between feature representations from different modalities. The similarity function between vectors \(v\in R^{d}\) and \(u\in R^{d}\) is defined as:  
\[s(v,u;W)=\frac{W|v-u|^{2}}{||W|v-u|^{2}||_{2}} \tag{2}\]  
where \(|\cdot|^{2}\) and \(||\cdot||_{2}\) indicate element-wise square and L2-norm respectively, and \(W\in\mathbb{R}^{m\times d}\) is a learnable parameter matrix to obtain the \(m\)-dimensional similarity vector. Local Semantic Alignment. We compute the local semantic similarity representation \(s_{e}^{v}\) between the image region \(v_{e}\) and its semantically matched relevant words \(a_{e}^{t}\) with Equation (3):  
\[s_{e}^{v}=s(v_{e},a_{e}^{t};W_{M}^{v})\]where \(W^{v}_{M}\in\mathbb{R}^{m\times d}\) aims to learn the local similarity representation. Following [12], we define a weighted combination of word representations to attend to words with respect to each image region. Specifically, this combination is referred to as the attended text vector \(a^{t}_{e}\), and is defined with respect to the image region \(v_{e}\):  
\[a^{t}_{e}=\sum_{e=1}^{M}\alpha_{e}t_{r}, \tag{4}\]  
where  
\[\alpha_{e}=\frac{\exp^{(\lambda\mathcal{L}_{\sigma})}}{\sum_{e=1}^{M}\exp^{( \lambda\mathcal{L}_{\sigma})}}, \tag{5}\]  
where \(\mathcal{L}_{\sigma}=\frac{\left[c_{\sigma}\right]_{+}}{\sqrt{\sum_{e=1}^{L} \left[c_{\sigma}\right]_{+}^{2}}}\) and \(c_{\sigma}\) is the cosine similarity between region \(v_{e}\) and word \(t_{r}\). Correspondingly, the establishment of semantic similarity \(s^{t}_{r}\) between a word \(t_{r}\) and its visually-matched context \(a^{v}_{r}\) within the image is facilitated by \(s^{t}_{r}=s(t_{r},a^{v}_{r};W^{t}_{L})\). Global Semantic Alignment. We compute the global semantic similarity representation \(s^{glo}\) between the image global representation \(v^{glo}\) and the text global representation \(t^{glo}\) with Equation (6):  
\[s^{glo}=s(v^{glo},t^{glo};W_{G}) \tag{6}\]  
where \(W_{G}\in\mathbb{R}^{m\times d}\) aims to learn the global similarity representation. Globally Guided Confidence Enhancement. Salient image regions' visual semantics become fragmented when they are viewed separately, which results in regionally matched region-word pairs that might not be compatible with the overall image-text semantics. We use confidence to weigh how well each region (word) ascribes to the global image-text view, thereby boosting the confidence of meaningful local matching pairs. Specifically, we verify the semantic similarity between text global representation \(t^{glo}\) and its semantically matched relevant images (text) \(a^{v}_{r}\) (\(a^{t}_{e}\)) as:  
\[s^{context-v}_{r}=s(a^{v}_{r},t^{glo};W^{v}_{c}) \tag{7}\]  
and  
\[s^{context-t}_{e}=s(a^{t}_{e^{\prime}}t^{glo};W^{t}_{c}) \tag{8}\]  
where \(W^{v}_{c}\in\mathbb{R}^{m\times d}\) and \(W^{t}_{c}\in\mathbb{R}^{m\times d}\) are learnable parameter matrices. Then, we use the global semantic similarity \(s^{glo}\). The normalized correlation between \(s^{context-v}_{r}\) and \(s^{context-t}_{e}\) for confidence perception with Equation (8) is defined as:  
\[c^{v}_{e}=\sigma(LN(w^{v}_{e}(s^{glo}\odot s^{context-t}_{e}))) \tag{9}\]  
and  
\[c^{t}_{r}=\sigma(LN(w^{t}_{r}(s^{glo}\odot s^{context-v}_{r}))) \tag{10}\]  
respectively, where \(\left\{c^{v}_{e}\Big{|}e=1,2,\dots,M,c^{v}_{e}\in\mathbb{R}^{d}\right\}\) and \(\left\{c^{t}_{r}\Big{|}r=1,2,\dots,L,c^{t}_{r}\in\mathbb{R}^{d}\right\}\). \(w^{t}_{r}\in\mathbb{R}^{1\times d}\) and \(w^{t}_{r}\in\mathbb{R}^{1\times d}\) are learnable parameter vectors, and \(\odot\) indicates the element-wise product, \(\sigma\) represents the sigmoid function and LayerNorm represents the normalization of layers. Specifically, a text (image) representation fused with an image (text) context and global pooled features work together to generate matching pairs' confidence.