# Globally Guided Confidence Enhancement Network for Image-Text Matching
## 3 Methodology
### Feature Representation  
Visual Representations. For each input image, \(M\) region-level visual features are extracted according to [4] using the Faster R-CNN [30] and ResNet-101 [31] models pre-trained on the Visual Genomes dataset [32]. In particular, the Faster R-CNN model is used to detect salient regions in image \(I\) and encode a visual representation for each detectedimage region. We convert them to \(d\)-dimensional vectors by adding a fully-connected layer. The image \(I\) can be denoted as \(\left\{v_{e}\middle|e=1,2,\ldots,M,v_{e}\in\mathbb{R}^{d}\right\}\), where \(M\) is the number of regions in image \(I\).  
Textual Representations. Given a text, we tokenize it into \(L\) words and input the word embeddings sequentially into a bidirectional GRU(Bi-GRU) [33] to improve the representation of the text using context semantics. After this, the representation of each word is produced by taking an average of the forward and backward hidden state at each time step, as \(\left\{t_{r}\middle|r=1,2,\ldots,L,t_{r}\in\mathbb{R}^{d}\right\}\).  
Global Representation and Self-Attention Enhancement. Globally guided local alignment plays an important role in constructing effective contextual features that improve the retrieval performance of the model while reducing the training complexity. Our idea is to mine local alignment information using a globally guided model. Inspired by [23], we use GPO, a generalized pooling operator that automatically adapts to different data modalities and feature extractors, to find well-encoded global features, then we use self-attention to enhance them.  
Specifically, the pooling coefficients \(\theta=\left\{\theta_{i}\right\}_{i=1}^{N}\) are generated by GPO, and the global features \(f^{glo}\) are defined as a weighted sum of locally sorted features. Note that \(v^{glo}\) and \(t^{glo}\) are denoted as \(f^{glo}\) in Equation (1):  
\[f^{glo}=\sum\limits_{i=1}^{N}\theta_{i}\cdot\max_{i}(\{\varphi_{n}^{i}\}_{n=1 }^{K}),\forall_{i} \tag{1}\]  
The size of the coefficients \(\theta\) is \(N\). The \(i\)th maximum value among the \(N\) elements is denoted by \(\theta_{i}\). We enforce the constraint \(\sum\limits_{i=1}^{N}\theta_{i}=1\) on the resulting weighted vector using Softmax. The positional encoding is used to enable the model to capture the sequential information of the input sequence. To achieve this, the positional encoding strategy in Transformers [34] is adopted, which vectorizes the positional index \(i\) and converts it to a dense vector \(p_{i}\), \(p_{i}^{x}=\left\{\begin{array}{ll}\sin(w_{y},i),&\text{when}\quad x=2y\\ \cos(w_{y},i),&\text{when}\quad x=2y+1\end{array}\right.,\forall i\), where \(w_{y}=\frac{1}{10,000^{2y/4}}\) and the positional encoding has \(d_{3}\) dimensions. Then, a sequence model is learned to generate the pooling weights, which consists of a bi-directional GRU (Bi-GRU) [33] and a multi-layer perceptron (MLP): \(\left\{h_{i}\right\}_{i=1}^{N}=\text{BIGRU}(\{p_{i}\}_{i=1}^{N}),\ \ \theta_{i}=\text{MLP}(h_{i})\) that takes the location-encoded sequence \(p=\{p_{i}\}_{i=1}^{N}\) as input and outputs the pooling coefficients \(\theta=\left\{\theta_{i}\right\}_{i=1}^{N}\). At the same time, each dimension of the vector is sorted from the largest to smallest as \(\varphi\) and then multiplied by the corresponding coefficients for weighting to obtain the global features \(f^{glo}\). In addition, we use a self-attention mechanism to update the global features to enhance the high-level semantic representation of the global features.