# Gaze-Dependent Image Re-Ranking Technique for Enhancing Content-Based Image Retrieval
## 3 Gaze-Dependent Image Re-Ranking
### Gaze-Dependent Image Caption  
To model image, captioning, and gaze information, we have constructed the CGT model, employing previous methods [22] to acquire knowledge of their interrelations. This model comprises three modules: an image encoder, a caption encoder-decoder, and a gaze trace encoder-decoder. As shown in Figure 3, each module is built on a transformer and receives \(\mathbf{c}_{\text{v}}\), \(\mathbf{c}_{\text{w}}\), \(\mathbf{c}_{\text{r}}\) as input, denoting the features of image, caption, and gaze trace, respectively. The image encoder \(\mathcal{T}_{\text{v}}\) is defined as follows:  
\[\mathcal{T}_{\text{v}}=\text{FFN}(\text{MHA}_{\text{v}}(Q,V,K)), \tag{2}\]  
where \(\text{FFN}(\cdot)\) refers to the feed-forward network with two ReLU-based linear transformation layers, based on a previous study [52]. It also employs multi-head attention (MHA) with a weight matrix of query \(Q\), value \(V\), and key \(K\) to deal with the multi-modal input during training and is defined as follows:  
\[\text{MHA}_{\text{v}}(Q,K,V)=\text{Concat}(\text{head}_{1},\dots,\text{head}_{ \text{c}})P^{o}, \tag{3}\]  
Figure 2: An overview of our method. We use gaze traces associated with initial retrieval results as feedback to generate gaze-dependent image captions. Moreover, we comprehensively use feature information from gaze-dependent image captions and query images for re-ranking. \[\text{head}_{i}=\text{Attention}(QP_{i}^{Q},KPi_{i}^{K},VP_{i}^{V}), \tag{4}\]  
where \(P^{o}\), \(P_{i}^{Q}\), \(P_{i}^{K}\), and \(P_{i}^{V}\), the parameter matrices, are the projections following the definition [52]. As illustrated in Figure 3, it is worth noting that the outputs of the image encoder \(\mathcal{T}_{\text{v}}\) have a significant impact on both the caption encoder-decoder \(\mathcal{T}_{\text{w}}\) and the gaze trace encoder-decoder \(\mathcal{T}_{\text{r}}\). The CGT model is designed to effectively capture the interdependence between the image, caption, and gaze trace. Our method employs a symmetrical structure to generate captions and gaze traces to achieve this. The caption encoder-decoder \(\mathcal{T}_{\text{w}}\) and the trace encoder-decoder \(\mathcal{T}_{\text{r}}\) are defined as follows:  
\[\mathcal{T}_{\text{w}}=\text{MHA}_{\text{w}}^{2}(\text{MHA}_{\text{w}}^{1}( \mathbf{c}_{\text{w}},\mathbf{c}_{\text{w}},\mathbf{c}_{\text{w}}),\mathcal{ T}_{\text{v}},\mathcal{T}_{\text{v}}), \tag{5}\]  
\[\mathcal{T}_{\text{r}}=\text{MHA}_{\text{r}}^{2}(\text{MHA}_{\text{r}}^{1}( \mathbf{c}_{\text{r}},\mathbf{c}_{\text{r}},\mathbf{c}_{\text{r}}),\mathcal{ T}_{\text{v}},\mathcal{T}_{\text{v}}). \tag{6}\]  
To train the CGT model, the total function \(L_{\text{all}}\) is defined as follows:  
\[\mathcal{L}_{\text{all}}=\lambda_{1}\mathcal{L}_{\text{r}}+\lambda_{2} \mathcal{L}_{\text{w}}+\lambda_{3}\mathcal{L}_{\text{sum}}+\lambda_{4} \mathcal{L}_{\text{rw}}, \tag{7}\]  
In this total function, \(\mathcal{L}_{\text{r}}\) evaluates the difference between the bounding boxes of the predicted and actual gaze traces, which affects the performance of gaze trace generation. Conversely, \(\mathcal{L}_{\text{w}}\) is a cross-entropy loss function that measures the dissimilarity between the human-generated and predicted captions. The total loss function, \(L_{\text{sum}}\), is computed as the sum of \(\mathcal{L}_{\text{r}}\) and \(\mathcal{L}_{\text{w}}\). Furthermore, the model utilizes \(\mathcal{L}_{\text{rw}}\), which is a cycle consistency loss that compares the output caption and trace. The weighting coefficients for these loss functions are represented as \(\lambda*\), and their values were selected based on the method proposed in a previous study [53]. Figure 3: An overview of our connected caption and gaze trace (CGT) model. The model includes three transformer-based modules to learn the relevance of image, captioning, and gaze information.