# A Unified Visual and Linguistic Semantics Method for Enhanced Image Captioning
## 4 Methodology
### SegmentCLIP
Commencing with \(l=1\), for each grouping stage, we first concatenate \(s_{i}\) and \(g_{i}\) together and then input them into a series of Transformer layers, each of which facilitates information propagation between the concatenated tokens via  
\[\{\mathbf{g}_{i}^{l}\},\{\mathbf{s}_{i}^{l}\}=\text{Transformer}((\{\mathbf{g }_{i}^{l}\};\{\mathbf{s}_{i}^{l}\})), \tag{6}\]  
Here, the symbol \([;]\) symbolizes the concatenation operation. Subsequently, we assimilate the updated \(M_{l-1}\) image segment tokens \(\{\mathbf{s}_{i}^{l}\}\) into \(M_{l}\) new segment tokens \(\{s_{i}^{l+1}\}_{i=1}\) via a Segmenting Block as follows:  
\[\{\mathbf{s}_{i}^{l+1}\}=\text{SegmentingBlock}(\{\mathbf{g}_{i}^{l}\},\{ \mathbf{s}_{i}^{l}\},\{\mathbf{g}_{i}^{l}\}). \tag{7}\]  
At every grouping stage, the relationship \(M_{l}<M_{l-1}|\) holds true, indicating a progressive reduction in the number of group tokens and, concurrently, a coalescence into fewer but larger image segments. Upon reaching the terminal grouping stage \(L\), we apply Transformer layers on all segment tokens and leverage their outputs to derive the final global image representation \(z^{L}\) as follows:  
\[\{\mathbf{s}_{i}^{l+1}\}=\text{Transformer}(\{\mathbf{s}_{i}^{l+1}\}), \tag{8}\]  
\[z^{L}=\text{Transformer}(\text{AvgPool}(\{\mathbf{s}_{i}^{L+1}\})). \tag{9}\]  
As delineated in Figure 1(a), SegmentCLIP reconfigures visual information into an array image segments that are not restricted by a regular grid structure, thus optimizing the visual information arrangement. In the case of the Segmenting Block, as depicted in Figure 1(b), at the conclusion of each grouping stage, the Segmenting Block unifies the learned group tokens and segment tokens attributed to identical groups into a single new image segment based on similarity in the embedding space. This process is mathematically formulated as  
\[\mathbf{A}_{i,j}^{l}=\frac{\exp(W_{q}\mathbf{g}_{i}^{l}\cdot W_{k}\mathbf{s}_ {j}^{l}+\gamma_{i})}{\sum_{k=1}^{M_{l}}\exp(W_{q}\mathbf{g}_{k}^{l}\cdot W_{k} \mathbf{s}_{j}^{l}+\gamma_{k})}, \tag{10}\]  
where \(W_{g}\) and \(W_{k}\) represent the weights corresponding to the learned linear projections for the group and segment tokens, respectively. The term \(\gamma\) denotes the learnable bias pertaining to the Gumbel \((0,1)\) distribution. The computation of segment token assignment to groups is conducted by employing the one-hot operation of an argmax over all the group tokens. Since the argmax operation is non-differentiable, we utilize the straight-through trick as a proxy, which results in a differentiable surrogate assignment matrix \(\mathbf{A}\):  
\[\mathbf{\hat{A}}^{l}=\text{one-hot}(\mathbf{A}_{\text{argmax}}^{l})+\mathbf{ A}^{l}-\text{sg}(\mathbf{A}^{l}), \tag{11}\]where \(sg\) denotes the stop gradient operator. Utilizing this straight-through trick, \(\mathbf{A}^{I}\) retains the one-hot assignment of a token to a single group, but its gradient is equivalent to the gradient of \(\mathbf{A}\), ensuring the model remains end-to-end trainable. Subsequent to the allocation of segment tokens to various learned groups, we amalgamate the embedding of all tokens belonging to the same group into a single new segment token \(\mathbf{s}^{I+1}_{i}\). For each group, the output of the Segmenting Block is a weighted sum of the segment tokens assigned to that group, computed as  
\[\mathbf{s}^{I+1}_{i}=\hat{\mathbf{g}}^{I}_{i}+W_{o}\frac{\sum_{j=1}^{M_{I-1}} \hat{\mathbf{A}}^{I}_{i,j}W_{o}\hat{\mathbf{s}}^{I}_{j}}{\sum_{j=1}^{M_{I-1}} \hat{\mathbf{A}}^{I}_{i,j}}, \tag{12}\]  
where \(W_{o}\) and \(W_{o}\) represent the parameters learned to integrate the features. An alternate method to the hard assignment is the soft assignment, which utilizes \(\mathbf{A}\) rather than \(\mathbf{A}^{I}\), as prescribed in Equation (5).