# A Unified Visual and Linguistic Semantics Method for Enhanced Image Captioning
## 4 Methodology
### SegmentCLIP  
We present the SegmentCLIP image encoding mechanism as depicted in Figure 2, which employs a Transformer-derived framework for the hierarchical and progressive categorization of visual concepts. Within the architecture of SegmentCLIP, Transformer layers are compartmentalized into several stages of grouping. At each of these stages, a series of group tokens are acquired as learnable entities through the mechanism of self-attention, enabling the global assimilation of information from all image tokens (segments). Subsequently, these acquired group tokens are utilized to amalgamate similar image tokens by employing a Segmenting Block. By orchestrating a structured progression of grouping stages, we facilitate the consolidation of smaller image segments into larger conglomerates. The subsequent sections will elucidate each component in detail. Building upon the ViT [32] architecture, we initially partition an input image into N discrete non-overlapping segments, subsequently mapping each segment linearly into a latent dimensional space. These transformed segments are regarded as individual image tokens, collectively represented by the notation \(\{\mathbf{I}_{i}\}_{i=1}^{N}\) At every stage of grouping, in addition to these image tokens, we integrate a suite of learnable group tokens. This augmented set is then fed into the Transformer designated for the respective stage. In the context of Multi-stage Grouping, as illustrated in Figure 2a, rather than channeling all \(N\) input image tokens through the entirety of the Transformer's layers, we distribute  
Figure 2: **(a) The architecture and training pipeline of segmentCLIP. ** easing size. The images to the right illustrate the visual segments that manifest across different grouping stages. In the lower stages, pixels are grouped into parts of objects, such as the hands and legs of a woman or a little girl; in the higher stages, these are further amalgamated into complete entities, such as the entire body of the woman and the little girl. **(b) The architecture of the segmenting block. ** At the end of each grouping stage, there is a segmenting block that calculates the similarity between the learned tokens and the segment (image) tokens. The assignment is determined via a gumbel softmax operation over the learned tokens and is then converted into a hard one-hot assignment. The segment tokens that are assigned to the same group are merged, forming new segment tokens that serve as input for the subsequent grouping stage. these layers across multiple stages of grouping, establishing a hierarchical structure. At the conclusion of each stage, a Segmenting Block is employed to amalgamate smaller entities into larger conglomerates. To articulate this process formally, let us posit \(L\) grouping stages, each identified by an index \(l\) and associated with a collection of learnable group tokens \(g_{i}\). For the sake of clarity, the initial set of image patches \(\{\mathbf{I}_{i}\}_{i=1}^{N}\), serving as inputs to the first grouping stage, are designated as the preliminary segments, with \(N=M_{0}\). This simplification allows us to equate the initial set of image patches with the initial group of segments. Commencing with \(l=1\), at each stage of grouping, we commence by amalgamating the image patches \(\{\mathbf{I}_{i}\}\) and group tokens \(\{\mathbf{g}_{i}\}\), subsequently inputting this combined set into a sequence of Transformer layers. These layers are tasked with facilitating information exchange between the segments and group tokens, thereby enabling effective propagation of information. For expediency, the set of image patches \(\{\mathbf{I}_{i}\}\) input to the initial grouping stage is treated as the set of starting segments \(\{\mathbf{s}_{i}\}_{i=1}^{M_{0}}\), where \(N=M_{0}\). We simplify the notation of the starting segments to \(s_{i}\) and the learnable group tokens to \(g_{i}\).