# Detailed Image Captioning and Hashtag Generation
## 3 Methods
### Detailed Image Captioning  
For image captioning, we experimented with and evaluated a number of state-of-the-art models. These models are discussed below. Florence-2 is a vision foundation model capable of handling various visual tasks across different spatial hierarchies and semantic granularities. The architecture of Florence-2 consists of two main components: an image encoder and a multi-modality encoder-decoder. The model is trained on a collection of 5.4 billion comprehensive visual annotations on 126 million images, which [PERSON] et al. [20] call FLD-5B. The image encoder in Florence-2 utilizes a DaViT (Dual-Attention Vision Transformer) model. Developed by [PERSON] et al. [28], DaViT is a vision transformer that incorporates two complementary types of self-attention to efficiently capture both local and global visual dependencies. The key components of DaViT's architecture include spatial window attention, channel group attention, alternating attention blocks, and a hierarchical structure. Spatial window attention divides the image into separate windows and applies self-attention within each window, allowing the model to efficiently capture detailed local features. Channel group attention operates on \"channel tokens\" instead of spatial tokens. Each channel token contains a summary of the entire image, enabling the model to capture overall context naturally. DaViT's design alternates between spatial window attention and channel group attention blocks throughout its structure. This approach helps the model refine local details while maintaining an understanding of the whole image. Additionally, DaViT employs a hierarchical structure with multiple stages. As the image progresses through these stages, the spatial resolution decreases while the feature dimension increases [28]. The DaViT component of Florence-2 transforms an input image \(I\in\mathbb{R}^{H\times W\times 3}\) into a set of visual token embeddings \(V\in\mathbb{R}^{N_{v}\times D_{v}}\), where \(H\) and \(W\) represent the height and width of the image, \(N_{v}\) is the number of visual tokens, and \(D_{v}\) is their dimensionality [20]. After the image encoding process, the multi-modality encoder-decoder processes both the visual information from DaViT and any text input. This component is based on a standard Transformer architecture. The multi-modality encoder-decoder operates as follows. 1. The visual embeddings \(V\) from DaViT undergo linear projection and normalization to obtain \(V^{\prime}\in\mathbb{R}^{N_{v}\times D}\). 2. Any text input, such as prompts or questions, is converted into embeddings \(T_{prompt}\in\mathbb{R}^{N_{v}\times D}\) using a tokenizer and embedding layer. 3. These visual and text embeddings are combined to form the input for the encoder: \(X=[V^{\prime},T_{prompt}]\). 4. The multi-modality encoder processes this combined input through several Transformer layers to capture relationships between visual and textual elements. Figure 1: Block diagram: CapFlow image captioning and Hashtag generator. Finally, the decoder generates output text based on the encoder's output and any previously generated tokens. This design allows Florence-2 to handle a wide range of tasks that involve both images and text, such as image captioning, visual question answering, etc. The combination of DaViT's efficient image encoding and the flexible multi-modality processing enables Florence-2 to achieve high performance across various vision-language tasks. To adapt Florence-2 for detailed image captioning, we employed Low-Rank Adaptation (LoRA) [29], an efficient fine-tuning technique. LoRA allows for the update of a small number of model parameters while maintaining overall model performance. Our implementation of LoRA focuses on adapting the attention mechanisms within the model, facilitating more nuanced caption generation without significantly increasing computational requirements. The LoRA technique works by representing weight updates during fine tuning as low-rank decompositions.