# Detailed Image Captioning and Hashtag Generation
## 3 Methods
### Detailed Image Captioning
For a pre-trained weight matrix \(W_{0}\in\mathbb{R}^{d\times k}\), LoRA constrains its update as  
\[W=W_{0}+\Delta W=W_{0}+BA, \tag{1}\]  
where \(B\in\mathbb{R}^{d\times r}\), \(A\in\mathbb{R}^{r\times k}\), and the rank \(r\) is much smaller than \(\min(d,k)\). We developed three LoRA adapters through fine tuning the original Florence model using three different datasets: Descriptions of Connected and Contrasting Images (DOCCI), Pixelprose, and Recap DataComp. These specific datasets were selected due to their variety and their capability to strengthen the model's performance in producing detailed, contextually nuanced captions. Besides Florence-2, we also experimented with and evaluated the alternative models for image captioning. 1. Large Language and Vision Assistant (LLaVA) [30] stands out as a significant development in multimodal modeling. At its core, LLaVA combines a CLIP ViT-L/14 image encoder with a large language model, which is linked through a linear projection layer. The latest iteration, LLaVA-1.6, incorporates the Mistral 7B model as its language component, resulting in a robust 7 billion parameter system. LLaVA's use of Low-Rank Adaptation (LoRA) for fine tuning allows it to efficiently adapt to detailed captioning tasks without compromising overall performance. LLaVA makes use of GPT-4 to generate multimodal language-image instruction following data, enabling impressive chat abilities that sometimes mirror multimodal GPT-4's performance on novel images and instructions. When applied to specific tasks like Science QA, the combination of LLaVA and GPT-4 has set new benchmarks in accuracy. 2. PaliGemma [31] takes a different approach, focusing on versatility and efficiency for deployment on resource-constrained devices. The base PaliGemma-3b-mix-448 model, with its 3 billion parameters, builds on the SigLIP-So400m vision encoder and Gemma-2B language model. Despite its relatively compact size, PaliGemma holds its own against much larger models like Mistral 8x7B and GPT-3.5. This impressive performance stems from a training regimen involving 3.3 trillion tokens, which is drawn from carefully filtered web data and synthetic sources. PaliGemma's Llama-2-like architecture facilitates easy integration with existing tools. The model family includes variants optimized for multilingual, multimodal, and long-context scenarios. Particularly noteworthy is PaliGemma's ability to run locally on modern phones when quantized to 4 bits, while still delivering strong performance. 3. The Phi-3 model series [32] represents another leap forward in efficient, high-performance language models. The base phi-3-mini model, at just 3.8 billion parameters, competes with much larger models on key benchmarks like MMLU and MT-bench. The Phi-3-vision-128k-instruct variant, slightly larger at 4.2 billion parameters, specializes in multimodal tasks and can handle impressively long context windows of up to 128K tokens. Phi-3's success lies in its \"data optimal regime\" training approach, which carefully balances knowledge and reasoning ability. This methodology allows Phi-3 to achieve top-tier performance while remaining compact enough for local deployment on mobile devices. Extensive post-training, including supervised fine tuning and direct preference optimization, further enhances Phi-3's capabilities across various tasks and improves its safety features. * CLIP-based models, such as ClipCap [11], offer strong visual-textual alignment but often require separate image encoders and text decoders. While powerful, this architecture can increase computational complexity, potentially limiting real-time applications. * BLIP-2 [33], though not directly compared in our results, offers robust performance but with a larger parameter count. This makes it less suitable for our goal of real-time processing in browser environments. Each of these approaches brings unique strengths to the table. Larger models often excel across a wide range of tasks but at the cost of increased computational demands.