KOSMOS-2
Source: transformers

# KOSMOS-2## OverviewThe KOSMOS-2 model was proposed in [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://arxiv.org/abs/2306.14824) by Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Furu Wei.KOSMOS-2 is a Transformer-based causal language model and is trained using the next-word prediction task on a web-scaledataset of grounded image-text pairs [GRIT](https://huggingface.co/datasets/zzliang/GRIT). The spatial coordinates ofthe bounding boxes in the dataset are converted to a sequence of location tokens, which are appended to their respectiveentity text spans (for example, `a snowman` followed by `<patch_index_0044><patch_index_0863>`). The data format issimilar to “hyperlinks” that connect the object regions in an image to their text span in the corresponding caption.The abstract from the paper is the following:*We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent refer expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where object descriptions are sequences of location tokens. Together with multimodal corpora, we construct large-scale data of grounded image-text pairs (called GrIT) to train the model. In addition to the existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), Kosmos-2 integrates the grounding capability into downstream applications. We evaluate Kosmos-2 on a wide range of tasks, including (i) multimodal grounding, such as referring expression comprehension, and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This work lays out the foundation for the development of Embodiment AI and sheds light on the big convergence of language, multimodal perception, action, and world modeling, which is a key step toward artificial general intelligence. Code and pretrained models are available at https://aka.ms/kosmos-2.*<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/kosmos_2_overview.jpg"alt="drawing" width="600"/><small> Overview of tasks that KOSMOS-2 can handle. Taken from the <a href="https://arxiv.org/abs/2306.14824">original paper</a>. </small>## Example```python>>> from PIL import Image>>> import requests>>> from transformers import AutoProcessor, Kosmos2ForConditionalGeneration>>> model = Kosmos2ForConditionalGeneration.from_pretrained("microsoft/kosmos-2-patch14-224")>>> processor = AutoProcessor.from_pretrained("microsoft/kosmos-2-patch14-224")>>> url = "https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg">>> image = Image.open(requests.get(url, stream=True).raw)>>> prompt = "<grounding> An image of">>> inputs = processor(text=prompt, images=image, return_tensors="pt")>>> generated_ids = model.generate(...     pixel_values=inputs["pixel_values"],...     input_ids=inputs["input_ids"],...     attention_mask=inputs["attention_mask"],...     image_embeds=None,...     image_embeds_position_mask=inputs["image_embeds_position_mask"],...     use_cache=True,...     max_new_tokens=64,... )>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]>>> processed_text = processor.post_process_generation(generated_text, cleanup_and_extract=False)>>> processed_text'<grounding> An image of<phrase> a snowman</phrase><object><patch_index_0044><patch_index_0863></object> warming himself by<phrase> a fire</phrase><object><patch_index_0005><patch_index_0911></object>.'>>> caption, entities = processor.post_process_generation(generated_text)>>> caption'An image of a snowman warming himself by a fire.'>>> entities[('a snowman', (12, 21), [(0.390625, 0.046875, 0.984375, 0.828125)]), ('a fire', (41, 47), [(0.171875, 0.015625, 0.484375, 0.890625)])]```This model was contributed by [Yih-Dar SHIEH](https://huggingface.co/ydshieh). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/kosmos-2).## Kosmos2Config[[autodoc]] Kosmos2Config## Kosmos2ImageProcessor## Kosmos2Processor[[autodoc]] Kosmos2Processor    - __call__## Kosmos2Model[[autodoc]] Kosmos2Model    - forward## Kosmos2ForConditionalGeneration[[autodoc]] Kosmos2ForConditionalGeneration    - forward