README.md

---
license: apache-2.0
datasets:
- VITA-MLLM/Long-VITA-Training-Data
base_model:
- Qwen/Qwen2.5-14B-Instruct
---


# Long-VITA-16K

Github: https://github.com/VITA-MLLM/Long-VITA

## 👀 Overview

Long-VITA is a strong long-context visual language model and supports more than 1 million tokens.

- Long-VITA-16K weights are trained on Ascend NPUs with MindSpeed. The original weight is at https://huggingface.co/VITA-MLLM/Long-VITA-16K.

- We also implemented Long-VITA on Megatron with the Transformer Engine to infer and evaluate on Nvidia GPUs. The converted weight is at https://huggingface.co/VITA-MLLM/Long-VITA-16K_MG.

- We also implemented Long-VITA on DeepSpeed with the Huggingface Transformers to infer and evaluate on Nvidia GPUs. The converted weight is at https://huggingface.co/VITA-MLLM/Long-VITA-16K_HF.


## 📈 Experimental Results
- **Comparison of image understanding**.

![image](https://github.com/user-attachments/assets/235bdb0e-37e6-4a5f-b20b-21b0bb83278a)
![image](https://github.com/user-attachments/assets/72250c5b-7d33-4dba-98ab-0539bae08703)


- **Comparison of video understanding**.

![image](https://github.com/user-attachments/assets/7f09662b-bd53-4504-927a-0e45214a049d)

![image](https://github.com/user-attachments/assets/87bd2f4d-baf5-4a63-8002-151e30f52147)


- **Effectiveness of Logits-Masked LM Head**.

![image](https://github.com/user-attachments/assets/7a06b4dd-267c-470f-80f2-d26c87e23460)



## Models

 Model          | LLM Size | Training Context | Training Frames | MindSpeed Weights                               | Megatron Weights                                   | Huggingface Weights                                
---------------:|---------:|-----------------:|----------------:|------------------------------------------------:|---------------------------------------------------:|---------------------------------------------------:
 Long-VITA-16K  | 14B      | 16,384           | 64              | https://huggingface.co/VITA-MLLM/Long-VITA-16K  | https://huggingface.co/VITA-MLLM/Long-VITA-16K_MG  | https://huggingface.co/VITA-MLLM/Long-VITA-16K_HF  
 Long-VITA-128K | 14B      | 131,072          | 512             | https://huggingface.co/VITA-MLLM/Long-VITA-128K | https://huggingface.co/VITA-MLLM/Long-VITA-128K_MG | https://huggingface.co/VITA-MLLM/Long-VITA-128K_HF 
 Long-VITA-1M   | 14B      | 1,048,576        | 4,096           | https://huggingface.co/VITA-MLLM/Long-VITA-1M   | https://huggingface.co/VITA-MLLM/Long-VITA-1M_MG   | https://huggingface.co/VITA-MLLM/Long-VITA-1M_HF   




## ACCEPTABLE USE POLICY

Any license on the model is subject to your compliance with the Acceptable Use Policy, and You must not violate (or encourage or permit anyone else to violate) any term of the Acceptable Use Policy. Tencent reserves the right to update this Acceptable Use Policy from time to time.

Tencent endeavors to promote safe and fair use of its tools and features, including Freeze-Omni. You agree not to use Freeze-Omni or any of its derivatives:
1. In any way that violates any applicable national, federal, state, local, international or any other law or regulation;
2. To harm Yourself or others;
3. To repurpose or distribute output from Freeze-Omni or any of its derivatives to harm Yourself or others; 
4. To override or circumvent the safety guardrails and safeguards We have put in place;
5. For the purpose of exploiting, harming or attempting to exploit or harm minors in any way;
6. To generate or disseminate verifiably false information and/or content with the purpose of harming others or influencing elections;
7. To generate or facilitate false online engagement, including fake reviews and other means of fake online engagement;
8. To intentionally defame, disparage or otherwise harass others;
9. To generate and/or disseminate malware (including ransomware) or any other content to be used for the purpose of harming electronic systems;
10. To generate or disseminate personal identifiable information with the purpose of harming others;
11. To generate or disseminate information (including images, code, posts, articles), and place the information in any public context (including –through the use of bot generated tweets), without expressly and conspicuously identifying that the information and/or content is machine generated;
12. To impersonate another individual without consent, authorization, or legal right;
13. To make high-stakes automated decisions in domains that affect an individual’s safety, rights or wellbeing (e.g., law enforcement, migration, medicine/health, management of critical infrastructure, safety components of products, essential services, credit, employment, housing, education, social scoring, or insurance);
14. In a manner that violates or disrespects the social ethics and moral standards of other countries or regions;
15. To perform, facilitate, threaten, incite, plan, promote or encourage violent extremism or terrorism;
16. For any use intended to discriminate against or harm individuals or groups based on protected characteristics or categories, online or offline social behavior or known or predicted personal or personality characteristics;
17. To intentionally exploit any of the vulnerabilities of a specific group of persons based on their age, social, physical or mental characteristics, in order to materially distort the behavior of a person pertaining to that group in a manner that causes or is likely to cause that person or another person physical or psychological harm;
18. For military purposes;
19. To engage in the unauthorized or unlicensed practice of any profession including, but not limited to, financial, legal, medical/health, or other professional practices.