Add comprehensive model card for DisTime (#1)

- Add comprehensive model card for DisTime (4eaa95da8cd288dec6a058c3c7656d5a15051dac)


Co-authored-by: Niels Rogge <nielsr@users.noreply.huggingface.co>

README.md

@@ -1,3 +1,96 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+pipeline_tag: video-text-to-text
+library_name: transformers
+tags:
+- multimodal
+- video-understanding
+- temporal-localization
+- qwen
+---
+
+# DisTime: Distribution-based Time Representation for Video Large Language Models
+
+This repository contains the official implementation and checkpoints for the paper:
+[**DisTime: Distribution-based Time Representation for Video Large Language Models**](https://huggingface.co/papers/2505.24329) (ICCV 2025).
+
+For more details, including installation, training, and evaluation scripts, please refer to the official [GitHub repository](https://github.com/josephzpng/DisTime).
+
+<div align="center">
+  <img src="https://github.com/josephzpng/DisTime/raw/main/images/network.png" width="600px"/>
+</div>
+
+## Abstract
+
+Despite advances in general video understanding, Video Large Language Models (Video-LLMs) face challenges in precise temporal localization due to discrete time representations and limited temporally aware datasets. Existing methods for temporal expression either conflate time with text-based numerical values, add a series of dedicated temporal tokens, or regress time using specialized temporal grounding heads. To address these issues, we introduce DisTime, a lightweight framework designed to enhance temporal comprehension in Video-LLMs. DisTime employs a learnable token to create a continuous temporal embedding space and incorporates a Distribution-based Time Decoder that generates temporal probability distributions, effectively mitigating boundary ambiguities and maintaining temporal continuity. Additionally, the Distribution-based Time Encoder re-encodes timestamps to provide time markers for Video-LLMs. To overcome temporal granularity limitations in existing datasets, we propose an automated annotation paradigm that combines the captioning capabilities of Video-LLMs with the localization expertise of dedicated temporal models. This leads to the creation of InternVid-TG, a substantial dataset with 1.25M temporally grounded events across 179k videos, surpassing ActivityNet-Caption by 55 times. Extensive experiments demonstrate that DisTime achieves state-of-the-art performance across benchmarks in three time-sensitive tasks while maintaining competitive performance in Video QA tasks. Code and data are released at [this URL](https://github.com/josephzpng/DisTime).
+
+## Dataset
+
+The InternVid-TG dataset proposed in the paper is released at: [yingsen/internvid-tg](https://huggingface.co/datasets/yingsen/internvid-tg).
+
+## Usage
+
+You can load the model using the `transformers` library and use it for video understanding tasks.
+
+```python
+import numpy as np
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM, AutoProcessor
+from decord import cpu, VideoReader
+
+# Load model, tokenizer, and processor
+tokenizer = AutoTokenizer.from_pretrained("UserJoseph/DisTime-1B")
+model = AutoModelForCausalLM.from_pretrained("UserJoseph/DisTime-1B", trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="auto")
+processor = AutoProcessor.from_pretrained("UserJoseph/DisTime-1B")
+
+model.eval()
+
+# Example video input
+video_path = "./examples/video1.mp4" # Replace with your video path
+qs = "Describe this video in detail"
+
+# Load video frames
+vr = VideoReader(video_path, ctx=cpu(0), num_threads=1)
+fps = float(vr.get_avg_fps())
+frame_indices = np.array([i for i in range(0, len(vr), round(fps))])
+video_frames = []
+for frame_index in frame_indices:
+    img = vr[frame_index].asnumpy()
+    video_frames.append(img)
+video_frames = np.stack(video_frames)
+
+# Prepare inputs
+messages = [{"role": "user", "content": [{"type": "video", "video": video_frames}, {"type": "text", "text": qs}]}]
+inputs = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = tokenizer(inputs, return_tensors="pt").to(model.device)
+
+# Generate response
+with torch.inference_mode():
+    output_ids = model.generate(
+        **inputs,
+        do_sample=False,
+        temperature=0.2,
+        max_new_tokens=128,
+        use_cache=True,
+    )
+
+pred = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
+print(pred)
+```
+
+## Citation
+
+If you find this work useful, please cite the paper:
+
+```bibtex
+@article{zeng2025distime,
+  title={DisTime: Distribution-based Time Representation for Video Large Language Models},
+  author={Zeng, Yingsen and Huang, Zepeng and Zhong, Yujie and Feng, Chengjian and Hu, Jie and Ma, Lin and Liu, Yang},
+  journal={arXiv preprint arXiv:2505.24329},
+  year={2025}
+}
+```
+
+## Acknowledgement
+
+DisTime is developed with the codebases of the following projects: [InternVL](https://github.com/OpenGVLab/InternVL) and [LLaVA-NeXT](https://github.com/LLaVA-VL/LLaVA-NeXT). We would like to express our sincere gratitude to these open-source contributions, which have greatly facilitated our research and exploration of time representation for video large language models.