Hugging Face's logo

CRAG - MM Challenge 2025
Team
company
https://www.aicrowd.com/challenges/meta-crag-mm-challenge-2025
Activity Feed

AI & ML interests

None defined yet.

Recent Activity

jackiewang92  updated a Space about 2 months ago
crag-mm-2025/README
jackiewang92  updated a dataset about 2 months ago
crag-mm-2025/web-search-index-public-test
jackiewang92  updated a dataset about 2 months ago
crag-mm-2025/web-search-index-validation
View all activity

Organization Card
Community About org cards

CRAG-MM 2025 Challenge

crag-mm-logo

🚀 Important Links: 🏆 Official Challenge Page | 📖 CRAG-MM Starter Kit

Welcome to the official Hugging Face space for datasets and indices related to the CRAG-MM 2025 Challenge (KDD Cup 2025).

Below are the core datasets and search indices needed for the CRAG-MM benchmark, organized in tables for clarity.

📌 Single-Turn Resources

Type Resource Description Latest Release Previous Release
📚 Dataset 🤗 crag-mm-single-turn-public Single-turn conversations, images, and metadata v0.1.2 v0.1.1
🌐 Web Search Index 🤗 web-search-index-validation Web index for validation (via CRAG Search API) Dataset version: v0.1.2, Index: v0.6, API: v0.5.0 Dataset version: v0.1.1, Index: v0.4, API: v0.4.0
🌐 Web Search Index 🤗 web-search-index-public-test Web index for public test (via CRAG Search API) Dataset version: v0.1.2, Index: v0.6, API: v0.5.0
🖼️ Image Search Index 🤗 image-search-index-validation Image index for validation (via CRAG Search API) Dataset version: v0.1.2, Index: v0.5, API: v0.5.0 Dataset version: v0.1.1, Index: v0.4, API: v0.4.0
🖼️ Image Search Index 🤗 image-search-index-public-test Image index for public test (via CRAG Search API) Dataset version: v0.1.2, Index: v0.5, API: v0.5.0

Note: For the final evaluation, we will use a different (and larger) index. It will cover everything in the validation and "public_test" index, so you can assume that the behaviors are similar.

📌 Multi-Turn Resources

Type Resource Description Latest Release Previous Release
📚 Dataset 🤗 crag-mm-multi-turn-public Multi-turn conversations with images and URLs v0.1.2 v0.1.1
🌐 Web Search Index 🤗 web-search-index-validation Web index for validation (via CRAG Search API) Dataset version: v0.1.2, Index: v0.6, API: v0.5.0 Dataset version: v0.1.1, Index: v0.4, API: v0.4.0
🌐 Web Search Index 🤗 web-search-index-public-test Web index for public test (via CRAG Search API) Dataset version: v0.1.2, Index: v0.6, API: v0.5.0
🖼️ Image Search Index 🤗 image-search-index-validation Image index for validation (via CRAG Search API) Dataset version: v0.1.2, Index: v0.5, API: v0.5.0 Dataset version: v0.1.1, Index: v0.4, API: v0.4.0
🖼️ Image Search Index 🤗 image-search-index-public-test Image index for public test (via CRAG Search API) Dataset version: v0.1.2, Index: v0.5, API: v0.5.0

👉 Check out the Starter Kit for detailed code examples and baseline models.

⌚️ Release History

[June 2nd] We released new web search index for validation and public_test. Current tag version is v0.6.

  • fixed corrupted validation index, resulting in unexpected retrieval results.
  • added additional documents (10-20% more web chunks) to validation and public-test search corpus respectively to improve recall.
  • no change to image search index
  • no change to the seach API

[May 25th] We released v0.1.2 validation and public_test data alongside new validation and public_test indices (v0.5). Major changes include:

  • Cleaner data: Removed low-quality or ambiguous image–QA pairs.
  • Expanded volume: More samples added across all three tasks.
  • Bug fixes: Minor corrections (e.g. consistent domain values).
  • Improved web search: Upgraded to BGE text encoder.

[April 24th] We released a new API version (v0.4.0) alongside new validation indices (v0.4). Major changes include:

  • fixed the corrupted chromadb index for both web and image search.
  • upgraded web search API: enabled web chunk level similarity search, while previous version is at page level. page urls are deduped during retrieval.
  • enable downloading the index from Huggingface repo by tags. previous index was tagged with v0.3. current index is v0.4 / main. we recommend always loading from main.
  • reset num_threads to os.cpu_count() when loading the web/image search index.

models 0

None public yet