ROMA-7B

I saw, I touched, I understood.

This repository hosts the model checkpoint of ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception. The code, web demo, and usage instructions are in the GitHub repository. The datasets (ROMI-2K and ROMA Bench) are released separately in GeWu-Lab/ROMA.

Introduction

Humans build an understanding of the physical world through an active process: when sensory evidence is insufficient, we decide what information is missing, how to acquire it, and when enough evidence has been obtained. Existing multi-sensory robot systems, in contrast, mostly integrate whatever sensory inputs they are given.

ROMA is an LLM-based system for Real-World Object-Centric Multi-Sensory Active Perception. It integrates vision, audio, touch, and force into a reasoning-interaction-feedback loop: the LLM identifies the missing evidence and selects the target object, the interaction (lift, press, collide, shake, rotate, squeeze), and the sensory modalities, while a physical interface executes the interaction and returns the multi-sensory feedback.

ROMA-7B is the multi-sensory LLM at the core of this system. It is built on Qwen2.5-Omni-7B and extended with:

• action / modality tokens for choosing interactions and sensory modalities;
• an audio branch (fine-tuned audio encoder and adapter) for contact sounds;
• an AnyTouch 2 tactile branch (tactile encoder and adapter).

It is trained with multi-sensory alignment followed by active-perception SFT.

Performance

Active perception task success rate on ROMA Bench (%). See the paper for the full results.

Model	ROMA Bench (total)	Real-world execution (total)
GPT-5.4	43.5	40.2
Gemini 3.5 Flash	53.0	41.7
Qwen 2.5-Omni	22.0	23.5
Qwen 3-Omni	19.7	23.5
ROMA-7B	72.9	61.4

Checkpoint Files

File	Description
model-0000*-of-00005.safetensors, model.safetensors.index.json	Base Qwen2.5-Omni-7B weights
anytouch2.pth, config-anytouch2.json	AnyTouch 2 tactile encoder
audio.bin	Fine-tuned audio adapter and encoder
tactile.bin	Fine-tuned tactile adapter
ROMA-LLM.bin	ROMA LLM weights
config.json, generation_config.json, preprocessor_config.json, chat_template.json, spk_dict.pt	Model, generation, and processor configs
tokenizer.json, tokenizer_config.json, vocab.json, merges.txt, added_tokens.json, special_tokens_map.json	Tokenizer

Usage

ROMA-7B cannot be loaded with plain transformers: the tactile modality and the extra adapters are loaded by the code in the GitHub repository, in the same order as during training (base Qwen2.5-Omni, tactile encoder, new tokens, audio adapter, tactile adapter, ROMA LLM).

1. Set up the environment following the Installation section of the GitHub repository.

2. Download this checkpoint:

   hf download GeWu-Lab/ROMA-7B --repo-type model --local-dir resources/models/ROMA-7B
   

   The demo reads the checkpoint directory from MODEL_PATH_2_5 in config/global_name.py, which defaults to resources/models/ROMA-7B. Edit it if you download the checkpoint elsewhere.

3. Launch the web demo:

   python demo.py --gpu 0 --port 7860
   

   Then open http://localhost:7860, type a question (or pick an example), and press Ask ROMA.

The full checkpoint is about 39 GB on disk.

Related Resources

• ROMI-2K (dataset): a real-world multi-sensory object interaction dataset covering nearly 2,000 objects and 6 atomic interactions with synchronized visual, audio, tactile, and force feedback.
• ROMA Bench (dataset): 2,100 scene-level tasks (single-chain, multi-chain, and intent-driven) for evaluating active perception.

The datasets are hosted in a separate repository, GeWu-Lab/ROMA, and are not included in this model repository. See the project page and the GitHub repository for details.

License

This model is released under the Apache 2.0 license (see LICENSE), following its base model Qwen2.5-Omni-7B.

Citation

If you find this work useful, please consider citing:

@article{feng2026roma,
  title   = {ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception},
  author  = {Feng, Ruoxuan and Chen, Yutong and Song, Ruihua and Yang, Huan and
             Wang, Zhongyuan and Yao, Guocai and Hu, Di},
  journal = {arXiv preprint arXiv:2610.06955},
  url     = {https://arxiv.org/abs/2610.06955},
  year    = {2026}
}

Acknowledgements

ROMA builds on Qwen2.5-Omni, AnyTouch 2, and AnyGrasp. We thank the authors for their open-source contributions.