ULIP-2: Towards Scalable Multimodal Pre-training For 3D Understanding
Authors: name: Le Xue; name: Ning Yu; name: Shu Zhang; name: Junnan Li; name: Roberto Martín-Martín; name: Jiajun Wu; name: Caiming Xiong; name: Ran Xu; name: Juan Carlos Niebles; name: Silvio Savarese
Published: 2023-05-14 23:14:09

Abstract: Recent advancements in multimodal pre-training methods have shown promising
efficacy in 3D representation learning by aligning features across 3D modality,
their 2D counterpart modality, and corresponding language modality. However,
the methods used by existing multimodal pre-training frameworks to gather
multimodal data for 3D applications lack scalability and comprehensiveness,
potentially constraining the full potential of multimodal learning. The main
bottleneck lies in the language modality's scalability and comprehensiveness.
To address this bottleneck, we introduce ULIP-2, a multimodal pre-training
framework that leverages state-of-the-art multimodal large language models
(LLMs) pre-trained on extensive knowledge to automatically generate holistic
language counterparts for 3D objects. We conduct experiments on two large-scale
datasets, Objaverse and ShapeNet55, and release our generated three-modality
triplet datasets (3D Point Cloud - Image - Language), named "ULIP-Objaverse
Triplets" and "ULIP-ShapeNet Triplets". ULIP-2 requires only 3D data itself and
eliminates the need for any manual annotation effort, demonstrating its
scalability; and ULIP-2 achieves remarkable improvements on downstream
zero-shot classification on ModelNet40 (74% Top1 Accuracy). Moreover, ULIP-2
sets a new record on the real-world ScanObjectNN benchmark (91.5% Overall
Accuracy) while utilizing only 1.4 million parameters(~10x fewer than current
SOTA), signifying a breakthrough in scalable multimodal 3D representation
learning without human annotations. The code and datasets are available at
https://github.com/salesforce/ULIP.