Interfacing Foundation Models' Embeddings
Authors: name: Xueyan Zou; name: Linjie Li; name: Jianfeng Wang; name: Jianwei Yang; name: Mingyu Ding; name: Zhengyuan Yang; name: Feng Li; name: Hao Zhang; name: Shilong Liu; name: Arul Aravinthan; name: Yong Jae Lee; name: Lijuan Wang
Published: 2023-12-12 18:58:02

Abstract: We present FIND, a generalized interface for aligning foundation models'
embeddings. As shown in teaser figure, a lightweight transformer interface
without tuning any foundation model weights is enough for a unified image
(segmentation) and dataset-level (retrieval) understanding. The proposed
interface has the following favorable attributes: (1) Generalizable. It applies
to various tasks spanning retrieval, segmentation, etc., under the
same architecture and weights. (2) Prototypable. Different tasks are able to be
implemented through prototyping attention masks and embedding types. (3)
Extendable. The proposed interface is adaptive to new tasks, and new models.
(4) Interleavable. With the benefit of multi-task multi-modal training, the
proposed interface creates an interleaved shared embedding space. In light of
the interleaved embedding space, we introduce the FIND-Bench, which introduces
new training and evaluation annotations to the COCO dataset for interleave
segmentation and retrieval. Our approach achieves state-of-the-art performance
on FIND-Bench and competitive performance on standard retrieval and
segmentation settings. The training, evaluation, and demo code as well as the
dataset have been released at https://github.com/UX-Decoder/FIND.