Title: Learning Deep Modality-Shared Self-Expressiveness for Image Clustering with Textual Information
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: deep clustering, self-expressive model, multimodal

Abstract: Leveraging textual information for image clustering has emerged as a promising direction, driven by the powerful representations of vision-language models. However, existing approaches usually leverage modality alignment, which merely shapes the representations implicitly, failing to preserve and exploit modality-specific structures, and leaving the overall representation distribution unclear. In this paper, we propose a simple but principled approach, termed deep modality-shared self-expressive model (DeepMORSE), which simultaneously learns structured representations that conform to the union of modality-specific subspace structures and, via a modality-shared self-expressive model, discovers structures shared across modalities. We evaluate our DeepMORSE approach on seven widely used image clustering benchmarks and observe performance improvements exceeding 4\% on the UCF-101, DTD-47, and ImageNet-Dogs datasets. In addition, we demonstrate the strong transferability of the learned representations by achieving state-of-the-art performance on downstream tasks such as image retrieval and zero-shot classification—without requiring any task-specific losses or post-processing.