Towards Out-of-Modal Generalization without Instance-level Modal Correspondence

Authors: Zhuo Huang, Gang Niu, Bo Han, Masashi Sugiyama, Tongliang Liu
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Multi-Modal Learning; Generalization

Abstract:
The world is understood from various modalities, such as appearance, sound, language, etc. Since each modality only partially represents objects in a certain physical meaning, leveraging additional ones is beneficial in both theory and practice. However, exploiting novel modalities normally requires cross-modal pairs corresponding to the same instance, which is extremely resource-consuming and sometimes even impossible, making knowledge exploration of novel modalities largely restricted. To seek practical multi-modal learning, here we study Out-of-Modal (OOM) Generalization as an initial attempt to generalize to an unknown modality without given instance-level modal correspondence. Specifically, we consider Semi-Supervised and Unsupervised scenarios of OOM Generalization, where the first has scarce correspondences and the second has none, and propose connect & explore (COX) to solve these problems. COX first connects OOM data and known In-Modal (IM) data through a variational information bottleneck framework to extract shared information. Then, COX leverages the shared knowledge to create emergent correspondences, which is theoretically justified from an information-theoretic perspective. As a result, the label information on OOM data emerges along with the correspondences, which help explore the OOM data with unknown knowledge, thus benefiting generalization results. We carefully evaluate the proposed COX method under various OOM generalization scenarios, verifying its effectiveness and extensibility.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729514462851

thread_type:
official_review

<thread_object>
creator: reviewer_knt4

<content>
summary:
This paper propose a novel out-of-modal generalization problem that focuses on generalizing learned knowledge to unknown modalities. In addition, it considers both semi-supervised and unsupervised scenarios. The proposed connect & explore (COX) scheme tries to explore the shared and unique between modalities via variational information bottleneck. The proposed method can build connections between known and unknown modalities, further generating correspondences to explore more knowledge. The expleriments verify the effectiveness of COX.

strengths:
1. This paper proposes a novel OOM problem, which is valuable because of the importance on generalizing exsting knowledge to unknown modalities without paired instances. 
2. The proposed COX method sounds reasonable and explores the connections between different modalities without priori knowledge. The motivation and method are clear to me. The results are convincing to me.

weaknesses:
1. I acknowledge the novelty and value of the proposed OOM scenario. However, since the method lacks paired samples, it only shows limited effectiveness in learning shared knowledge, making it significantly inefficient in exploring unknown modalities. This is evident in the experimental results, which show a large gap compared to "aligned". Although the compared methods contain ERM and SSL, I question whether the proposed approach is better than SOTA unsupervised uni-modal solutions. Only surpassing or enhancing existing uni-modal methods with the proposed approach can make it really valuable. 
2. As I said in 1, the comparison with existing uni-modal methods is necessary.

questions:
see in Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730067831916

thread_type:
official_review

<thread_object>
creator: reviewer_ode6

<content>
summary:
This paper addresses the problem of out-of-model (OOM) generation and proposes a novel technique, COX. The authors perform experiments on five datasets, comparing their method with several baseline approaches.

strengths:
The problem statement is clear.

weaknesses:
Insufficient and weak baselines: I am especially concerned the proposed approach's performance compared with SSL. 

Some notations should be improved, and a few detailed explanations should be added.

questions:
From my understanding, the authors consider a scenario where the training data consists of two parts: IM, where both inputs and labels are available, and OOM, where only inputs are provided without any labels (unsupervised case), and only a limited amount of labeled data is present. Please correct me if I’m mistaken, but it seems that no data contains both IM and OOM inputs. In other words, there is no data where the pair (x1, x2)—representing two modalities—is known.

If the above assumption is correct, how do we sample from the joint distribution p(xO | xI)p(xI) in Equation (6)?

I find the assumption stated below Equation (1) too strong. It implies that the proposed method only applies when xO can be directly inferred from xI. For example, let’s assume a1 = a2 = a, representing the sound of a dog barking, and b1 and b2 represent images of a golden retriever and a border collie, respectively. V is a constant vector, and Y represents the class "golden retriever." While it is true that P(V|xI=a1)P(Y|xI=a1) = P(V|xI=a2)P(Y|xI=a2), given that a1 = a2, we see that P(V,Y=is_golden_retriever|xO=b1, xI=a1) does not equal P(V,Y=is_golden_retriever|xO=b2, xI=a2).

In Line 255, do h1* and h2* refer to the classifiers corresponding to modality 1 and modality 2, respectively? If so, how can they be applied to xO, which belongs to a different modality?

The baselines should be reconsidered, especially by including stronger baselines. For instance, there are many state-of-the-art semi-supervised learning (SSL) approaches, and simply adding Gaussian perturbations to the input may not be sufficient. Stronger SSL baselines are crucial, as otherwise, it could be argued that using SSL alone is enough, which raises the question, why COX? Additionally, the 'Random' baseline is too simplistic, effectively leaving the unsupervised setting in Table 2 with only one baseline. 

Furthermore, for the semi-supervised case, it is not surprising that the baseline performs worse, given the very limited amount of labeled data used for training.

I would like to bring [1] to the authors' attention, which could be helpful for better understanding multimodality:

[1] Zihui Xue et al., The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation, ICLR 2022.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730208439286

thread_type:
official_review

<thread_object>
creator: reviewer_pujv

<content>
summary:
This work focuses on the semi-supervised and  unsupervised scenarios of *Out-of-Model* (OOM) Generalization without given instance-level model correspondence. This work proposes an OOM generalization method based on the interactive relationship between modalities, *connect & explore* (COX), in an attempt to extract the commonality that can help partially comprehend OOM data based on IM data. It introduce a variational information bottleneck framework to connect OOM data and In-Model (IM) data and extract shared information. Finally, several sets of experiments demonstrate the effectiveness and extensibility of the proposed method.

strengths:
This work focuses on the novel and practical issues in the field of *Out-of-Model* (OOM) generalization.

This work proposes an approach that attempts to combine common knowledge based on connections across modalities with unique knowledge of OOM data based on modality disagreement.

This work conducts multiple sets of experiments to verify the effectiveness and extensibility of the proposed method.

weaknesses:
(i) The OOM problem has also been discussed in other works, such as IB for modality missing, cross-modal generalization, etc., ([1-5] for example) but not mentioned in the paper (nor experiments). Please explain the difference between this paper and these works. [1] Unpaired image-to-speech synthesis with multimodal information bottleneck. [2] Visual explanations of image-text representations via multi-modal information bottleneck attribution. [3] Dynamic Multimodal Information Bottleneck for Multimodality Classification. [4] SimMMDG: A simple and effective framework for multi-modal domain generalization

(ii) The work mentions "prediction uncertainty" in section 4.2, but after carefully reviewing all the contents, it is hard to find the definition or description of prediction uncertainty, while without the direct description of this concept in the experiment section for verification. 

(iii)The work mentions "the performance of COX is affected by the quality of IM perceptors", but only contrast the performance between ImageBind and LanguageBind in the experiment section can not fully illustrate this conclusion.

(iv) Has some loopholes, e.g.,

  - Tense and grammatical errors, e.g., "OMM data" -> "OOM data"
  - Confusing notations, e.g.,"given IM data *X<sup> O </sup>*and OOM data *X<sup> I </sup>*"

Please correct the grammatical mistakes and polish them if possible.

(v) In the experimental part, the results in Table 2 and Table 3 can not adequately illustrate this conclusion "leveraging the knowledge from IM perceptors can indeed help OOM generalization compared to using OOM data alone". How do the results in Table 2 and Table 3 show the performance improvement of the proposed method compared to the methods using only OOM data?

questions:
Please see 'weakness', which simply can be summarised as:

(i) What the advantages of this work vs. previous works?

(ii) What is "prediction uncertainty" described in the article and how is it reflected? 

(iii) Can the conclusion that COX performance is affected by quality of IM perceptors be fully explained based on the final performance results of only two IM perceptors, ImageBind and LanguageBind? What are the quality differences between ImageBind and LanguageBind reflected?

(iv) How do the data in Table 2 and Table 3 reflect that leveraging the knowledge from IM perceptors can indeed help OOM generalization compared to using OOM data alone?

(v) There are some grammatical errors and confusing notations in the article. Can it be further polished and revised?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730528728447

thread_type:
official_review

<thread_object>
creator: reviewer_zg1q

<content>
summary:
This paper explores a novel model generalization problem, specifically focusing on out-of-modal generalization in the absence of label and modality correspondence. The authors employ a two-stage generalization approach, first constructing modality-shared information and then focusing on modality-specific information. They propose different training losses for both semi-supervised and completely unsupervised scenarios.

strengths:
1.The problem addressed in the paper is quite novel, and the semi-supervised scenario provides valuable insights."

2.Extensive experiments are conducted to demonstrate the effectiveness of the proposed method.

weaknesses:
1.The paper contains several typos that affect readability and need to be corrected. Regarding line 195, it seems that the symbols for IM data $ X^O$ and OOM data $ X^I$ are used incorrectly. Also in Eq.3, why is there an integral sign before $p ( X^O, X^I ) $

2.The proof in the paper lacks rigor. How is the lower bound derived from Equation 3 to Equation 4? Although the proof draws on VIB, there are still significant discrepancies.

3.Similarly, the proof in Appendix A.1 for Equation 20, which relies on the non-negativity of the KL divergence, is not sufficiently rigorous.

4.In Section 3.2, there is a lack of necessary descriptions regarding the trained model, such as the input and output of the decoder $q( X^O, X^I)$.

questions:
1.Does the inclusion of $L_{con}$ with label $ y $ imply that exploring connections across modalities requires the use of samples with correspondence for training?

2.In a completely unsupervised setting, the feature distributions of different modalities may differ significantly. Is the choice of the anchor method reasonable in this context?

3.Could you provide a more detailed proof regarding the upper and lower bounds in Sec 3.2? 

4.In the ablation experiments, the results of removing any module are significantly lower than those of EntMin. Would applying these two losses individually on EntMin lead to improvements?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691782264

thread_type:
official_review

<thread_object>
creator: reviewer_2ymo

<content>
summary:
This paper proposes a novel method for Out-of-Modal (OOM) generalization, which uses a COX framework to achieve cross-modal knowledge transfer and generalization to unknown modalities. At the same time, this paper considers both semi-supervised and unsupervised scenarios. Experiments show that this method has a certain level of effectiveness.

strengths:
1. The motivation behind this approach is clear, and studying out-of-model (OOM) generalization is meaningful.
2. The authors provide a detailed description of the proposed method, making it easy to follow and reproduce.

weaknesses:
1. In the theoretical analysis, the paper uses $h^*$ to represent ideal optimal classifiers. However, in practice, it is difficult to find the optimal classifier. Although the authors compared the performance of ImageBind and LanguageBind, they should conduct more experiments to demonstrate the impact of using classifiers with different levels of accuracy on the final results.
2. The paper does not theoretically or experimentally explain why using the IM Perceptor is more effective than directly performing semi-supervised or unsupervised training.
3. If using the IM Perceptor indeed improves training results as mentioned in the issues above, then for the same task, would knowing more modalities enhance the generalization of OOM?
4. The paper contains numerous writing errors, such as "OMM" in line 494. Additionally, the logic of symbol definitions in the mathematical proof section is confusing, for example, using $X^O$ to represent IM data and $X^I$ to represent OOM data.

questions:
Please see Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732181235136

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear Reviewers, AC, and SAC:

We deeply thank the hard work done by AC and SAC such as assigning reviewers, guiding the process, and further organizing the discussion. We also sincerely appreciate the reviewers for taking the time to read our paper, provide constructive opinions, and get involved in our discussion. Without your elaborative help, our paper could not have been improved.

We appreciate all the reviewers for their insightful and constructive reviews of our manuscript. Delightfully, we are glad that the reviewers found that:
- The proposed problem is **novel, insightful, and practical**. (Reviewers 2YMo, Zg1q, PuJV, and KNt4)
- Our research is **well-motivated, clearly-formulated, and easy-to-follow**. (Reviewers 2YMo, oDE6, and KNt4)
- The experimental evaluation is **effective, convincing, and extendable**. (Reviewers 2YMo, Zg1q, PuJV, and KNt4)

Based on all the comments from the reviewers, we provide a general response to the concerns raised by multiple reviewers. The individual responses are commented on below each review.

- Comparison with SOTA uni-modal baselines.
    - Since an unsupervised method that is modality-agnostic is hard to find, therefore, we employ a very competitive method Momentum Contrast (MoCo) as a baseline.
    - For the semi-supervised case, we use MoCo combined with EntMin to train OOM data.
    - For the unsupervised case, we directly employ MoCo. The result on MSR-VTT with R@1 is shown below:
        | Setting | Method      |  Aud |  Lan |  Vid |
        |---------|-------------|:----:|:----:|:----:|
        | Semi    | MoCo+EntMin | 20.5 | 21.1 | 23.4 |
        | Semi    | COX         | **23.3** | **23.4** | **26.5** |
        | Unsup   | MoCo        | 10.5 | 10.2 | 12.6 |
        | Unsup   | COX         | **13.5** | **16.5** | **15.2** |
        - We can see that under both scenarios, the performance improvement of COX is still significant, which justifies its effectiveness. We will incorporate such a baseline into our revision. If there are any other baseline methods to be incorporated, please let us know.

- Justification of "IM perceptors can help OOM generalization".
We vary the number of correspondences in OOM data from MSR-VTT to justify the benefits brought by COX.
    - We use ERM and EntMin as baselines, and vary the correspondence number to 90%, 60%, 30%, and 10%, as shown below:
        | Method   |  90% |  60% |  30% |  10% |
        |----------|:----:|:----:|:----:|:----:|
        | ERM      | 68.4 | 60.1 | 46.9 | 38.2 |
        | EntMin   | 69.2 | 61.6 | 48.5 | 39.4 |
        | COX      | 69.4 | 63.9 | 55.1 | 48.8 |
        | $\Delta$ |  0.2 |  2.3 |  5.6 |  9.6 |
        - First of all, we observe that COX brings more benefits when correspondence is more scarce.
        - This is because sufficient correspondence can maximally uncover the knowledge of OOM data. As correspondence gets less, the knowledge that can be explored from correspondence decreases.
        - However, COX leverages the knowledge from IM data which brings more benefits even with less correspondence.

For other detailed concerns, we have tried our best to fully address every single concern raised by each reviewer. Thanks to all reviewers for your valuable time in reviewing this paper, your help and support have largely improved this manuscript.

Since the interactive discussion phase has begun, we want you to know that **we are actively engaged in any discussion about any question, and will try our best to address all concerns or misunderstandings left.** Thank you so much for your time.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732523743083

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
We are here to address any remaining concerns

comment:
Dear Reviewers, AC, and SAC:

We want to thank all reviewers for taking the time to review this paper, all of them have been greatly helpful and supportive. Moreover, we would like to thank AC and SAC for handling and guiding the reviewing process. Without your effort, we could not have reached so far.

The interactive discussion phase is one of the most effective features of ICLR, which allows us to clarify misunderstandings and improve the quality of papers. We highly value such an opportunity to be open-minded and learn from the experts in this field. So, before the discussion ends in two days, we hope if there are **any questions or suggestions** from **all** reviewers or AC, please don't hesitate to let us know. We will be truly grateful to hear your valuable opinions.

Sincerely,  
Authors.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732767493662

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Paper Revision

comment:
Dear Reviewers and AC,

Thank you for all your constructive suggestions and comments. Here we have uploaded our revision on the following aspects:

- Incorporating all mentioned references and discussing their problem setting: **none of them** studies the problem of generalizing out-of-modality.
- Intuitively justifying our problem assumption through a **real-world** example, which is also consistent with the previous VIB **theoretical** framework.
- Providing detailed discussions of our training process based on the theoretical VIB framework.
- Incorporating additional baseline in the main comparison.
- Clarifying terms and notations.
- Conducting additional experiments to:
    - justify the benefits of leveraging IM data from IM perceptors.
    - study the effect of varied IM perceptor qualities on OOM performance.
    - testify the superiority of COX over competitive uni-modal baselines.

Moreover, for other specific questions or concerns, we have
- provided comprehensive details of our upper and lower bound derivation.
- release public source code to help understand our implementation.
- conducting additional ablations, such as various scales of IM perceptors, increasing modality numbers, and anchor methods.

We understand that you might be very busy at the time, if you could spare a little time to take a look at our revision, we would be genuinely grateful.

If any questions remain, we sincerely hope that you could kindly inform us. After all, we hope to make sure that no misunderstandings are left and a fair judgment of our contribution to this field can be made.

Sincerely,  
Authors.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733274965811

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary

comment:
Dear Reviewers, AC, and SAC:

We deeply thank the hard work done by AC and SAC such as assigning reviewers, guiding the process, and further organizing the discussion. We also sincerely appreciate the reviewers for taking the time to read our paper, providing constructive opinions, and getting involved in our discussion. Without your elaborative help, our paper could not have been improved.

Here we summarize our rebuttal from a high-level perspective which could hopefully help grasp our contribution and modification quickly.

Through rebuttal, several consensuses have been achieved:

- Our paper is **interesting, practical, and meaningful**.
- Our explanation is **clear and helpful**, the intuition of this paper is **straightforward**.
- We have carefully addressed every question with proper justification such that 4 out of 5 reviewers are willing to further consider raising the score while the other reviewer stays positive.

To achieve this, we have strengthened our paper by:
- Justifying the effectiveness of COX through comparison with additional baseline and state-of-the-art uni-modal methods.
- Enhancing the related work discussion and details about the training process.
- Analyzing the benefits of both IM data and IM perceptors with additional studies.
- Clarifying the derivations and assumptions.

During the discussion phase, we tried our best to connect with reviewers and address every single concern, such as:
- Providing source code.
- Explaining the derivation details.
- Justifying our assumptions through both existing studies and real-world understandings.

It is such a great pleasure to know that **all reviewers** find our explanation helpful, and we will keep making our maximum effort to improve this paper by organizing the source code for open-source, polishing the assumptions with clearer demonstration, etc.

For any further internal discussion results, we kindly hope the AC could let us know in the meta-review, and we are more than happy to address them in our future revisions.

Thanks again for all your effort and contribution, we could not achieve this far without you.

Sincerely,  
Authors.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents an early attempt to attack out-of-modal generalization issues in machine learning. It has experienced a fruitful author-reviewer discussion. All reviewers are now positive about this paper. I'm happy to suggest accepting this paper. Nonetheless, the authors are required to include the additional experiments and discussion in the final version.