Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Summary:
* This paper proposes a novel contrastive learning method of sentence embeddings, using unpaired examples from another modality. Experiments on semantic textual similarity benchmarks demonstrate that this method is able to produce sentence embeddings higher quality.

strengths_and_weaknesses: Strength:
* This paper presents a novel method to conduct multimodal contrastive learning for sentence encoders. 
* This method is simple and generalizable, i.e. not only applied to vision but also to other modalities such as audio. 

Weakness:
* I’m not confident that the improvement is significant. On STS tasks, on average it improves 1% accuracy.  In comparison, the original SimCSE paper improves more than 4% accuracy and is able to push the state-of-the-art in a supervised setting.  I wonder how much variance these datasets have, and how much hyper-parameters sweeps is conducted for both methods. 
* The model setup is a bit unintuitive. In order to conduct multimodal contrastive learning, it assumes that we can replace the embedding layer of a sentence encoder with a visual embedding layer to get an image encoder. 

questions: * Did you do hyper-parameters search for both SimCSE baseline and VisualCSE method?
* Have you tried to compare to SimCSE with “supervised models” setting?
* What’s the intuition behind sharing everything else besides the embedding layer for image and sentence encoders? 
* Have you tried tasks besides STS?


limitations: n/a
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work drives a plausible approach for enhancing unsupervised sentence embedding learning by leveraging unpaired examples from another modality. Motivated by the previous success of Lu et al. [33], Transformer models can transfer knowledge by learning from another modality, e.g. vision or audio. The text part uses the same training paradigm as SimCSE, while an additional image contrastive loss is incorporated by using the same language transformer for image encoding. Results show that the proposed method is language-agnostic and outperforms vanilla SimCSE across various tasks. Analysis versus supervised SimCSE also illustrated that the proposed approach performs better than SimCSE learned with a noisy and smaller NLI dataset.
strengths_and_weaknesses: Strengths:
- The idea of using non-linguistic supervision for sentence embedding learning is novel.
- The proposed model does not require paired multi-modal data for training, which can be applied to low-resource language learning.
- The finding that Transformer models can generalize by learning a similar task across different modalities may shed light on future research on multi-modal representation learning.
- The experiments demonstrate its success on both VisualCSE and AudioCSE.

Weaknesses:
- Though the authors attempted to answer the underlying rationale of the proposed model in Sec. 3.4, how the language model can be trained using other modalities remains unclear to me. More analysis experiments are expected to strengthen the theoretical support for the framework.
- The work has discussed how language dataset supervision can affect the final performance. However, the non-linguistic dataset quality is not discussed. What kind of non-linguistic dataset is suitable for the paradigm? Does ImageNet fit for all language training?

Minors:
- Line 172: "thesame" $\rightarrow$ "the same"
questions: Please refer to weaknesses for score-related questions.

- Can the supervision on non-linguistic dataset (e.g. image labels) further enhance the proposed model?
limitations: The authors discussed limitations on the reduction in Uniformity of the sentence embeddings. Other possible limitations could be the marginal improvement with larger non-linguisitic dataset.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This work is an extension of existing contrastive sentence representation learning. Instead of only using text data, this work further bring the other modality to improve the model performance. In this work, the author proposed unpaired image or audio data as additional supervision signals. The extra modality data share the same encoding backbone models and similar contrastive learning losses. The author conducted experiments on standard sentence similarity evaluation datasets, and achieved improvements over text-only models. 
strengths_and_weaknesses: Strengths: The proposed idea is simple and reasonable. By introducing image/audio through the same transformer encoder, the model transferred from text only to modal agnostic model. This is quite different from other works which use separated structures for each modality. One good point of the model is that the image data size (500) is much smaller than the text training data. This means the model extra cost is small. The author also well analyzed the proposed model.

Weakness: The model performance improvement is relatively small. Compared to other CL sentence representation learning work, e.g., DiffCSE, DebiasedCSE, PromptBERT, the proposed method performance with more data is not that significant. The author also did not evaluate the performance on other modalities, then it is hard to give a overall judgment of the model as a modal agnostic mode. The overall training loss on different modality losses are also missing which in my view in also important to understand the contribution of different modalities. 
questions: 1. Is it possible to use same CL losses for different modalities?
2. What if we use both image and audio data together? Can if further improve the perforamnce?
3. Is there token-level alignment between different modals?
4. If the proposed mode can further improve the image-text alignment compared with our models, e.g., CLIP?
limitations: I do not see important social limitations of this work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the non-linguistic supervision to contrastive-learning-based sentence embedding learning. The author proposes a Transformer-based multi-modal framework to learn text-contrastive and vision/audio-contrastive tasks with unpaired samples. Experiments on various semantic textual similarity tasks show the effectiveness of the proposed method.  

strengths_and_weaknesses: *Summary Of Strengths
- The paper is clearly presented and well-written. 
- Extensive empirical experiments with multiple runs not only clearly show the pros/cons for the sts tasks but may suggest the path for future improvement in the low-resource language. 
- A simple yet effective modification is proposed and shows consistent improvements over all systems.  

*Summary Of Weaknesses
- It seems the paper is missing a comparison with MCSE [1], which is a very relevant work that augments the SimCSE with multimodal contrastive objective; 
- It is unclear, except for the objective and hyperparameters, how other factors like pre-trained image/audio encoder, augmentation methods, dataset, and image/audio quality could affect the final performance of the proposed methods; 
- It is unclear to me how many epochs/images will the proposed methods improve over the SimCSE methods and how will that affect the final quality; 

[1] Zhang, Miaoran, Marius Mosbach, David Ifeoluwa Adelani, Michael A. Hedderich, and Dietrich Klakow. "MCSE: Multimodal Contrastive Learning of Sentence Embeddings." arXiv preprint NAACL (2022).
questions: - Could the authors elaborate more on the deficiency of the proposed method in the supervised setting (is this also true for different languages), and could that be alleviated with more unlabeled training data? 
- Could the authors elaborate on the epochs used for training the audio and vision model, and will the quality/amount of image/audio affect the performance of the proposed methods? 
limitations: - It would be good for more discussion around the usage (quality and amount) of external multi-modality datasets and will aligned dataset can also be leveraged by the proposed method;
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a contrastive method for learning sentence embeddings that incorporates additional modalities using unaligned data. The method follows the SimCSE approach of clustering similar sentence representations and expands upon it by additionally clustering visual or audio input, named VisualCSE and AudioCSE respectively. Because the approach does not require aligned data between the text and additional modality, the method works for any language and additional modalities can be easily evaluated as well. The authors show that VisualCSE and AudioCSE have clear performance gains in both new approaches across a range of downstream tasks compared to the baseline SimCSE. Additional ablations and experiments also show that the learned sentences embeddings have higher alignment and perform similarly to supervised CSE when training with less data or increasing the level of noise.
strengths_and_weaknesses: **Strengths**

*originality:*

- This is a really interesting concept that I'm a big fan of! It's an interesting take on using multimodal input to benefit language learning by using another modality in a completely unaligned way to improve the language embeddings.

*quality:*

- This paper shows a number of experiments analyzing different aspects of the approach -- raw performance on downstream tasks when compared to SimCSE, performance across 2 modalities compared to text-only training, performance across 3 models, performance across different seeds, performance across different languages, and ablations. Because there are so many experiments, it's easy to analyze and think about the contributions of the approach for different communities.
- The visual ablation was interesting as it showed whether supervision from the visual domain, in the form of labels, helps performance. Seeing that it did not contribute much, likely due to the smaller batch size, also leaves room for future work.
- Section 5.1 is another strong ablation, providing a relationship between the number of examples and amount of noise in supervised SimCSE versus unsupervised VisualCSE.
- Experimental setup is thorough, especially by comparing across seeds and therefore across different subsets of the training data.

*clarity:*
- The paper is clearly written and I could follow the motivation, experimental setup and results without really needing additional background searches.

*significance:*
- Not requiring aligned data means virtually any modality can be plugged in here; in addition, the data being used in the existing modalities can easily be expanded or modified. In addition, not being tied to single language is a major contribution! This is especially true when incorporating additional modalities as most work using images is quite limited in the number of languages with annotations.
- For a quantitative perspective, the results in Table 2 show that VisualCSE and AudioCSE nearly always outperform SimCSE across different models and across downstream tasks. These are impressive gains and really validates the idea of the approach.

**Weaknesses**
*clarity:*
- In equation's 2 and 3, the loss's superscript wasn't immediately clear to me on the first read and doesn't make sense there to me. Can these just be explicitly mentioned in the text (along the lines of "The unsupervised loss based on data augmentation is as follows" then "For a supervised task like NLI, its loss is defined as:").

*significance:*
- While the authors showed the benefits of VisualCSE on languages other than English (French, German and Russian), I would have liked to see a low-resource language from outside of Europe.
questions: *clarity:*
- In equation's 2 and 3, the loss's superscript wasn't immediately clear to me on the first read and doesn't make sense there to me. Can these just be explicitly mentioned in the text (along the lines of "The unsupervised loss based on data augmentation is as follows" then "For a supervised task like NLI, its loss is defined as:").

*significance:*
- While the authors showed the benefits of VisualCSE on languages other than English (French, German and Russian), I would have liked to see a low-resource lanugage like TODO. This would likely require using another model framework besides BERT.

*questions:*
- In the optimization details section, can you expand on how the number of epochs were chosen for training? It appears to be similar to CSE, though 1 epoch seems quite small.
- Any consideration about intentionally batching images of the same label together to better evaluate the contribution of image labels?
- Seeing how image labels were used for SupCon, is there a correlate experiment in the audio domain to incorporate labels?

*typos:*
- L34: dataset -> datasets
- L43: difference -> different
- L124: representation -> representations
- L218: the improve -> the improvement
- L286: of labeled dataset -> of the labeled dataset
- L345: extend -> extent

*Citations:*
MCSE: Multimodal Contrastive Learning of Sentence Embeddings -> This is essentially concurrent work (put on arXiv in April, presented at NAACL), so it's understandable that it's not cited. I would suggest reading and editing a few small aspects of the paper, e.g. saying this work is the first to look at this approach.
limitations: yes, limitations are addressed
review_title: null

Meta-review: metareview: This paper improves contrastive learning of sentence embedding by using unpaired examples from the image or audio modality.

Reviewers liked the significance of this work due to its simplicity and general applicability, but some questioned the amount of improvement and advocated for the inclusion of low resource languages. The authors included Chinese, which is non-European but not low resource.
confidence: Certain
Decision: Accept