Large (Vision) Language Models are Unsupervised In-Context Learners

Authors: Artyom Gadetsky, Andrei Atanov, Yulun Jiang, Zhitong Gao, Ghazal Hosseini Mighan, Amir Zamir, Maria Brbic
Primary area: foundation or frontier models, including LLMs
Keywords: llm, unsupervised, in-context learning, few-shot learning

Abstract:
Recent advances in large language and vision-language models have enabled zero-shot inference, allowing models to solve new tasks without task-specific training. Various adaptation techniques such as prompt engineering, In-Context Learning (ICL), and supervised fine-tuning can further enhance the model’s performance on a downstream task, but they require substantial manual effort to construct effective prompts or labeled examples. In this work, we introduce a joint inference framework for fully unsupervised adaptation, eliminating the need for manual prompt engineering and labeled examples. Unlike zero-shot inference, which makes independent predictions, the joint inference makes predictions simultaneously for all inputs in a given task. Since direct joint inference involves computationally expensive optimization, we develop efficient approximation techniques, leading to two unsupervised adaptation methods: unsupervised fine-tuning and unsupervised ICL. We demonstrate the effectiveness of our methods across diverse tasks and models, including language-only Llama-3.1 on natural language processing tasks, reasoning-oriented Qwen2.5-Math on grade school math problems, vision-language OpenFlamingo on vision tasks, and the API-only access GPT-4o model on massive multi-discipline tasks. Our experiments demonstrate substantial improvements over the standard zero-shot approach, including 39% absolute improvement on the challenging GSM8K math reasoning dataset. Remarkably, despite being fully unsupervised, our framework often performs on par with supervised approaches that rely on ground truth labels.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730209473755

thread_type:
official_review

<thread_object>
creator: reviewer_tgs1

<content>
summary:
This paper aims to enhance the zero-shot inference capabilities of existing open- and closed-weight models. To achieve this, it introduces a joint inference framework that processes and makes predictions for all task-related inputs simultaneously, rather than handling each input individually. The paper presents two efficient approximations—unsupervised fine-tuning and unsupervised in-context learning (ICL)—to reduce the computational complexity of joint inference. The results demonstrate the effectiveness of the proposed method across NLP and vision tasks, using two large language models (LLaMA-3 and GPT-4o) and one vision-language model (OpenFlamingo).

strengths:
- I believe the idea of self-refinement using model predictions is solid, as is the approach of zero-shot evaluation without a labeled support set, which contrasts with the traditional definition of supervised in-context learning (ICL). In my view, the paper aligns well with the concept of inference-time compute scaling, as evidenced by the O1 model by OpenAI, where generating the final answer requires multiple steps (turns) by the model. Hence, the problem definition is quite sound. 

- The paper presents two distinct approximations for the joint inference task, which appear to outperform all baselines. The authors demonstrate this rigorously across several NLP and vision tasks.

weaknesses:
In the following, I will provide a list of weaknesses and a follow-up question for each in case the authors want to reply.

1. **Model Cacapcity and ICL Abilities**: The proposed method assumes that the pretrained model already possesses few-shot in-context learning (ICL) abilities, which are typically found in large-scale models (those with over 8 billion parameters). What about models that don't have these ICL capabilities? Can they leverage their context more effectively during joint inference? To be more specific, if the authors use the Phi-1.5 model and conduct unsupervised fine-tuning or unsupervised ICL again, what would be the outcome?

2. **Reliance on Task Encoder Approximation**: I think the design of the Task Encoder has not been explored properly in the paper. The authors only did not mention why the same pre-trained model + LoRA can be a good approximation for the task encoder approximation tasks. Why is this case? Based on which ablation in the paper? What are the other alternatives? It’s important to note that, as shown in other research, LoRA often tends to underfit a task, which can lead to suboptimal or inconsistent improvements. This raises concerns that the proposed task inference model may not accurately reflect the tasks that the model is intended to predict.

3. **Limited Applicability to Open-Ended Tasks**: Aside from the single open-ended experiment mentioned in line 395 for the GSM8K benchmark, I do not see any other experiments demonstrating the effectiveness of the proposed method for open-ended generation. With just one experiment, it is not convincing that the method is effectively generalizable for both unimodal and multimodal open-ended generation. Do the authors have any additional ablations or experiments to support the strengths of their proposed method?

4. **Missing Compute-related Ablations**: In multi-turn inference settings, it is crucial to provide the number of FLOPs used to arrive at the final answer. Do the authors have any experiments that demonstrate the scaling curve, with FLOPs on the x-axis and performance on the y-axis?

5. **Lack of Proper Evaluation in Multi-modal Setting**: In the paper, the authors used OpenFlamingo and demonstrated that the proposed method shows some effectiveness. However, I'm curious about their design choice and why they opted for OpenFlamingo. What about other alternatives that may be more powerful and capable of handling larger contexts? For example, LLaVA-Next-Interleaved is a suitable candidate, as it is trained with the in-context learning (ICL) objective function. Could the proposed method work properly if LLaVA-Next-Interleaved were used instead of OpenFlamingo? Additionally, regarding multimodal evaluation, assessing a model using COCO-color and COCO-count might not provide an accurate reflection of the proposed method's performance. Have the authors considered evaluation on the MMMU Benchmark? This benchmark is recognized as a legitimate evaluation tool for multimodal large language models, and LLaVA-Next-Interleaved has reported strong results on it.

questions:
Asked above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730604184874

thread_type:
official_review

<thread_object>
creator: reviewer_nhh8

<content>
summary:
The paper introduces an unsupervised joint inference framework to improve the performance of LLMs and VLMs by considering the entire set of questions for a given task instead of making predictions on each question independently. They utilize the shared context of these questions to improve model accuracy It includes a self-improvement loop where the model iteratively refines its predictions by using its own prior answers as in-context examples to guide the next round of predictions. This self-refinement continues over multiple iterations, allowing the model to gradually improve and make more consistent predictions without needing labeled data. The authors show that this

strengths:
- the motivation of utilizing the entire corpus of the task to improve overall performance is compelling, as there is often shared context that is underutilized
- this method is versatile and applies to both open and closed source models as well as across modalities
- the results indicate that this can provide substancial improvements over zero shot methods

weaknesses:
- the main weakness of this paper is clarity: sections 3 and 4 are hard to follow and it would have been significantly clearer to provide a methods figure which outlines the iterations and IC examples used. If I understand this method correctly, it uses responses to questions from previous iterations as in context examples for questions in the current iteration, optimizing for the joint likelyhood over all the examples in the batch. If this is the case, then it should have been stated clearly in prose in section 4. I find the entirety of section 3 to be unnecessary as it is explaining commonly known procedures for model training and inference, and much of section 4 to use unnecessary notation in explaining a relatively straightforward method. Again, I have no major concerns about the method itself, but I urge the authors to remove or significantly shorten section 3, add a methods figure, and rewrite section 4. A methods figure could be in the form of showing the inputs to the LLM at each iteration for the no weight-update option to indicate the change in in context examples. 
- I am not fully convinced that adding iterations helps for the no weight update setting. if the authors added a 1-iteration baseline to their results this would resolve my concern
- all the benchmarks used in the results section are out of date and focus on easy tasks. The results would be much more compelling if the authors included a benchmark like MMLU. That being said, given the short rebuttal period I would put this experiment as a lower priority than the two above concerns 

I acknowledge that I may have misinterpreted the method so some of these critiques could be unfounded. If that's the case, I am happy to learn more about the method - preferably via a methods figure - in the rebuttal and change my rating

questions:
- how does your method answer multiple questions with one model call? What is the exact prompt used and how are the in context examples formatted? This should be in the appendix, table C1 does not indicate where in context examples are placed nor does is show how multiple questions are answered in a single pass. If the authors could give a template of the prompt used in the iteration steps for a benchmark that would be sufficient
- are the in context examples used different than the questions the LLM is tasked to answer in a given call?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730616382918

thread_type:
official_review

<thread_object>
creator: reviewer_uqwz

<content>
summary:
The paper proposes an 'unsupervised joint inference' framework to improve the performance of large language / vision-language models on tasks without using labeled data. The proposed approach predicts output sequences simultaneously for multiple inputs, leveraging inter-sample dependencies. There are two variants in the proposed approach: 1. use the model's own predictions to finetune its parameters. 2. Use in-context learning to refine predictions iteratively. The paper conducts experiments on some nlp and vision-language datasets and claim substantial improvements to vanilla zero-shot approach and often on-par performance with supervised approaches.

strengths:
1. The paper proposes an novel framework to adapt llm or vlm for unseen tasks. 
2. The paper intents to build solid theoretical foundation for the proposed approach. It establishes a theoretical connection between the proposed unsupervised ICL method and the joint inference objective, showing how the former approximates the latter.
3. The experiments conducted are diverse on different tasks: text classification, image classification, question answering, visual question answering, natural language inference, common-sense reasoning, and math problem-solving.
4. Experiments show that the proposed approach is effective and outperform the baselines by a large margin.

weaknesses:
1. The paper overclaims it's contribution and effectiveness of its approach. In abstract, the paper claims that 'our approach, although unsupervised, often performs on par with supervised approaches that use ground truth labels.' However, its an overclaim for the following reasons: (1) In the results shown in the paper, most of the time supervised approach is better than the proposed approach. (2) Supervised approach uses LoRA instead of full model training. 
2. The experiments are not solid. Some of the experiments in the paper use non-standard (potentially easy/small/old/deprecated) datasets. Most notably - CIFAR 10/100 for image classification and COCO-colar and COCO-number for VQA. The results are much less convincing compared to ImageNet for classification and VQA v2 for VQA.

questions:
Is there a reason for picking these datasets for experiments? Is it possible to perform the same experiments on more recent and broadly adopted datasets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699709332

thread_type:
official_review

<thread_object>
creator: reviewer_ufdr

<content>
summary:
This paper presents a joint inference framework for large language and vision-language models, with an optimized and efficient approximation techniques to enable unsupervised adaptation to new tasks through two main methods: unsupervised fine-tuning and unsupervised in-context learning (ICL). This approach makes predictions jointly across multiple inputs, and shows significant improvements over zero-shot inference, sometimes achieving results close to supervised approaches without needing labeled data.

strengths:
1) A big strength of this method is that it provides a framework to effectively allows unsupervised fine-tuning and ICL, reducing the need for labeled examples which is usually expansive in practice.
2) It also demonstrates considerable accuracy and generalization improvements across a range of language and vision tasks, even comparing with supervised fine-tuning methods. The experiments performed on multiple datasets, models, and tasks, demonstrating the framework’s effectiveness in both text and vision-language scenarios.
3) It generally works for various model types such as open-weight models (e.g., Llama-3.1, OpenFlamingo) and close-weight models (e.g., GPT-4), showing flexibility and robustness in application.

weaknesses:
1) The effectiveness of unsupervised ICL seems to rely on the model’s inherent in-context learning abilities. This may restrict its applicability to models lacking strong ICL capabilities. It is unclear how does the same method work for other models with relative weak capability.
2) While the experimental results are promising, a deeper theoretical analysis of the joint inference framework’s optimization behavior and potential limitations or failures would strengthen the contribution.

questions:
Q1: Do you have any evaluations on how the joint inference framework affects model robustness, especially when inputs are noisy or out-of-distribution?

Q2: It is also very interesting to see if there are any common failure cases/scenarios of the proposed method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732272918874

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all the reviewers for their time and provided feedback. We are glad that the reviewers recognize the importance of enabling effective unsupervised adaptation of foundation models (UFdR, UqwZ, TGs1), diversity of our experimental study (UFdR, UqwZ, TGs1), broad applicability (UFdR, Nhh8) and significance of improvements brought by our framework (UFdR, Nhh8, TGs1).

We respond to each reviewer individually. Below, we summarize additional experiments added during the rebuttal and answer a common question about the dependence on ICL capabilities raised by reviewers UFdR and TGs1.

# Additional Results:
We add the following new results for the rebuttal, which will also be included in the manuscript:
- We evaluate our methods on 6 recent challenging benchmarks: MMLU, MMLU-Pro for LLMs, and MMMU, MMMU-Pro, VQAv2, and VizWizz for VLMs (see Reb-Tab. 1-3 below.) We show that our method is applicable to these benchmarks and improves upon zero-shot inference.
- We provide additional results of our methods on 3 small models: Llama-3.2-1B/3B and Phi-1.5. We find that our methods can be applied to a broad range of model sizes and improve upon their zero-shot performance (see additional discussion below.)
- We explore different task encoder parameterizations for the unsupervised FT method. We show that our method can be applied to any parametrization, with LoRA resulting in the best performance.


| Method| MMLU | MMLU-Pro |
|-|-|-|
| Zero-shot| 62.6 | 26.1|
| Unsupervised ICL| 65.4 | 32.5|
| Supervised ICL| 65.4 | 37.2|

__Reb-Tab.1: MMLU and MMLU-pro with Llama-3.1-8B__ We use CoT for MMLU-Pro. 

|Method|MMMU|MMMU-Pro|
|--|-|-|
|Zero-shot|66.4|54.7|
|Unsupervised ICL|68.6|55.5|

__Reb-Tab.2: MMMU and MMMU-Pro with GPT-4o.__ We use CoT. Note that supervised ICL is not possible as no ground truth reasonings is available for these datasets to perform CoT.

|Method| VQAv2 | VizWiz |
|-|-|-|
|Zero-shot| 58.1|41.6|
|Unsupervised ICL| 59.7| 46.7|
|Supervised ICL| 60.6| 55.8|

__Reb-Tab.3: VQAv2 and VizWiz with OpenFlamingo-9B__.
# Reliance on the Model’s ICL capabilities
First, we define ICL capabilities to support the discussion.
__ICL Capabilities definition:__ A model exhibits “ICL capabilities” if its few-shot ICL performance is higher than its 0-shot performance.

Indeed, in order for our joint inference framework (via unsupervised ICL or unsupervised fine-tuning) to improve the model's 0-shot performance, the underlying model should exhibit ICL capabilities in the first place. In these cases, our approach allows one to invoke the model’s ICL capability and improve upon 0-shot without the need for ground truth labels. This is similar to how, for example, a widely adopted CoT method improves zero-shot performance relying on the models’ reasoning capabilities and doesn’t work otherwise [1, 2]. We believe that our proposed methods are of practical value given recent improvements in models’ ICL capabilities, which we briefly discuss below.

Most recent language-only models, indeed, exhibit ICL capabilities [6]. In our experiments, we find that even small models such as Llama-3B or Phi-1.5 can exhibit ICL capabilities improving upon zero-shot in many cases. In these cases, we find that our unsupervised ICL method also improves zero-shot performance (see Tab. 1, Fig. 3, and the table in response to TGs1). Even in cases when a small model doesn’t exhibit ICL capabilities, we show that our unsupervised fine-tuning method can be used to improve this small model using feedback from a larger model with stronger ICL capabilities (see the table in response to TGs1). This makes both our methods readily applicable to a broad range of LLMs.

Enabling ICL capabilities in vision-language models, on the other hand, is still an ongoing research direction (e.g., [4, 5]). We find that open-source models might not exhibit ICL capabilities on more recent benchmarks (e.g., LLava-Interleave on MMMU), an observation also made in [3]. However, for more capable closed-source and/or larger models such as GPT-4o, which do exhibit ICL capabilities, we show that our unsupervised ICL method is already applicable and improves upon 0-shot performance  (see Tab. 3 and Reb-Tab. 2-3). This demonstrates that our method is already applicable to existing VLMs with ICL capabilities and will be readily available for a broader range of models when they acquire this capability.


[1]Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Wei et al. NeurIPS 2022

[2]Large Language Models are Zero-Shot Reasoners. Kojima et al. NeurIPS 2022

[3]The Devil in the Details of Benchmarking Multimodal In-Context Learning. Zong et al., arXiv preprint

[4]MM1: methods, analysis and insights from multimodal LLM pre-training. McKinzie et al., ECCV 2024

[5]MMICL: Empowering vision-language model with multi-modal in-context learning. Zhao et al., ICLR 2024

[6]Language models are few-shot learners. Brown et al., arXiv preprint, 2020
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary:**

This paper proposes an unsupervised joint inference framework for large language models and vision-language models. This approach performs unsupervised adaptation through unsupervised fine-tuning and unsupervised in-context learning, leveraging inter-sample dependencies. Experimental results demonstrate that the proposed method substantially improves performance of language models (LLaMA-3 and GPT-4o) and a vision-language model (OpenFlamingo), compared to zero-shot models and achieves performance comparable to supervised approaches.

**Strengths:**

1. **Underexplored unsupervised adaptation of LLM and VLM.** This paper studies an underexplored topic, unsupervised adaptation of LLM and VLM using unsupervised fine-tuning and in-context learning. The proposed method iteratively improve the models predication at test time.
2. **Strong experimental results.** The proposed method significantly improves the performance compared to zero-shot predictions. 
3. **Generalizability.** The proposed method is effective for both LLM and VLM, including open and close-weight models. The authors demonstrated the effectiveness across a wide range of tasks: text classification, image classification, question answering, visual question answering, natural language inference, common-sense reasoning, and math problem-solving.

**Weaknesses:**

1. **Dependence on in-context learning capability.** Unsupervised ICL heavily relies on the strong ICL capabilities of the pre-trained models. 
2. **Suboptimal task encoder.** More task encoders could be considered rather than LoRA. 
3. **Presentation.** As Reviewer Nhh8 pointed out, the clarity of this paper needs to be improved.

**Main reasons:**

This paper studies an underexplored problem, which is the unsupervised adaptation of pre-trained LLM and VLM. It is an important problem and the proposed method achieves strong performance and it is applicable to a wide range of tasks and models, including open and closed-weights.