Large (Vision) Language Models are Unsupervised In-Context Learners
Authors: Artyom Gadetsky
Andrei Atanov
Yulun Jiang
Zhitong Gao
Ghazal Hosseini Mighan
Amir Zamir
Maria Brbic
Venue: ICLR 2025 Poster (2025)
Keywords: llm
unsupervised
in-context learning
few-shot learning
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce a joint inference framework for large (vision) language models to perform unsupervised adaptation on a given task, resulting in the improved performance upon independent zero-shot predictions.

Abstract: Recent advances in large language and vision-language models have enabled zero-shot inference, allowing models to solve new tasks without task-specific training. Various adaptation techniques such as prompt engineering, In-Context Learning (ICL), and supervised fine-tuning can further enhance the model’s performance on a downstream task, but they require substantial manual effort to construct effective prompts or labeled examples. In this work, we introduce a joint inference framework for fully unsupervised adaptation, eliminating the need for manual prompt engineering and labeled examples. Unlike zero-shot inference, which makes independent predictions, the joint inference makes predictions simultaneously for all inputs in a given task. Since direct joint inference involves computationally expensive optimization, we develop efficient approximation techniques, leading to two unsupervised adaptation methods: unsupervised fine-tuning and unsupervised ICL. We demonstrate the effectiveness of our methods across diverse tasks and models, including language-only Llama-3.1 on natural language processing tasks, reasoning-oriented Qwen2.5-Math on grade school math problems, vision-language OpenFlamingo on vision tasks, and the API-only access GPT-4o model on massive multi-discipline tasks. Our experiments demonstrate substantial improvements over the standard zero-shot approach, including 39% absolute improvement on the challenging GSM8K math reasoning dataset. Remarkably, despite being fully unsupervised, our framework often performs on par with supervised approaches that rely on ground truth labels.

Published as a conference paper at ICLR 2025
LARGE (VISIon) LANGUAGE MODELS ARE UNSUPER- VISED IN-CONTEXT LEARNERS
Artyom Gadetsky
Andrei Atanov
Yulun Jiang'
Zhitong Gao
Ghazal Hosseini Mighan
Amir Zamir
Maria Brbić
Swiss Federal Institute of Technology (EPFL)
ABSTRACT
Recent advances in large language and vision-language models have enabled zero- shot inference, allowing models to solve new tasks without task-specific training. Various adaptation techniques such prompt engineering. In-Context Learning (ICL), and supervised fine-tuning can further enhance the model's performance a downstream task, but they require substantial manual effort construct ef- fective prompts or labeled examples. In this work, we introduce joint infer- ence framework for fully unsupervised adaptation, eliminating the need for man- ual prompt engineering and labeled examples. Unlike zero-shot inference, which makes independent predictions, the joint inference makes predictions simultane- ously for all inputs in given task. Since direct joint inference involves compu- tationally expensive optimization, we develop efficient approximation techniques, leading to two unsupervised adaptation methods: unsupervised fine-tuning and unsupervised ICL. We demonstrate the effectiveness of our methods across di- natural language
tasks and models, including language-onle Llama-3.1 processing tasks, reasoning-orienn Qwen2.5-Math on grade school math prob- lems, lasiuanlanguage OpenFlamingo on vision tasks, and the API-only access GPT-4o model on massive multi-disciphie tasks. Our experiments demonstrate substantial improvements over the standard zero-shot approach, including 39% absolute improvement on the challenging GSM8K math reasoning dataset. Re- markably, despite being fully unsupervised, our framework often performs on par with supervised approaches that rely ground truth labels.
INTRODUCTION
Recent progress in large language and vision-language models, which we collectively refer to as foundation models (FMs), have made possible to adapt them to solve new tasks via zero-shot inference by leveraging their general knowledge obtained during pre-training (Brown et al. 2020). For a given task, zero-shot inference obtains the prediction y for an input sequence by maximizing the probability of the next token, i.e., arg max,, p(y
Various methods have been proposed to enable better task adaptation, with In-Context Learning (ICL.) (Brown et al., 2020) Agarwal et al., 2024 Jiang al.) 2024), fine-tuning (Hu et al.) 2022 Jia et al., 2022). and prompt engineering (Wei et al. 2022b] [Snell et al. 2025) emerging as the most prevalent techniques. While these methods improve upon zero-shot inference, they rely on labeled examples or require manual effort craft effective prompts, which can pose practical limitations.
In this work, we propose joint inference framework that enables fully unsupervised adaptation on new task. Our framework generalizes the standard zero-shot inference to joint inference over N > inputs, resulting in the following optimization problem:
arg max p(y1. UN 21...., IN):
(I)
81.--UM
"Equal contribution.
TThis usually also involves having task-specific textual instruction which omit here for simplicity.

---

Published as a conference paper at ICLR 2025
the joint predictions for the corresponding inputs 21...... ........2 NN. Compared to the zero-shot independent predictions, joint inference can guide the model to make consistent pre- dictions and reason over multiple inputs simultancously (Figure (left)). Performing joint inference requires solving the optimization problem that is intractable for a large number of examples N. To address this, we develop approximation techniques, resulting in two unsupervised adaptation meth- ods: unsupervised fine-tuning and unsupervised ICL.
where V......
UN ................................
CIFAR--100 with OpenFlemingo-
Zero-shot independent inference:
Joint inference:
0,750
0.725
0.703
0.675
Joint Inference vie
FM
FM
FM
Unsupervised Fire-Turing
0,625
-0- Unsugervaed
0.800
Zero-shot Inference
y/ V/N
y/1 YN
0.575
2
2
2'
Figure I: Joint inference framework for foundation models. Left: Unlike the standard zero-shot inference that makes a prediction y independently for each input v, the joint inference makes pre- dictions for multiple inputs at the same time, leveraging dependencies between all examples. Right: We develop two methods to perform the joint inference that achieve substantial improvements over traditional zero-shot inference: unsupervised mne-tuning and unsupervised ICL. Their performance increases as the number of examples N for the joint inference increases, showing the effectiveness of the proposed joint inference framework.
Unsupervised fine-tuning is principled method for fine-tuning FM to optimize its own joint pre- dictive probability [.. The key principle lies in the self-improvement mechanism, where the FM is enhanced based its own feedback. While this method yields strong performance, fine-tuning requires access to model weights and output probabilities, which are unavailable for close-weight models such as GPT-4 (Achiam et al. 2023). To enable the broad applicability of our joint infer- ence framework across all model types, we introduce unsupervised ICL, that relies only on access to next-token generation and uses the few-shot in- in-context prompting iteratively refine the predic- tions. Unlike supervised ICL, which uses ground truth labels for in- in-context examples to enhance the predictions, unsupervised ICL refines the predictions using the model's 's own outputs from previous iterations of refinement. We show that unsupervised ICI., in fact, implicitly maximizes the joint probability (Eq. [ and can be seen as an approximate joint inference under the same framework.
We evaluate the proposed methods across diverse set of tasks, including text and image classifi- cation, natural language inference, (visual) guestion-answering and grade-school math problems. Our evaluation spans both language-only and -lnguage FMs, including open-weigh Llama- 3.1 (Dubey et al.) 2024). Qwen2.5-Maid (Yang et al. (024) and OpenFlamingo (Awadalla et al.] 2023) models, and wlogh- GPT-4o via the corresponding API. Our results show that both pro- posed methods significantly outperform zero-shot inference and often approach the performance of fully supervised counterparts, despite not using any labeled examples. For instance, applying unsu- pervised ICL to Qwen2.5-Man yields remarkable 39% absolute improvement over zero-shot infer- ence on the GSM8K math reasoning dataset, closely matching its supervised counterpart. Similarly, unsupervised fine-tuning applied to Llama-3. 1-8B, results in substantial 23% absolute improve- ment over zero-shot inference on average over 13 natural language processing tasks and matching the performance of the supervised fine-tuning on 6 out of 13 tasks.
2 RELATED WORK
Adapting FMs via fine-tuning. Pre-training generalist foundation models followed by task- specific fine-tun mone-tunnng was shown be an effective approach to solving different language and vision tasks (Raffel et al. 2020
Radford et al., Beyer et al., [024 Chen et al. 2023a] McKinzic et al.) Dubey et al. 2024). The first pre-training stage usually involves optimizing an unsupervised
2024
2

---

Published as a conference paper at ICLR 2025
objective, e.g., next-token prediction for language or contrastive loss for vision, lcrge-scale
dataset (Raffel et al.
2020; Cherti et al. 2023] Radford et al. 2021 Kaplan et al. 2020; Schuh-
mann et al. 2022). The second stage involves either full-weight training parameter-effittnt
fine-tunning (Hu et al. 2022 Yosinski et al. 2014 Jia et al.] 2022 Chen et al.] 2023b Houlsby et al. 2019 Pfeitfer al. 2 Similar to the second stage, our unsupervised fine-tuning method updates the weights of a pre-trained FM to adapt to specific task. However, unlike other fine-tuning methods, our approach is based on a selffimprovemmnt mechanism and does not require labeled ex- amples
Adapting FMs via prompting. Prompting based approaches emerged as an alternative optimization-free way to adapt an FM to new task (Wei et al. 2022a; Radford et al. [019 Brown al. 2020, Alayrac al. 2022). A standard zero-shot inference provides input and task descrip- tion as a context for a model and generates the answer via next-token prediction. A large line of
works develop methods to improve this zero-shot inference by, e.g., prompting a model to generate additional "reasoning" steps (Wei et al.) 2022b Yao et al.) 2023; Snell et al. 2025) providing few labeled examples as a context (Brown et al., 2020). Similarly, our unsupervised ICL method improves upon the zero-shot inference by using few self- seifrageeertedd examples as a context that are labeled by the model itself, thus without requiring any labeled examples.
Reinforcement learning for FMs. This line of work uses reinforcement learning algorithms (Sut- ton & Barto 2018 Schulman et al.
2017) to fine-tune FMs to optimize a non-differentiable reward function. These reward functions
either based on human feedback (Ouyang et al. [202] Chris- tiano et al. 2017)) a metric (Pinto et al. 2023) or the output the same or another FM (Zheng et al,, 2023 Bai et al.] 2022 Lee et al. 2024). Related this, here we use the model's own feedback based on the joint probability (Eq.
fine-tune its weights via a reinforcement learning algorithm.
Probabilistic Inference in FMs. Recently, there has been significant interest in adapting general probabilistic inference techniques to perform inference in a probabilistic models defined by a foun- dation model. For example, Zhhao et al. (2024) build upon Sequential Monte-Carlo (Doucet et al. 2013) to sample from an unnormalized target distribution defined by foundation model. Another line of works (Hu et al. [024]] Yu et al.) 2024) employ GFlowNets framework (Bengio et al.) 2023) to solve the probabilistic inference problems. While these general probabilistic inference techniques could be possibly extended to perform the proposed joint inference, we develop the principled unsu- pervised fine-tuning approach that effectively leverages the structure of our optimization problem.
3 BACKGROUND
3.1 FOUNDATION MODELS PRE-TRAINING
In this work, we study the class of foundation models that are pre-trained on huge amount of data to model probabilities of a next token given the preceding ones, also known as the next-token prediction objective. In particular, given maximal context length L of a foundation model, models probabilities of token sequences as follows:
L.
PFM (T) = II pPM((
(2)
where 1, E and V the model's vocabulary. Such pre-training has shown remarkable scaling laws (Kallan et al., 2020), resulting in the predictable gains that can be delivered by increasing model size, the amount of available training data or compute budget. Furthermore, separately trained vision adapter can be integrated in such models to enable performing multimodal tasks (Alayrac et al. 2022). It allows a foundation model to ingest a multimodal sequence containing images and/or videos interleaved with text and produce text.
3.2 ADAPTING FOUNDATION MODELS TO DOWNSTREAM TASKS
Given a pre-trained foundation model Prm. different approaches can be used to improve the model's performance given task.
Supervised fine-tuning, Supervised fine-tuning is the prevalent approach to improve model per- formance on downstream task. Specifically, given labeled examples D model is trained to
3

---

Published as a conference paper at ICLR 2025
maximize the probability of the correct outputs, i.e., cross-entropy E(x, (xx.we))DDrr log PrM (Ver where yGT denotes the ground truth (GT) output sequence for input Although supervised fine- tuning is the best performing, it requires access to labeled data and model weights.
Zero-shot Inference and In-Context Learning (ICL). Brown et al. (2020) have shown that large-scale pre-training via next-token prediction enables zero-shot inference. Specifically, with- out any additional training, foundation model can be prompted with an input instance of task x and the task description C to generate the corresponding solution via next-token prediction arg max, PrM(yla, (z, C*). was also demonstrated that the model's 'ss performance is susceptible the chosen prompts, giving rise to manual prompt engineering to produce more accurate solutions (Liu al. 2023).
Another way to improve the predictions is In-Context Learning (ICL), where a model is provided with a set of input instances and their corresponding ground truth answers. Subsequently, the model computes arg max,, (yM( {Ir,, " ] where (I,,. un denote ground truth in-context ex- amples and N denotes the number of in-context examples. Although such approach has proven itself effective, it requires having access to the set of labeled examples, thus, sharing the limitations
of conventional supervised learning setting.
Mhain-of-Thoughh (CoT). Kojima al. (2022) have recently proposed the off-the-shelf prompt- ing technique that surprisingly improves the performance a model. In particular, a model is prompted with Ccor "Think step by step" phrase, that, in turn, triggers it to generate a problem solving reasoning - 71..... I'm ~ PPM z). Subsequently, conditioning on such reasoning chain results in more accurate solutions arg max, PrM PrM(u)r) (y/r), ... T'my CCOT, 2). The authors have ,....
also demonstrated that such approach brings improvements upon both supervised ICL, and zero-shot inference.
THE JOINT INFERENCE FRAMEWORK
In this section, we first formally introduce the problem setting and then present a general form of the joint inference framework.
Definitions and Problem Setting. We refer task T X + y a mapping from the space of input instances X' to the set of plausible answers y For example, for question answering, the elements of X' and y correspond to questions and the corresponding plausible answers to these questions, respectively. Another example can be the sentiment classification task, where X' are sentences, and the set of plausible answers is as simple as y {Positive, Negative). We assume that are given set of input instances D { F'm E X' perform task these instances with a foundation model (+)()).
The question that aim to answer in our work is what is a principled approach to improve the pre- dictions of (.M(.) on a given task 7 in an unsupervised way, i.e., without having demonstrations of input instances x with their corresponding correct answers y? To simplify the narration, we consider close-ended tasks with K plausible answers, i.e., y = (,1,..., ). with each E ) comprising a single token. We discuss the general case of open-ended tasks Section and Appendix
4.1 GENERAL FORMULATION FOR THE JOINT INFERENCE
Here, we propose to perform joint inference to produce answers for set of instances D. In particu- lar, we define the joint likelihood of yı...., MM autoregressively given set of instances D and aim to optimize the following objective:
arg max log p( UM (1, ... IMr), where U1 V NMM EYIM
(3)
p(yi...., VM I1, .1...., ... TMM
Prs J.m, {1z (()..y))ee Vi)
4

---

where N limits the number of instances be processed in single model pass. Besides the fact that such formulation makes it possible to efficiently estimate the objective via Monte Carlo sampling, also incorporates the important inductive bias. Indeed, Eq. imposes particular order when processing the sequence (Z1, MI.............................. MM BMr) with (-M However, ground truth answers should depend on the particular order, and the expectation over different sequences 21..... ....,.... ,,.,,,,,, JN allows to effectively embed this constraint into the objective. Note that our objective is strict generalization of the standard zero-shot inference since J reduces to it. Furthermore, Figure and the results in Published as a conference paper at ICLR 2025
Given that foundation models have limited context length and processing the entire D might be infeasible, we consider the limited number of instances a single model pass as follows:
arg max J (U1...... .MM. where
(4)
Mir MM EYa
N
(VI,.... yM)
del
log (ym(ynlen {(ris 21 Jien);
Appendix [ demonstrate that increasing N leads to obtaining more accurate answers for the set of instances D compared to the standard zero-shot inference.
Eq. poses computationally expensive combinatorial optimization problem. In the subsequent sections, we introduce two methods to optimize the proposed joint inference objective, namely, unsupervised fine-tuning and unsupervised ICL.
4.2 UNSUPERVISED FINE-TUNING AS PRINCIPLED APPROACH
Although the objective J. admits efficient Monte Carlo estimation, optimizing it requires K M model calls which infeasible in practice. To address this challenge, we resort the following amortization:
max D1 UM MMEY
J (B1..... 3M) max (o. (y1, .., yas),
(5)
where we refer to a T0(-)2)))) as a task encoder which defines distribution over y parametrized by continuous parameters 0. As result, instead of solving the difficult combinatorial optimization problem, we can apply efficient stochastic optimization techniques to learn the parameters of the task encoder. In principle, having flexible enough To would result in the strict equality in Eq. After the optimization is done, arg max To (y) (.)) provides us with answers yn to the corresponding input yEy
instance n, independently from all other input instances Tigho allowing for the efficient inference.
Efficient optimization. Enabling efficient optimization requires obtaining an unbiased stochastic gradient estimator of the objective in Eq. 5 However, the objective J involves evaluating PFM on discrete tokens yo predicted by the task encoder To. rendering this process non non-differentiable. While relaxation is possible (Atanov al. 2022 Gadetsky & Brbic 2023 Gadetsky et al.) 22224), more prevalent approach such scenarios is using the REINFORCE gradient estimator (Williams) 1992). Despite its generality, naive implementation of REINFORCE suffers from the high variance when used for the optimization over combinatorial spaces (Gadetsky et al. 2020 Paulus et al.) 2020 Struminsky et al. 2021). To address this challenge, we develop an effective stochastic gradient timator that leverages the structure of our objective to substantially improve the convergence speed. We provide the complete derivation of this estimator and compare it REINFORCE Appendix
B.I
Task encoder parametrization. We employ a foundation model itself to serve as our task encoder To To(.)..). In particular, we constrain Prm to model a distribution over y as follows:
P (r. E
To (v) (x)) ΣSEy Pfan (r)r)))
(6)
where [-] denotes Iverson bracket and PfM denotes the same foundation model parametrized by LoRA (Hu et al.) 2022) with the corresponding trainable parameters 6. The LoRA parameters set such that, the beginning of training. PPM corresponds to the zero-shot predictions of PrM. providing good initialization for our REINFORCE- optimization, which is known to lead to
faster convergence (Greensmith et al. 2004). Noteworthy, this parametrization, coupled with
5

---

Published as a conference paper at ICLR 2025
unsupervised objective, can be seen as instantiation of self-training, in which model improves by obtaining feedback from itself.
In principle. unsupervised fine-tuning can be combined with any fine-tuning strategy. We pro- vide the ablation of the task encoder's 's design and its effect on the performance the unsupervised fine-tuning approach in Appendix D.2 We found that the LoRA fine-tuning provides the practical trade-off between parameter efficiency and the performance.
Regularization. Optimizing Eq. (5) can lead to degenerate solutions, i.e., converging to single answer for all the input instances, which is common in unsupervised learning (Gansbeke et al. 2020
Gadetsky & Brbie 2023 Gadetsky et al., [0224] Greic et al. 2024 |Atanov et al. This happens
because PFM assigns high probabilities to single answer after observing the same answer for all input instances in its context. We regularize our task encoder т to avoid such trivial solutions. Let
priir (v) = (y)p))(y((r), then the regularization term is RR(T)) - -Eyey" To
price (w) logtz" prior (v).
TO
Putting all together, the final optimization objective to train To is as follows:
max E (1) J (y1...., yMM) + R((T))
(7)
where we found = 10 is a good default choice for the regularization strength. We refer to this principled approach as the joint inference via unsupervised fin-tuning (Figure 2}. The pseudocode and the implementation details are provided in Appendix B.1
Step
Step
Step 3 Maximize joint probability the generated answers
Generate answerrs via encoder Independently question
Feed generated answers Jointly foundation model
Tass
Trng
Triq
Planicl
Parrot
y (et-be Viem Tanch
R
axtrrggroshs
Righ Hil
logprss(uulea Fou (sa2))cc)))
Taxe Encodit
Panst
Repeat until convergence
Taxi
Updato
REINFORCE
y/ipp........................... Mx) log (v(ulle))
Figure 2: Unsupervised fine-tuning a principled optimization method to perform joint inference, enabling wnsupervised adaptation new task. Given dataset of questions, each iteration of the optimization involves generating answers via task encoder independently for a batch of questions (Step /). Subsequently, these answers fed into foundation model to estimate the joint probabil- ity, providing the quantitative measure of the quality of the answers (Step 2). Finally, task encoder updated to maximize the joint probability (Step 3). These steps are repeated until convergence, yielding the task encoder adapted on a given task without any supervision.
4.3 UNSUPERVISED IN-CONTEXT LEARNING
Although amortization offers a principled approach to optimize the objective in Eq. 44 it requires access to model weights define a task encoder To and output probabilities PrM- This makes unsupervised fine-tuning suitable to open-weight models, but limits its applicability to most close- weight models, such as GPT-4 (Achiam et al. [003). To make the joint inference framework broadly applicable, our key insight is that each summand in Eq. 44p can be seen as ICL predictions:
arg max log (1 lxn. (12 y:) hien)
(8)
Unlike conventional supervised ICL, which relies on ground truth answers, method also opti- mizes answers.
We employ this insight to develop the unsupervised ICL method to optimize Eq. 44 in the multi-tum fashion (Figure3 Specifically, we iteratively refine answers for the set of instances D via con- ditioning the answers from the previous round, where the beginning they are initialized by the

---

Published as a conference paper at ICLR 2025
zero-shot predictions. In particular, for given D, let y PrM(-1z) be the answers the 0-th round. Then, for every consecutive refinement round t, we update the answers for x' E D by sam- pling y Pru(-12, {Iz,, V.. V )))=)) where 21...., EN ~ D. In such way, unsupervised ICL self-improves answers through the number of iteration steps. is important to note that this method is readily applicable to all existing foundation models, since it only requires obtaining samples from a model. Figure 3 (right) highlights that unsupervised ICL indeed optimizes the joint inference objective (Eq. 4 We also study the effect of increasing number of turns in Appendix D.3 We
provide the complete algorithm in Appendix B.2
Furthermore, one can observe that our unsupervised ICL resembles the well-known Gibbs sam- pling (Geman & Geman 1984) for sampling from untractable joint distribution by iterative sam- pling from tractable conditionals. Recently, similar ideas were also applied to refine CoT chains given questions and ground truth answers (Xu et al., (024). Our unsupervised ICL pushes this idea further and, when coupled with CoT, can be seen as refining both reasoning chains and answers (Section 5.1
Depodia-14 Liama-
Unsuperised Conttess Geamting
-0.6
Trog
FM
FM
-0.0
FM
+1.0
FM
Tuee
-1.2-
Join Irierrce déjective
Accuroty
Zero-Poe
4TAm onGuupee Sod KLL
Figure 3: Unsupervised In-Context Learning is broadly applicable method to perform joint in- ference for any task and any existing foundation model. Left: Our method generates answers for each question independently using shot prompting. Subsequently, it enters the multi-turn stage, where, at each turn, for each question, the model is prompted with randomly sampled in-context ex- amples from the dataset (excluding the considered question) with the corresponding answers from the previous turn. These examples are fed into the model in the left-to-right order along with the current question generate a refined answer. Such refinement is repeated for T' turns, yielding the final answers. Right: Both the joint inference objective and the performance improve with more optimization turns of the unsupervised ICL method.
5 EXPERIMENTS
Datasets and evaluation metric. We evaluate the performance of our two methods for joint infer- ence across wide range of tasks, including text classification, image classification, question an- swering, visual question answering, natural language inference, common-sense reasoning, and math problem-solving, A detailed description of each dataset, along with the prompts used, is provided in Appendix C.I We use accuracy as the evaluation metric for all the experiments.
Foundation models. We utilize three open-source foundation models to evaluate framework, namely, Llama-3.1 (Dubey et al. (2024) for text-based experiments, OpenFlamingo (Awadalla et al. Alayrac al., 2022) for lision-language experiments and Qwen2.5-Math-77 (Yang et al., for reasoning experiments. Specifically, Llama-3. 1-BBB is used as the default model for our main text experiments, instruction-tuned version and the larger 70B instruction-tune model are used for the ablations. For vision experiments, we use OpenFlamingo as the default model. Furthermore, employ GPT-4o (Achiam et al. 2023) to serve us a close-weight foundation model in our experiments.
2023
2024
Baselines. We incorporate the following baselines and upper bounds for our evaluations. (1) Zero- shot inference makes the predictions independently for each input example without task-specific fine-tuning or demonstrations. (2) Zero-shot with Thought (CoT) incorporates CoT rea- soning prompts to generate intermediate reasoning steps before the final answer (Kojima et al.) 2022 Wei et al. 2022b). We only use it for our language experiments, as we found that CoT does not show any benefit for the OpenFlamingo model, often significantly degrading the per- formance. (3) Supervised In-Context Learning (ICL) uses labeled training examples to provide them demonstrations to the model. Consequently, this serves an upper bound to our un- supervised ICL method, which does not use any labeled data. Similarly. (4) Fully-Superviae Fine-tuning (FT) employs LoRA (Hu et al. 2022) supervised fine-tuning using all labeled train-
7

---

Published as a conference paper at ICLR 2025
ing examples and serves upper bound our unsupervised fine-tuning method. We refer the reader to Appendix C.2 for the additional implementation details. Code is publicly available at https:// http::///tthhb... www.ghtw..../wwooooeoeeeee.eeeee
5.1 RESULTS ON NATURAL LANGUAGE PROCESSING TASKS
To study the performance of the joint inference framework on language tasks, we evaluate our meth- ods 13 benchmark datasets, spanning various NLP tasks. Our results highlight the effectiveness of our joint inference framework (Table D First, the results show that unsupervised fine-tuning substantially outperforms the standard zero-shot inference. In particular, it brings 23% absolute improvement on average over 13 considered datasets, with remarkable 52.5% 30.6%. 26.3% and 19.5% on the SUBJ, Amazon, DBPedia and HellaSwag datasets, respectively. Furthermore, it of- ten approaches the performance of its fully supervised counterpart, closely matching it on 6 out of 13 considered datasets. Secondly, unsupervised ICL also exhibits remarkable performance gains compared to the zero-shot inference, bringing 19.2% absolute improvement on average over 13 considered datasets. Remarkably, it is on par with the supervised ICL on 10 out of 13 considered datasets, overall demonstrating the effectiveness of the proposed joint inference framework.
Table 1: Results of the unsupervised fine-tuning and unsupervised in-context learning meth- ods on NLP tasks. For each dataset, we show the accuracy (in %) of the zero-shot inference, the proposed unsupervised fine-tuning (FT), and ICL methods, and their corresponding supervised counterparts, which represent the upper bound. We use the Llama-3.1- 1--8BB model in all cases. Both proposed unsupervised adaptation methods outperform zero-shot inference and approach the perfor- mance of the corresponding supervised methods in most cases.
Text Classfication
Language Inference
Question Answering
Adaatation Method
SST2
Amazon
AGNews
TREC
DBPedia
SUBJ
RTE
QNLI
MNLI
COPA
BoolQ
PIQA
HellaSwag
shot
77.7
66.5
62.7
55.5
81.0
59.0
46.0
Zero shot CoT
78.8
76.1
58.3
28.7
63.1
54.1
55.6
52.1
47.5
69.0
64.4
58.2
34.6
57.0
Fine-tumming (via LoRA):
Unsupervised
92.3
78.2
65.5
Fully- Supervised
92.1
96.0
90.4
93.7
98.8
96.3
89.0
89.5
84.7
85.7
85.6
82.1
87.1
90.1
In Context Learning tno weight updates):
Unsupervied ICL
92.4
96.6
86.2
59.0
97.9
74.2
79.8
67.4
93.5
82.6
78.4
58.2
Supervised ICL.
93.3
96.6
88.0
97.6
74.5
92.3
79.1
59.1
Mathematical reasoning and multitask language understanding. Furthermore, we evaluate unsupervised ICL on the GSM8K dataset that requires reasoning capabilities that covers broad knowledge of different disciplines.
and on the MMLU(-Pro) dataset We employ Chain-of- for our method and all the baselines on the GSM8K and MMLU-Pro datasets. Consequently, unsupervised ICL re- fines both reasoning chains and the answers in a fully unsupervised man- ner. Our results indicate that method is also applicable to these challenging benchmarks (Table 2 For instance, it brings remarkable 39.2% absolute improvement over the zero-shot baseline on the GSM8K dataset, also outperforming the super- vised counterpart.
Table 2: Results of the unsupervised ICL on the mathe- matical reasoning and multiple choice question answer- ing. Our unsupervised ICL method improves the perfor- mance of Llama-3.1-88 and Qwen2.5-Math. on chal- lenging math reasoning and multiple choice question an- swering.
Adaptation Method
GSM8K
MMLU
MMLU-Pro
Zero-shot Unsupervised
42.5
65.0
23.7
1.lama-3.1-8
33,1
Supervised ICL.
55.7
66.7
37.8
Zero-shot
52.2
61.0
37.3
Qwen2.5-Mat-78
Unsupervised ICL.
91.4
62.6
36.4
Supervised
Unsupervised ICL scales effectively at test-time. Our unsupervised ICL. method consists of two stages: (1) task adaptation stage, where we iteratively generate labels for unlabeled examples, creating a labeled support set; and (2) a test stage, where we perform ICL inference on new test examples using the labeled support set from stage one. Since the adaptation stage is performed once per task, test-time compute scaling depends on the number of (unsupervised) ICL examples used in the second test stage. Figure shows how performance improves as scale test-time compute by increasing the number of ICL examples. We find that test-time scaling of the 8B model with our method achieves a better corpute-perffrmmacc trade-off than zero-shot inference with the larger
8

---

Published as a conference paper at ICLR 2025
70B model. Further scaling of the 70B model leads to additional gains, outperforming CoT in both compute efficiency and performance.
Unsupervised ICL improves non-instruction-tumeet models. Figure shows the impact of in- struction tuning on the performance of different inference methods on the RTE dataset. We find that unsupervised ICL and FT methods with the base model perform better than both zero-shot and CoT with the instruction tuned model, suggesting less reliance on supervised finettning data. We provide further positive results of scaling 70B model in Appendix D.4
Zero-s1x
Unsupervssed (ouss) Unsuservsed (Dun))
61-3
CAT
65.0
62.5
600
Zoro Shot vitronec
775
Liena-d
Liana
750
725
40 82 Model
12th
BE-FTRTTUE
Teak-Time Onfenenes) FLDPs
Figure 4: Unsupervised ICL scales efffec- tively at test-time. We report performance on the RTE dataset with Llama-3.1 models. We scale test-time compute by using more (unsu- pervised) ICL examples and show provides better compute- prrpuuteperformacce trade-off than zero-shot inference with a bigger model.
Figure Unsupervised ICL and FT improve non-instruction-tunee models. We show the performance of different inference methods on the RTE dataset for base and instruction cunded Llama-8B models. Both our methods applied the base model outperform zero-shot infer- ence with the instruction tund model.
5.2 RESULTS FOR IMAGE CLASSIFICATION AND VISUAL QUESTION ANSWERING TASKS
To study the performance of the joint inference framework on tasks that require visual comprehen- sion, we evaluate methods seven vision datasets, spanning both image classification tasks (CIFAR-10, CIFAR-100, Food101) and visual question-answering tasks (COCO-Color, COCO- Number, VQAv2 and VizWiz). Our results demonstrate that both unsupervised fine-tuning and unsupervised ICL consistently outperform the standard zero-shot inference (Table 3.. In particular, unsupervised fine-tuning brings substantial absolute improvements of 14% average over the ap- plicable datasets with the remarkable gains 23% on the Food101 dataset, which is the challenging fine-grained image classification task for rision-language foundation model. Furthermore, reflect- ing our language experiments, unsupervised ICL closely matches the performance of its supervised counterpart out of 7 considered datasets, overall demonstrating the applicability of our joint inference framework to vision-language foundation models.
Table Results for image classification and VQA tasks. For each dataset, we report the accuracy (in %) of zero-shot inference, the proposed unsupervised fine-tuning and unsupervised ICL, and their corresponding supervised counterparts. We use OpenFlamingo-444 in all cases except VQAv2 and VizWiz, where we use OpenFlamingo-991 Note that unsupervised fine-tuning is not applicable to VQAv2 and VizWiz given that these datasets comprise open-ended questions. Both unsuper- vised fine-tuning and unsupervised ICL methods consistently outperform zero-shot inference and approach the performance of the corresponding supervised methods in most cases.
Image Classificatinn
Visual Question Answering
Adaptation Method
CIFARI0
CIFAR100
Food101
COCo-Color
COCo-Number
YQAv2
VizWiz
Zero-sbot
87.2
58.0
58.4
55.8
25.6
58.1
41.6
Fine-tunning (via LoRA):
Unsupervised
96.0
74.1
81.0
42.3
N/A
Pully-Supervised
84.9
91.5
85.4
N/A
N/A
In-Context Leamingg weight updates);
U Unsupervised ICL
92.6
69.0
61.8
57.5
36.8
59.7
46.7
Supervised ICL.
93.0
69.1
61.7
58.2
47.1
60.6
55.8

---

Published as a conference paper at ICLR 2025
Clesed-weight GPT-4o results. To demon- strate the applicability of the joint inference framework to wlighd-weight models, we em- ploy GPT-4o and study the performance of unsupervised ICL MMMU and MMMU-Pro datasets. For Ima- geNet, we construct support set containing 1000 images corresponding to 100 classes and we sample 5000 images for the evaluation pur- poses only. Specifically, to assess the general- ization, we refine the support set with our unsu- pervised ICL, for two rounds, and, then, exam- ine the performance on the evaluation set conditioned on the refined support set. For MMMU and MMMU-PMe we directly run methods on the validation split, since no data are available to construct support sets. Consequently, supervised ICL is not available for MMMU and MMMU-Pro. As be- fore, compare unsupervised ICL to the zero inference, and the suppervised ICL on ImageNet, employing ground truth labels for the support set. Table illustrates that unsupervised ICL brings an improvement of 3%, 2% and 1% compared zero-shot inference with Chain-of- Thought prompting on the ImageNet, MMMU and MMMU- -Pro datasets, respectively. Furthermore, unsupervised ICL approaches supervised ICL on the ImageNet dataset, overall demonstrating that our joint inference framework is also applicable to dossd-wwiigt models.
Table 4: Our unsupervised ICL method im- proves the performance of closed- GPT- 4o on challenging image classification and mul- tiple choice question answering. Note that pervised ICL is not possible as no support set is available for MMMU and MMMU-Pro datasets to perform CoT prompting.
subset of ImageNet,
ImageNet-100
MMMU
MMMU-Pro
Zero-shot CoT Unsupervised ICL.
76.1
66.4
54.7
79.0
68.6
55.5
Supervised ICL
79.5
N/A
N/A
6 CONCLUSION AND LIMITATIONS
In our work, we proposed the joint ffaaeenceeraamworr the standard independent zero-shot inference given task. The key idea behind our framework is to simultaneously make predictions for multiple input instances of a task. To perform such joint inference, which involves infeasible optimization, we develop two approximations resulting in two efficient unsupervised methods: unsupervised fine-tuming and unsupervised ICL. We show their effectiveness on range of datasets and tasks using large language and vision-language models. Below, we discuss the framework, both methods and their corresponding limitations.
framework that brings substantial improvements over
Reliance of the joint inference framework on ICL capabilities. order for our joint inference framework to improve the model's 's zero-shot performance, the underlying model should exhibit ICI. capabilities in the first place, i.e., the performance of model with supervised ICI. should be higher than its zero-shot performance. In such case, our framework allows one to invoke the model's ICL. capability and improve upon the zero-shot without the need for ground truth labels. We study the ef- fect of weak ICL capabilities on the performance of the joint inference framework in Appendix D.5
Unsupervised fine-tuning. Unsupervised fine-tuning is a principled method to optimize the pro- posed joint inference objective. Remarkably, although being unsupervised, it often approaches its supervised upper bound, which uses labeled examples for fine-tuning. This method has two main limitations. First, in its current form, it is limited to close-ended tasks with finite of plausible answers y. This stems from the fact that we need to constrain the output of the task encoder to ), which greatly benefits the optimization. One potential solution this could be using more advanced amortization optimization techniques such as (Hu et al. 2024 Zhao et al. (2024). Second, unsuper- vised fine-tuning not applicable to closed-weight proprietary models (Achiam al.) 2023) Anil
et al. 2023) since it requires access to model weights and output probabilities. We address these limitations with our unsupervised ICL method.
Unsupervised ICL. Unsupervised ICL offers simple yet powerful approximation to perform the joint inference compatible with any task and model. requires only obtaining samples from a model conditioned on the provided input, that is readily available for all the existing foundation models. Moreover, it can be easily coupled with modern prompting techniques such as Chain-of- Thought to further improve the performance in an unsupervised manner. Despite the flexibility of unsupervised ICL, similarly to how supervised ICL lags behind supervised fine-tuning unsupervised ICL generally falls short compared to unsupervised fine-tuning. This limitation could be addressed by the improved capabilities of newly released foundation models with larger context (Agarwal et al. 2024 Jiang et al. 2024).
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Mingqiao Ye for helping us with GPT-4o experiments. We gratefully acknowledge the support of the Swiss National Science Foundation (SNSF) grant IC0010-231922 and the SNSF start- ing grant TMSG12 2MS22,,226252,, Zeiss and CIFAR MHU Catalyst. This work was supported as part of the Swiss AI Initiative by a grant from the Swiss National Supercomputing Centre (CSCS) under project ID a08 on Alps. This material is based on work that partially funded by an unrestricted gift from Google.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, et al. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 2023.
Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Luis Rosias, et al. Many-Shot In- Context Learning. In Advances in Neural Information Processing Systems, 2024.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, et al. Flamingo: A Visual Language Model for Few-Shot Learning. In Advances in Neural Information Processing Systems, 2022.
Rohan Anil, Sehastian Borgeaud, Yonghui Wu, Jean-Batisste Alayrac, Jiahui Yu, et al. Gemini: A Family of Highly Capable Multimodal Models. arXiv preprint arXiv:2312. /805, 2023.
Andrei Atanov, Andrei Filatov, Teresa Yeo, Ajay Sohmshetty, and Amir Zamir. Task Discovery:
Finding the Tasks that Neural Networks Generalize on. Technical Report arXiv:2212.0026, arXiv, November 2022. [cs] type: article.
00261 arXiv:2212.00261
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, et al. OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models. arXiv preprint arXiv:2308.013390 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, et al. Constitu- tional Al: Harmlessness from Al Feedback. Anthropic Technical Report, 2022.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward Hu, Mo Tiwari, et al. GFlowNet Founda- tions. Journal of Machine Learning Research, 2023.
Lucas Beyer, Andreas Steiner, André Susano Pinto. Alexander Kolesnikov, Xiao Wang, et al. PaliGemma: A Versatile 3B VLM for Transfer. arXiv preprint arXiv:2407.07726, 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and Yejin Choi. PIQA: Reasoning about
Physical Commonsense in Natural Language. In AAAI Conference Artificial Intelligence, 2020.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101- Mining Discriminative Com-
ponents with Random Forests. In European Conference on Computer Vision, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, et al. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, 2020.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, et al. PaLI: A Jointly- Scaled Multilingual Language-Iman Model. In International Conference on Learning Represen- tations. 2023a.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, al. Vision Transformer Adapter for Dense Predictions. In International Conference on Learning Representations, 2023b.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, et al. Re- producible Scaling Laws for Contrastive Language-Image Learning. In Conference on Computer Vision and Pattern Recognition, 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, et al. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems, 2017.

---

Published as a conference paper at ICLR 2025
Christopher Clark, Kenton Lee, Ming- Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, et al. BoolQ: Exploring the surprising difficulty natural yes/no questions. In North American Chapter of the Association for Computational Linguistics, 2019.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, et al. Training Veri- fiers Solve Math Word Problems. arXiv preprint arXiv:2 14168, 2021.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and Memory-Ffficieet Exact Attention with 10-Awareness. In Advances in Neural Information Pro- cessing Systems, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, al. ImageNet: A Large-Scale Hierarchical Image Database. In Conference on Computer Vision and Pattern Recognition, 2009,
Arnaud Doucet, Nando Freitas, Kevin Murphy, and Stuart Russell. Sequential Monte Carlo Methods in Practice. Springer, 2013.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, et al. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783. 2024.
Artyom Gadetsky and Maria Brbic. The Pursuit of Human Labeling: A New Perspective on Unsu- pervised Learning. In Advances in Neural Information Processing Systems, 2023.
Artyom Gadetsky, Kirill Struminsky, Christopher Robinson, Novi Quadrianto, and Dmitry Vetrov. Low-Variance Black-Box Gradient Estimates for the Plackett-Luce Distribution. In AAAI Con- ference on Artificial Intelligence, 2020.
Artyom Gadetsky, Yulun Jiang, and Maria Brbic. Let Go of Your Labels with Unsupervised Transfer. In International Conference on Machine Learning, 2024.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and Luc Van Gool. SCAN: Learning to Classify Images without Labels. In European Conference on Computer Vision, 2020.
Stuart Geman and Donald Geman. Stochastic Relaxation, Gibbs distributions, and the Bayesian Restoration of Images. IEEE Transactions Pattern Analysis and Machine Intelligence, 1984.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. In International Journal on Computer Vision, 2018.
Matej Greie, Artyom Gadetsky, and Maria Brbic. Fine-grained Classes and How to Find Them. In International Conference on Machine Learning, 2024.
Evan Greensmith, Peter Bartlett, and Jonathan Baxter. Variance Reduction Techniques for Gra- dient Estimates in Reinforcement Learning. Journal of Machine Learning Research, 2004.
Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chin Lin, et al. Vizwiz Grand Challenge: Answering Visual Questions from Blind People. In Conference on Computer Vision and Pattern Recognition, 2018
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, et al. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations, 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, et al. Parameter-efficien Transfer Learning for NLP. In International Conference on Machine Learn- ing, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, et al. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations, 2022.
12

---

Published as a conference paper at ICLR 2025
Edward J Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, al. Amortiz- ing Intractable Inference in Large Language Models. In International Conference on Learning Representations, 2024.
Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised Prompt Learning for Vision-Language Models. arXiv preprint arXiv:2204. 03649, 2022.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual Prompt Tuning. In European Conference on Computer Vision, 2022.
Yixing Jiang, Jeremy Irvin, Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H. Chen, et al. Many-Shot In-Context Learning in Multimodal Foundation Models. arXiv preprint arXiv:2405.09798, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, et al. Scaling Laws for Neural Language Models. arXiv preprint arXiv:2001.0836/ 2020.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners, In Advances in Neural Information Processing Sys- tems, 2022.
Alex Krizhevsky, Geoffrey Hinton, et Learning Multiple Layers of Features from Tiny Images. Technical Repori, University of Toronto, 2009.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. In International Con- ference on Machine Learning, 2024.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, et al. DBpedia A Large-scale, Multilingual Knowledge Base Extracted from Wikipedia. Semantic Web, 2015.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhenghao Jiang, Hiroaki Hayashi, et Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Computing Surveys, 2023.
Julian McAuley and Jure Leskovec. Hidden Factors and Hidden Topics:
Understanding Rating Dimensions with Review Text. In ACM Conference Recommender Systems, 2013.
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, al. MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training. In European Conference on Computer Vision, 2024.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, et al. Human- level Control Through Deep Reinforcement Learning. Nature, 2015.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, et al. Training Lan- guage Models to Follow Instructions with Human Feedback. Advances in Neural Information Processing Systems, 2022.
Bo Pang and Lillian Lee. A Sentimental Education: Sentiment Analysis Using Subjectivity Sum- marization Based on Minimum Cuts. In Association for Computational Linguistics, 2004.
Max B. Paulus, Dami Choi, Daniel Tarlow, Andreas Krause, and Chris J. Maddison. Gradient Esti- mation with Stochastic Softmax Tricks. In Advances in Neural Information Processing Systems, 2020.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- Fusion: Non-Destructive Task Composition for Transfer Learning. European Chapter of the Association for Computational Linguistics, 2021.
André Susano Pinto, Alexander Kolesnikov, Yuge Shi. Lucas Beyer, and Xiaohua Zhai. Tuning Computer Vision Models with Task Rewards. International Conference on Machine Learning, 2023.
13

---

Published as a conference paper at ICLR 2025
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, et al. Language Models Unsupervised Multitask Learners. OpenAl Technical Report, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, et al. Learning Trans- ferable Visual Models From Natural Language Supervision. In International Conference on Ma- chine Learning, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, et al. Exploring the Limits of Transfer Learning with Unified Text-to-Text Transformer. Journal of Machine Learn- ing Research, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text. Conference on Empirical Methods in Natural Language Processing, 2016.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning. In AAAI Spring Symposium Series, 2011.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, et al. LAIÓN-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models. In Advances in Neural Information Processing Systems: Datasets and Benchmarks, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. urXiv preprint arXiv: 2017.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM Test-Time Compute Op- timally can be More Effective than Scaling Model Parameters. In International Conference Learning Representations, 2025.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, et al. Recur- sive Deep Models for Semantic Compositionality over a Sentiment Treebank. In Conference on Empirical Methods in Natural Language Processing, 2013.
Kirill Struminsky, Artyom Gadetsky, Denis Rakitin, Danil Karpushkin, and Dmitry Vetrov. Lever- aging Recursive Gumbel-Max Trick for Approximate Inference in Combinatorial Spaces. In Advances in Neural Information Processing Systems, 2021.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, 2018
Ellen M Voorhees and Dawn M Tice. Building a question answering test collection. In ACM SIGIR Conference on Research and Development Information Retrieval, 2000.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, et Glue: A multi-task benchmark and analysis platform for natural language understanding. In Conference on Empirical Methods in Natural Language Processing: BlackboxNI.P Workshop. 2018
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems: Datasets and Benchmarks, 2024.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, et al. Emergent Abilities of Large Language Models. Transactions on Machine Learning Research, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, et al. Chain-of-Though Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, 2022b.
Adina Williams, Nikita Nangia, and Samuel Bowman. Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In North American Chapter of the Association for Computational Linguistics, 2018
Ronald Williams. Simple Statistical Gradient- following Algorithms for Connectionist Reinforce- ment Learning. Machine Learning, 1992.
14

---

Published as a conference paper at ICLR 2025
Weijia Xu, Andrzej Banburski-Fahey and Nebojsa Jojic. Reprompting: Automated Chain-of- Thought Prompt Inference Through Gibbs Sampling. In International Conference on Machine Learning. 2024.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, et al. Qwen2.5-Math Tech- nical Report: Toward Mathematical Expert Model via Self-Improvement. arXiv preprint vrXiv:2409.12122 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas Griffiths, et Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Advances in Neural Information Processing Systems, 2023.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How Transferable are Features in Deep Neural Networks? In Advances in Neural Information Processing Systems, 2014.
Fangxu Yu, Lai Jiang, Haoqiang Kang, Shibo Hao, and Lianhui Qin. Flow of Reasoning: Efficient Training of LLM Policy with Divergent Thinking. arXiv preprint arXiv:2406.05673, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang. Tianyu Zheng, Ruoqi Liu, et al. MMMU: A Massive Multi- discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. Conference Computer Vision and Pattern Recognition, 2024a.
Xiang Yue, Tianyu Zheng. YUansheng Ni, Yubo Wang, Kai Zhang, al. MMMU-Pro: A More Ro- bust Multi-disciiline Multimodal Understanding Benchmark. arXiv preprint arXiv:2409.08833, 2024b.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can Machine Really Finish Your Sentence? In Association for Computational Linguistics, 2019.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level Convolutional Networks for Text Classification. Advances Neural Information Processing Systems, 2015.
Stephen Zhao, Rob Brekelmans, Alireza Makhzani, and Roger Baker Grosse. Probabilistic Infer- ence in Language Models via Twisted Sequential Monte Carlo. In International Conference Machine Learning, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, et al. Judging LLM- as-a-Judge with MT-Bench and Chatbot Arena. In Advances Neural Information Processing Systems, 2023.
15

---

Published as a conference paper at ICLR 2025
A GENERALIZATION TO MULTI-TOKEN LABELS
In the main paper, we assume for simplicity that each E y comprise a single token, which might be the case for many datasets. Let yı [t [.i...., [/] y be multi-token label comprising l tokens. Then, to compute log prM((kk (ii, icmm) one would need to over all the tokens comprising yk:
log (yN ((I,, Me))
log PFM Po {x We))iem).
Given that our task encoder тo involves renormalization in Eq. ( summation over all the tokens for y E y would require impractical multiple model calls.
First token approximation. In case of absence of labels e y sharing their first corresponding token et, we found that the following approximation of Eq. (9) performs well in practice:
log PPM (t/) 1t Tma {(2, ((mmmmm) = log PeM Emn {(2,, ve)
(10)
Bag-of- Tokens (Bo'T) approximation. First token approximation would not work in case there are labels ye, Uj y' that share prefix. Such scenario mostly occurs for fine-grained image classification problems. To address this challenge, we, first, find the minimal prefix D1- [t* ..f..... t ticl my S my that allows to distinguish yı: y from the rest labels. Then, we propose to consider ýn as a Bag-of- Tokens, effectively ignoring the order of ..,..., i
l
log PFM (1) 15 Tm (1 3mmm)
logPrM((/ar (4 (1x, be)))mm)
(11)
easy to note that Bag-of- Tokees approximation reduces the first token approximation for datasets without labels that share prefix. Consequently, we use it by default for all the datasets.
16

---

Published as a conference paper at ICLR 2025
B IMPLEMENTATION DETAILS OF THE PROPOSED APPROACHES
B.1 AMORTIZED APPROACH
For the close-ended tasks it is feasible enumerate all y E V, thus renormalize conditional likelihoods over the entire set ) resulting in:
Wn ((T,, 34) (i)) def log (ym [nn (ar 96 Mi))))))
(12)
Eyey PFM i {(a y: Mie))
where it is important to note that this renormalization does not require additional model calls. It
well-known that rescaling the objective beneficial for the faster convergence REINFORCE-
based optimization methods (Mnih et al. 20155] Schulman et al. 2017 Sutton & Barto 2018 We
use this renormalization for all the summands in J in Eq. 7
Low-variance Gradient Estimator. Our objective in Eq. 5 has the following form:
E 21 E EN D. yn (10()1 (m)
Ine -U1.--.;00 Un), where
(13)
(v ynd
N log log p(ynll Lnn {(r,, Me)e)n).
Without loss of generality, let's consider particular samples 21,..... EN ~ D, since averaging over multiple samples does not introduce any bias. Thus, after rearranging terms, we need obtain the unbiased gradients for the following objective:
V y To(1) (
(y1...., Un)
(14)
Considering only nith term, let's note that:
Vg II ((l.))) (......, .y)) = Vo Va. VoBy......
Σ Jh (11. ( NEY
(15)
The key insight here is that marginalization over E y can be done efficiently without additional model calls as before. Let's denote ( 0n-1,0) Eyer JNN (V1... 11 I.,, then
..............-.. vn- J (B1..... ..n..........0 1n
(16)
E Byg.... [(,,...... 1n-1
Va log TO (us (u)l);)) + Ey 00
Jvm.........
Un 1n-1.0), 0),
where the first term can be seen as the REINFORCE gradient estimator for J(v1 12-11 0) and the second term low-variance pathwise derivative. To reduce the variance of the overall estimator even further, we introduce simple yet effective control variate for the first term. In particular, let V = arg max To (y (T)), j I,..., (n 1), then our final gradient estimator is: yEY
E
Jen. vn- 0) B 1........... 2n)
V log (o(
(17)
+ E
J (y1 1n-1, 0), where
Un- Nn- 00
B(a )nn) Σ JNN
u) (v)))))
,EY
The obtained estimator admits the unbiased estimate by sampling Vı, VI..... yN (e(: and calcu- lating what inside expectations. Figure Bl demonstrates the effectiveness of the proposed gradient estimator several tasks.
B.2 MULTI-TURN FOR UNSUPERVISED IN-CONTEXT LEARNING
17

---

Published as a conference paper at ICLR 2025
DBPedia14 with Llama-3.1
CIFAR100 OpenFlamingo-48
-1.0
0.80
-1.2
0.85
Our objective
-1.4
-1.6
0.95
-1.00
-188
-1.05
Our Gradient Estimator Naive
-2.0
-1100
0
200
400
600
800
200
400
600
800
Training Iteration
Training Iteration
Figure B1: Comparison of gradient estimator with the naive approach. The plot shows the convergence rate during optimization of the joint inference objective. Our proposed gradient estimator achieves faster convergence and leads to the higher values of the objective.
Algorithm B1 Amortized Approach
1: Input: Dataset D, Foundation model (-). hyperparameter N, LoRA task encoder ro(.) with parameters 0, regularization strength 7, number of iterations T,, learning rate a, batch size
2: Initialize 0g such that 78, PFM
3: for t to T do
4: Sample mini- 24
N
D, b
B
5: Sample answers 3 Te,
N:b
B
6:
Estimate Te. prior = NxB b=1
ΣN Em=1 To, (11x))
Compute the objective O E > DM
JRN (y1s..., Un) + RR(To, (iirr
7:
8: Compute the gradient estimator via Eq.
(17
9: Updante the parameters: 0c+1 02 agt
10: end for
Produce answers Un arg max ,o (3) x for all
DEY
Output: Answers for DD
Algorithm B2 Multi-Turn Approach
1: Input: Dataset D, Foundation model prM(-), hyperparameter N, number of turns T, number of
repeats N,
2: Initialize answers with zero-shot predictions: Da {(x, v) D, y p=M(+1x:)}
3: for w- to T do
4: Initialize D:- =
5: for x E D do
6: for n to N, do Sample support examples labeled by previous turn: (01, 1/-111 )..., ..... (KN. D.-1
7:
8:
..... Obtain answer: Vã PFM(-11, (#I, )..., (IN, 111)) end for
9:
10:
Take majority vote over N, options: 8" MAJ(Y ...,
11:
Update answers: D, D; U {
12:
end for
13:
end for
14:
Take answers from the last turn: {yn (Ins Und Dr
Output: Answers for D
18

---

Published as a conference paper at ICLR 2025
C EXPERIMENTAL DETAILS
C.I DATASETS AND PROMPTS
Text. We evaluate our method on 16 NLP datasets covering various tasks. For sentiment analysis, we use SS72 (Socher et al. 2013), which contains movie reviews classified as positive or negative, and Amazon (McAuley & Leskoved 2013). dataset of product reviews with similar labels. For topic classification, we use AG-News (Zhang et al. 2015). which consists of news articles catego- rized into four topics (World, Sports, Business, and Technology), TREC (Voorhees & Tice, 2000) for classifying questions into six types, and DBpedia- 14 (Lehmann et al. Wikipedia articles grouped into 14 categories. SUBJ (Pang & Leel 2004) is used for classifying sentences as subjective or objective. For natural language inference, we use RTE (Wang et al. 2018]
2015), which includes
assess entailment relationships, QNLI (Rajpurkar al. 2016) for sentence-answeriig tasks, and MNLI (Williams et al. 2018), which involves classifying sentence pairs into entailment, contradic-
tion, or neutral. We also melude COPA (Roemmele et al. 2011 I and HellaSwag (Zellers al. 2019) for story completion, BoolQ (Clark et al.) (019) for yes/no question answering, and PIQA (Bisk) 2020) for physical commonsense reasoning. For open-ended questions, GSM8K (Cobbe et al. assesses mathematical reasoning through multi-step word problems, MMLU (Hendrycks)
al.
2021
ct al. 202I) and MMLU-Pro (Wang et al. 2024) measure multi-task language understanding and knowledge across diverse range of subjects.
For each dataset, we randomly sample 2, 000 examples as the train split for unsupervised learning. and 1, 000 examples as the test split for evaluation (except for COPA where there only 500 examples in total). We balance labels in both train split and test split. For GSM8K (Cobbe et al. 2021],
the whole test which contains 1319 examples for the evaluation. The datasets and corresponding prompts are summarized at Table CT
Vision. We evaluate method on ten vision datasets, including four image classification tasks and six visual question-answering tasks. For image classification, we use CIFARIO (Krizhevsky et al., 2009). benchmark dataset with color images across 10 different classes, CITAR1OO (Krizhevsky et al. 2009) and ImageNet-100 (Deng et al., 2009). which provide a more detailed classification challenge with 100 classes. We also include Foodlol (Bossard al. 2014). a large-scale dataset featuring wide variety of food categories. For visual question answering. we use COCO-Color and COCO-Number, both derived from VQAv2 (Goyal al., 2018), VQAr2 itself and VizWiz (Gurari et al. 2018). COCO-Color focuses on questions about the dominant colors of objects in images, testing the model's ability to understand color attributes, while COCO-Number involves predicting numerical attributes such object counts, evaluating the model's numeric reasoning based visual input. Furthermore, we use challenging MMMU Yuu et al. 2024a) and MMMU-PM (Yue et al., 2024b) datasets that assess multi-discipline multimodal understanding capabilities of foundation models.
For all vision datasets, unless mentioned otherwise, train the model the entire training set and report performance on the test set. Details of the prompts used for each dataset can be found in Table C1
C.2 IMPLEMENTATION DETAILS/ / HYPERPARAMETER
Unsupervised Fine-tuning. We use LoRA Hu et al. 2022) for parameter-efficient fine-tuning on NLP and vision tasks. For NLP tasks with Llama-3.1, we also use ftshhattention (Dao et al. 2022) and 4-bit quantization of the model provided by the Unsloth library to improve efficiency. We found that with improved gradient estimator, the training is less sensitive to the syper-parameters. Thus we do not customize hyperparamters for each datasets, and instead using learning rate of le-5 with Adam optimizer for all datasets. The model is fine-tuned for 6,000 iterations and usually the training converges at around 2,000 iterations. We train our model with 64 examples at each mini-batcd We use context-length N = 16 for the main experiments and provide ablation study on the effect of N Appendix D.I Similarly, for vision experiments, we train our model for 3,000 iterations with learning rate of le-4, and 256 examples at each iteration. The typical training time is 12h for text tasks and 4h for vision tasks, on NVIDIA H100 GPU.
TThe library could he found httpppp////gtgttb..c///////////p
19

---

Published as a conference paper at ICLR 2025
Unsupervised ICL. For unsupervised in-context learning (ICL). initialize pseudo-labels using zero-shot predictions and iteratively refine them based on ICL predictions. At each iteration, the label of query example is updated based on the ICL prediction from N support examples. For tasks that involve reasoning process, such as GSM8K and MMI.U-pro, we also initialize the reasoning chain with zero-shot Chain-of-Thought (CoT) inference and then update it through unsupervised ICL. In addition, we manually filter out unformatted responses when selecting support examples, as find this step helpful improve the quality of demonstrations for open-ended tasks. For both supervised and unsupervised ICL, we manually balance the labels when sampling support examples, as this helps prevent biased predictions. Additionally, we sample support sequences per iteration and apply majority vote to reduce variance (except for MMLU-pro). The labels are updated across turns, if not specifically mentioned, after which we report the aceuracy on the test set.
GPT-4o evaluation. We use the version of "gpt-4o-2024-08-06" for evaluation. We experiment on subset the ImageNet dataset with 1000 support images and 5000 evaluation images corre- sponding 100 classes. We perform two-turn pseudo labeling for unsupervised ICL and 16-shot for evaluation. The total cost for the API call and evaluation $200.
20

---

Published as a conference paper at ICLR 2025
Table CI: Datasets and corresponding prompts used in this paper.
Dutaset
Procapts
<yeneence
The sentiment of the sentence is <label>,
<comaorr
Amazon
sentiment sentence <clobelts,
AG-Newt News
The tepic of the seutence about clabefo..
<teup
TREC
The topic of the sentence about <labiots.
DBpedia-14
<Rile><ccmmt
The topic sentence ahout clahets. Sleats
SUBJ
The sentence <label>.
Spremize
RTE
Qusstion: Does this imply that "chppotiesiss" yes or no?
Anewer: <lobol>.
S
QNLI
Question: Does that sentence have all you need answer the question "<questio>>" yes or no?
Anewer: <label>.
Spremize>>
MNLI
Based on the previnus passage, is tiie that "cckynothssie>""
Answer: <labol>.
Consider the following premise: "preemeee>"
Choice 1: <cholce
COPA
Choice <cholce2>
Q: Which one more likely to be the <question> choice choice 2?
A: <llbbll..
<pnssage>>
BoolQ
Question: Afler reading this passage. the answer the question <quertion>>> is yes or no?
Answer: <label>.
<goul>>
Solution 1: <>mll>>>
PIQA
Solution 2: <>ml2>
Question: Given the goal, what the correct solution, solution solution 2?
Answer: <lobelt.
Consider the following description: "crtio"
Choice <
Choice <endingr2>
HellaSwag
Choice 3: <endings?>>
Choice <ondingsl>>
Question: Which most plausible ending, choice choice 2, choice choice 4?
Answer: <labell>.
the following problem, and give final answer the problem.
GSMSK
Problem: <'questio>>>
Answer: <lobel>>
CIFAR
<lmoge> image <clbbll.. (oudlb::/oddoooouu Jondefobouk
CIFAR 100
Aimoge'AAn
<label>. (endoj-land [>
Food101
<imoge> An image of clabelo... (ondefchank] >
COCDColor
Quungrrueesion:::: Equestion>?
<lobel> endofooond)
CoCoeNumber
Image: Question: <questtion>> Short answer: <label> < (udogohunk
Image>-uusstion::
VQAV2
Cinngr>Quesion:: Question: <question>? Short answer: <lobe)> endojrbbud)
ViZWiz
Cimage Question: <question?? Short answer: <label> < londefokunk)
ImageNet-100 100
specified in the system prompt
Simage
Beange-PPeesee Please identity the class of the image provided, The class has belong to one of the classss
The lollowing are muliiple choice questions (with answers) about cabbjett..
MMLU
<question>
Options: 2option''
The following are multiple choice questions (with anewers) about category. T Think step by step and then finish your answer with "the answer is (X) where X is the conrect letter choice.
MMLU-Pro
Question: <question>
Options: <optious'>
Answer: <onswer>
Prompt mulliple-chooce question:
System prompt: Answer the following multiple-choie question. The last line of your response should be of the following format:
'Answer: SL SLETTER' (without quotes) where LETTER is one of the options. Think step by step before answering.
prompt:
Squextions
<options>>
The line of response should be the following
'Anewar: B.ETTERR' (without quotes) where LEETER
options. Think step by step before answering.
Conswer
MMMU, MMMU-Pro
Prompt for open question:
Syctem prompt: Answer following open open-ended question. not use lotex. step by step before answering.
Provide the Einal answer in the last line of your response in the following format
'The final answer SANSWER (without quotes) where ANSWER is single word, phrase number. prompt:
Equestion>>
Coptions?>
use latex. Think step by step before answering, Provide final answer the line your response in the following format 'The final answer is SANSWER' (without quotes) where SANSWER is single word. phrase or numbher
21

---

Published as a conference paper at ICLR 2025
D ADDITIONAL EXPERIMENTS
D.1 THE INFLUENCE OF THE CONTEXT LENGTH N
We examine the impact of the context length N on the performance of our method across both language-only and vision-language tasks. As shown in Figure DI] increasing the context length consistently improves the performance for both our methods, demonstrating the benefits of the joint inference framework to improve the predictions a foundation model upon the zero-shot inference. Remarkably, unsupervised ICL closely matches the performance of its corresponding supervised upper bound for different values of N. is also worth noting that the well-known self-training principle, e-g. |Huang et al. (2022), resembles as the special case of our unsupervised fine-tuning with 1.
RTE with Llama-3.1-88
CIFAR100 with OpenFlamingo-48
90
85
Supervised Fine-Tuning
Supervised Fine-Tuning
85
80
Accuracy (%)
80
75
75
Unsupervised Fine-Tuning
70
Unsupervised ICL
70
Supervised ICL
65
65
Zero-shot Inference
60
Zero=shot Inference
21
23
2
21
23
2
27
N
Figure D1: The effect of the context length N. We show the performance of both our methods for different context lengths (N). For both text (left) and image (right) classification tasks, our method displays consistent improvement as N increases. This demonstrates the benefits of making joint predictions for multiple examples under the proposed joint inference framework.
D.2 ABLATION OF TASK ENCODER PARAMETRIZATION
The choice of the parametrization the task encoder for fine-tuning can lead to different results in terms of under-fitting and weer-fitingg. Notably, method can be applied to any fine-tuning strategy, the only difference with supervised fine-tuning is that we do not need labels. We chose LoRA widely-adopted adaptation method that trades off over-fitting and under-fitting. To further strengthen our experimental evaluation, we conduct additional experiments of fine-tuning via (i) full fine-tuning and (ii) using a linear head on top of the fixed model (Table DI.
First, the results suggest that unsupervised fine-tuning with all three task encoders substantially outperforms the zero-shot performance each of the considered datasets. For instance, even with the simple linear task encoder, unsupervised fine-tuning improves by 12%, 10% and 12% on the SST2, RTE and Boolq dataset respectively. On the other hand, we can observe that linear task encoder significantly underfits both LoRA and full fine-tuning. In particular, LoRA fine-tuning outperforms the linear task encoder by 3%, 9% and 3% on SST2, RTE and Boolq datasets respectively.
Furthermore, it can be observed that supervised fine-tuning with full model training severely overfits on the RTE dataset compared to LoRA parameter efficient fine-tuning, In turn, the unsupervised fine- tuning with full model training inherits this problem and performs worse compared to the LoRA fine-tuning. In particular, compared to LoRA supervised fine-tuning, full supervised fine-tuning drops by 7% on the RTE dataset. This, in turn, results in a 2% drop of the performance full unsupervised fine-tuning compared to LoRA unsupervised fine-tuning. Overall, the obtained results
22

---

Published as a conference paper at ICLR 2025
suggest that the LoRA fine-tuning provides the practical trade-off between parameter efficiency and the performance.
Table DI: Task encoder ablation. LoRA fine-tuning provides the practical trade-off between pa-
rameter efficiency and the performance.
Trainable Param.
SST2
RTE
BoolQ
Zero-shot
77.7
62.7
66.7
Unsupervised FT
LoRA
92.3
81.7
81.7
Linear
89.6
72.8
78.6
Full
92.3
80.3
81.3
Supervised FT
LoRA
92.1
89.0
85.6
Linear
90.1
74.4
79.9
Full
92.0
82.4
85.7
D.3 CONVERGENCE RATE THE MULTI-TURN UNSUPERVISED IN-CONTEXT LEARNING
In addition, we investigate the performance of our unsupervised ICL with respect to the number of turns. The results are shown in Figure D2] Interestingly, we find that the method often converges to near-optimal performance with only few turns, approaching the supervised ICL upper bound.
DBPedia-1 14 with Llama-3.1-8B
CIFAR100 with Open
100
Supervised ICL
70
Supervised ICL
Accuracy (%)
90
65
Unsupervised ICL
Unsupervised ICL
85
80
60
75
Zero-shot Inference
shro-shottio Inference
Zero-shot 1
2
3
5
Zero-shot 1
3
4
Initialization
Initialization
# #Turn of Unsupervised ICL
#Tum of Unsupervised ICL
Figure D2: The convergence analysis of the multi-turns unsupervised ICL method. We study the number of relabeling turns need for the unsupervised ICL method converge. We find that the proposed method converges to near-optimal performance after only a few turns and approaches the upper bound supervised ICI. performance.
D.4 THE INFLUENCE OF INSTRUCTION-TUNING AND MODEL SIZE
We study the performance of our methods, unsupervised fine-tuning and unsupervised ICL, when applied to the iistruction-tuned Llama-3.1-8F and the larger scale Llama-3.1-7088 models. Results show that our joint inference framework effectively applicable across different model sizes com- pared to Chain-of-Thoughi (Figure D3). which can improve foundation model only for large-scale models. In addition, even for the large-scale Llama-3. 3.170B model, our unsupervised fine-tuning and unsupervised ICL significantly outperform the Mhain-of-Thouue prompting technique. In par- ticular, it surpasses CoT by 5% and 4% on the SST-2 and RTE datasets, respectively, providing a principled approach to enhance predictions for models across different sizes.
23

---

Published as a conference paper at ICLR 2025
SST-2
RTE
90
90
Zaro shot
88
Unsupervised ICL (Ours)
Far Superviser ICL
80
Unsupervised
Supervisec
75
50
/UM-iistruct
/UU-inntruct
Model
Model
Figure D3: Using instruction- tuned and larger scale models. We evaluate our methods on the base 8B, instruction-tur both proposed methods scale to instruction -unndd and larger-scale models consistently outperform zero-shot baselines. Notably, our methods applied the base non-tuned 8B model outperform or work closely to the zero-shot methods on a x9 larger 70B-Instruct that also benefits from additional training.
instrust and larger rOB-Instruct from the Llama-3.1 family. We find that
D.5 WEAK IN-CONTEXT LEARNING CAPABILITIES AND JOINT INFERENCE FRAMEWORK
Small models can still exhibit ICL capabilities and improve upon zero-shot performance in some cases. To show that, we conduct extensive experiments on RTE, SST2 and BoolQ datasets using small models (Table D2). We find that whenever small model exhibits ICL capabilities, meaning that supervised ICL outperforms zero-shot baseline, one can expect improvements with the unsuper- vised ICL as well. For instance, on the SST2 dataset with the Phi 1.5 model supervised ICL achieves 42.7% relative improvement over zero-shot and the unsupervised ICL achieves 44% improvement. With the 3B parameter model, supervised and unsupervised ICL outperform zero-shot baseline by large margin consistently across all datasets.
Table D2: Small models can be improved via joint inference framework whenever they exhibit ICL capabilities.
RTE Umsup. ICL
SST2
BoolQ
Zero-shol
Sup. ICL
Zero-hol 77.7
Unsup. ICL
Sup. ICL
Zero-shot 66.7
Unsup, ICL
Sup. ICL.
Llama 3.1
62.7
78.8
80.8
92.4
93.3
82.6
84.1
Llama 3.2 3B
61.6
73.7
75.0
72.4
93.3
92.5
51.3
75.1
76.2
Phi 1.5
60.3
60.5
61.8
58.5
84.7
83.5
59.7
57.3
56.4
3.2
61.0
50.0
50.0
54.9
Furthermore, even in cases when small model doesn't exhibit ICL capabilities, unsupervised fine- tuning method can be used to improve this small model using feedback from a larger model with stronger ICL capabilities (Table D3) For example, for the smallest tried Llama-3.2-1B model, our unsupervised FT method significantly improves upon zero-shot when using larger model to compute the joint inference objective and provide feedback the small task encoder (Table D3). First, it is easy to see that even using the same 1B model to compute the joint inference objective leads to substantial gains in the performance (Table D2). For example, as shown in the experiment of previous paragraph, supervised ICL on the RTE dataset leads to only 5% improvement upon the zero-shot baseline. In turn, unsupervised fine-tuning leads 10% improvement, outperforming the supervised ICL (Table D3). Furthermore, using larger models for computing the joint inference objective further improves the performance of the small Llama-3.2-11 model, closely approaching the supervised fine-tuning upper bound. For instance, the BoolQ dataset, using the 70B model in the objective results a remarkable 24% absolute gain over the zero-shot baseline. We note that this is one-time adaptation cost per task, after which the inference is made with the low-cost small model. Overall, these results suggest that both our methods readily applicable to a broad range of I.LMs.
24

---

Published as a conference paper at ICLR 2025
Table D3: Large models that exhibit ICL capabilities can be used to improve small models with weak ICL capabilities.
Objective compute with
SST2
RTE
BoolQ
Zero-shot
68.6
50.8
50.0
Unsupervised FT
Llama 3.2 1B
90.5
60.8
57.4
Llama 3.2 8B
90.8
74.5
73.4
Llama 3.2 TOB-Instruct
90.6
75.6
74.4
Supervised FT
90.0
79.3
75.5
25


Reviews:
[{"review_id": "8sBU3UXZW8", "reviewer": "Reviewer_UFdR", "summary": "This paper presents a joint inference framework for large language and vision-language models, with an optimized and efficient approximation techniques to enable unsupervised adaptation to new tasks through two main methods: unsupervised fine-tuning and unsupervised in-context learning (ICL). This approach makes predictions jointly across multiple inputs, and shows significant improvements over zero-shot inference, sometimes achieving results close to supervised approaches without needing labeled data.", "questions": "Q1: Do you have any evaluations on how the joint inference framework affects model robustness, especially when inputs are noisy or out-of-distribution?\n\nQ2: It is also very interesting to see if there are any common failure cases/scenarios of the proposed method?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) A big strength of this method is that it provides a framework to effectively allows unsupervised fine-tuning and ICL, reducing the need for labeled examples which is usually expansive in practice.\n2) It also demonstrates considerable accuracy and generalization improvements across a range of language and vision tasks, even comparing with supervised fine-tuning methods. The experiments performed on multiple datasets, models, and tasks, demonstrating the framework’s effectiveness in both text and vision-language scenarios.\n3) It generally works for various model types such as open-weight models (e.g., Llama-3.1, OpenFlamingo) and close-weight models (e.g., GPT-4), showing flexibility and robustness in application.", "weaknesses": "1) The effectiveness of unsupervised ICL seems to rely on the model’s inherent in-context learning abilities. This may restrict its applicability to models lacking strong ICL capabilities. It is unclear how does the same method work for other models with relative weak capability.\n2) While the experimental results are promising, a deeper theoretical analysis of the joint inference framework’s optimization behavior and potential limitations or failures would strengthen the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BkTc4UuxSZ", "reviewer": "Reviewer_UqwZ", "summary": "The paper proposes an 'unsupervised joint inference' framework to improve the performance of large language / vision-language models on tasks without using labeled data. The proposed approach predicts output sequences simultaneously for multiple inputs, leveraging inter-sample dependencies. There are two variants in the proposed approach: 1. use the model's own predictions to finetune its parameters. 2. Use in-context learning to refine predictions iteratively. The paper conducts experiments on some nlp and vision-language datasets and claim substantial improvements to vanilla zero-shot approach and often on-par performance with supervised approaches.", "questions": "Is there a reason for picking these datasets for experiments? Is it possible to perform the same experiments on more recent and broadly adopted datasets?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper proposes an novel framework to adapt llm or vlm for unseen tasks. \n2. The paper intents to build solid theoretical foundation for the proposed approach. It establishes a theoretical connection between the proposed unsupervised ICL method and the joint inference objective, showing how the former approximates the latter.\n3. The experiments conducted are diverse on different tasks: text classification, image classification, question answering, visual question answering, natural language inference, common-sense reasoning, and math problem-solving.\n4. Experiments show that the proposed approach is effective and outperform the baselines by a large margin.", "weaknesses": "1. The paper overclaims it's contribution and effectiveness of its approach. In abstract, the paper claims that 'our approach, although unsupervised, often performs on par with supervised approaches that use ground truth labels.' However, its an overclaim for the following reasons: (1) In the results shown in the paper, most of the time supervised approach is better than the proposed approach. (2) Supervised approach uses LoRA instead of full model training. \n2. The experiments are not solid. Some of the experiments in the paper use non-standard (potentially easy/small/old/deprecated) datasets. Most notably - CIFAR 10/100 for image classification and COCO-colar and COCO-number for VQA. The results are much less convincing compared to ImageNet for classification and VQA v2 for VQA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FEKidV8yh1", "reviewer": "Reviewer_Nhh8", "summary": "The paper introduces an unsupervised joint inference framework to improve the performance of LLMs and VLMs by considering the entire set of questions for a given task instead of making predictions on each question independently. They utilize the shared context of these questions to improve model accuracy It includes a self-improvement loop where the model iteratively refines its predictions by using its own prior answers as in-context examples to guide the next round of predictions. This self-refinement continues over multiple iterations, allowing the model to gradually improve and make more consistent predictions without needing labeled data. The authors show that this", "questions": "- how does your method answer multiple questions with one model call? What is the exact prompt used and how are the in context examples formatted? This should be in the appendix, table C1 does not indicate where in context examples are placed nor does is show how multiple questions are answered in a single pass. If the authors could give a template of the prompt used in the iteration steps for a benchmark that would be sufficient\n- are the in context examples used different than the questions the LLM is tasked to answer in a given call?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- the motivation of utilizing the entire corpus of the task to improve overall performance is compelling, as there is often shared context that is underutilized\n- this method is versatile and applies to both open and closed source models as well as across modalities\n- the results indicate that this can provide substancial improvements over zero shot methods", "weaknesses": "- the main weakness of this paper is clarity: sections 3 and 4 are hard to follow and it would have been significantly clearer to provide a methods figure which outlines the iterations and IC examples used. If I understand this method correctly, it uses responses to questions from previous iterations as in context examples for questions in the current iteration, optimizing for the joint likelyhood over all the examples in the batch. If this is the case, then it should have been stated clearly in prose in section 4. I find the entirety of section 3 to be unnecessary as it is explaining commonly known procedures for model training and inference, and much of section 4 to use unnecessary notation in explaining a relatively straightforward method. Again, I have no major concerns about the method itself, but I urge the authors to remove or significantly shorten section 3, add a methods figure, and rewrite section 4. A methods figure could be in the form of showing the inputs to the LLM at each iteration for the no weight-update option to indicate the change in in context examples. \n- I am not fully convinced that adding iterations helps for the no weight update setting. if the authors added a 1-iteration baseline to their results this would resolve my concern\n- all the benchmarks used in the results section are out of date and focus on easy tasks. The results would be much more compelling if the authors included a benchmark like MMLU. That being said, given the short rebuttal period I would put this experiment as a lower priority than the two above concerns \n\nI acknowledge that I may have misinterpreted the method so some of these critiques could be unfounded. If that's the case, I am happy to learn more about the method - preferably via a methods figure - in the rebuttal and change my rating", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7iU3jqT5yu", "reviewer": "Reviewer_TGs1", "summary": "This paper aims to enhance the zero-shot inference capabilities of existing open- and closed-weight models. To achieve this, it introduces a joint inference framework that processes and makes predictions for all task-related inputs simultaneously, rather than handling each input individually. The paper presents two efficient approximations—unsupervised fine-tuning and unsupervised in-context learning (ICL)—to reduce the computational complexity of joint inference. The results demonstrate the effectiveness of the proposed method across NLP and vision tasks, using two large language models (LLaMA-3 and GPT-4o) and one vision-language model (OpenFlamingo).", "questions": "Asked above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- I believe the idea of self-refinement using model predictions is solid, as is the approach of zero-shot evaluation without a labeled support set, which contrasts with the traditional definition of supervised in-context learning (ICL). In my view, the paper aligns well with the concept of inference-time compute scaling, as evidenced by the O1 model by OpenAI, where generating the final answer requires multiple steps (turns) by the model. Hence, the problem definition is quite sound. \n\n- The paper presents two distinct approximations for the joint inference task, which appear to outperform all baselines. The authors demonstrate this rigorously across several NLP and vision tasks.", "weaknesses": "In the following, I will provide a list of weaknesses and a follow-up question for each in case the authors want to reply.\n\n1. **Model Cacapcity and ICL Abilities**: The proposed method assumes that the pretrained model already possesses few-shot in-context learning (ICL) abilities, which are typically found in large-scale models (those with over 8 billion parameters). What about models that don't have these ICL capabilities? Can they leverage their context more effectively during joint inference? To be more specific, if the authors use the Phi-1.5 model and conduct unsupervised fine-tuning or unsupervised ICL again, what would be the outcome?\n\n2. **Reliance on Task Encoder Approximation**: I think the design of the Task Encoder has not been explored properly in the paper. The authors only did not mention why the same pre-trained model + LoRA can be a good approximation for the task encoder approximation tasks. Why is this case? Based on which ablation in the paper? What are the other alternatives? It’s important to note that, as shown in other research, LoRA often tends to underfit a task, which can lead to suboptimal or inconsistent improvements. This raises concerns that the proposed task inference model may not accurately reflect the tasks that the model is intended to predict.\n\n3. **Limited Applicability to Open-Ended Tasks**: Aside from the single open-ended experiment mentioned in line 395 for the GSM8K benchmark, I do not see any other experiments demonstrating the effectiveness of the proposed method for open-ended generation. With just one experiment, it is not convincing that the method is effectively generalizable for both unimodal and multimodal open-ended generation. Do the authors have any additional ablations or experiments to support the strengths of their proposed method?\n\n4. **Missing Compute-related Ablations**: In multi-turn inference settings, it is crucial to provide the number of FLOPs used to arrive at the final answer. Do the authors have any experiments that demonstrate the scaling curve, with FLOPs on the x-axis and performance on the y-axis?\n\n5. **Lack of Proper Evaluation in Multi-modal Setting**: In the paper, the authors used OpenFlamingo and demonstrated that the proposed method shows some effectiveness. However, I'm curious about their design choice and why they opted for OpenFlamingo. What about other alternatives that may be more powerful and capable of handling larger contexts? For example, LLaVA-Next-Interleaved is a suitable candidate, as it is trained with the in-context learning (ICL) objective function. Could the proposed method work properly if LLaVA-Next-Interleaved were used instead of OpenFlamingo? Additionally, regarding multimodal evaluation, assessing a model using COCO-color and COCO-count might not provide an accurate reflection of the proposed method's performance. Have the authors considered evaluation on the MMMU Benchmark? This benchmark is recognized as a legitimate evaluation tool for multimodal large language models, and LLaVA-Next-Interleaved has reported strong results on it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)