Can LLMs Reason Over Non-Text Modalities in a Training-Free Manner? A Case Study with In-Context Representation Learning
Authors: Tianle Zhang
Wanlong Fang
Jonathan Woo
Paridhi Latawa
Deepak A. Subramanian
Alvin Chan
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Training-Free Adaptation
Foundation Models
Multi-Modal Large Language Models
In-Context Learning
Primary area: deep_learning

Abstract: The remarkable performance of Large Language Models (LLMs) can be enhanced with test-time computation, which relies on external tools and even other deep learning models.
However, existing approaches for integrating non-text modality representations into LLMs typically require additional costly supervised training, restricting on-the-fly adaptation to new domains and modalities. 
In this work, we explore the feasibility of integrating representations from non-text foundational models (FMs) into text-based LLMs in a training-free manner. We propose In-Context Representation Learning (ICRL) as a proof-of-concept to allow LLMs to adaptively utilize non-text modality representations with few-shot learning. 
Unlike traditional in-context learning, which incorporates text-label pairs, ICRL replaces text inputs with FM representations, enabling the LLM to perform multi-modal inference without fine-tuning. 
We evaluate ICRL on a suite of tasks in the molecular domain, investigating three core research questions: (i) how to map FM representations into LLMs in a training-free manner, (ii) what factors influence ICRL performance, and (iii) what mechanisms underlie the effectiveness of ICRL. 
To the best of our knowledge, ICRL is the first training-free framework for integrating non-text modality representations into text-based LLMs, presenting a promising direction for adaptable, multi-modal generalization.

Can LLMs Reason Over Non-Text Modalities in a Training-Free Manner? A Case Study with In-Context Representation Learning
Tianle Zhang'; Wanlong Fang2:1; Jonathan W. Paridhi Latawa", Deepak A. Subramanian', Alvin Chan 1.3,4
Colleee of Computing and Data Science, Nanyang Technological University
2 AIIX, Interdisciplinary Graduate Programme, Nanyang Technological University Lee Kong Chian School of Medicine, Nanyang Technological University
4 *Centre of Al in Medicine (C-AIM), Nanyang Technological University
University of Toronto, Brrihham and Women's 's Hospital, Harvard Medical School "Massachusetts Institute of Technology
Abstract
The remarkable performance of Large Language Models (LLMs) can enhanced with test-time computation, which relies external tools and even other deep learning models. However, existing approaches for integrating non-text modality representations into LLMs typically require additional costly supervised training, restricting on-the-fly adaptation to new domains and modalities. In this work, we explore the feasibility of integrating representations from non-text foundational models (FMs) into text-based LLMS in a training-free manner. We propose In- Context Representation Learning (ICRL) a proof-of-concept allow LLMS to adaptively utilize non-text modality representations with few-shot learning. Unlike traditional in-context learning, which incorporates text-label pairs, ICRL replaces text inputs with FM representations, enabling the LLM to perform multi- modal inference without fine-tuning. We evaluate ICRL on suite of tasks in the molecular domain, investigating three core research questions: (i) how to map FM representations into LLMs in a training-free manner, (ii) what factors influence ICRL performance, and (iii) what mechanisms underlie the effectiveness of ICRL. To the best of our knowledge, ICRL is the first training-free framework for integrating non-text modality representations into text-based LLMss presenting a promising direction for adaptable, multi-modal generalization.'
Introduction
Large Language Models (LLMs) have demonstrated remarkable versatility in leveraging test-time computation [57, 48], allowing them to dynamically adapt new tasks and perform complex reason- ing without additional training. A key extension of this capability is their ability to integrate external tools, including other deep learning models [4, 10], to enhance their problem-solving potential. Existing multi-agent frameworks allow LLMs to incorporate predictions from external models during inference, expanding their applicability to complex, multi-step tasks [49, 24). Despite the advances, these methods typically use only the final outputs of external models, limiting the effective use of their internal knowledge [25, 49]. To address this, recent efforts have explored enabling LLMs to leverage intermediate representations from external models. A promising direction involves allowing
"Equal contribution. I Corresponding author: Email: guoweialvin.chant ntu. edu.sd Our code is available at https://github. wwt.:/ccnteeeee/eeeiiiemmmmm Ftt.eeeeeeeeELLMMIIII_ACIE
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

text-based LLMs to process non-text modalities (e.g., images [34]) by incorporating representations from modality-specific foundation models (FMs) (Fig. 1(b)). However, these approaches typically require additional supervised training either for modality-specifft projection layers or even for the LLM itself-to enable a text-based L.MM to incorporate a new modality. This process is typically computationally intensive and requires specialized paired dataset between text and the target new modality. This raises a fundamental question: Can a text-based LLM leverage representations from other
foundational models during inference, without such training?
ICL
MLLM
ICRL
Text Level
Embeddiieg Level
N
Emboddiru Laper
Projecttoo
Embedding Layer
Embbedding Layer
Projector
Rep Label
Embedoling Layer
Input xu Labelw
Test input i) Modaliy trpul: Label y
Pre trincd Medality fronder
Pre-traned FM
Prainndned
FM
ni. Lacely
Iose il (SNLLE cocypocc Lotull hooprttt 2.3
Irpet Io sepporent Octovend tebec y (Preleted hovpentt 9.35
(a)
(b)
(c)
Traivresufred
*
Tralefice
Texeroken
Modality token
Aligned tolion
N: Example rame
Injection drategy
Figure 1: Comparison of (a) ICL, (b) multi-modal LLM, and (c) in-context representation learning.
In-context learning (ICL), core capability of large language models (LLMss,, enables task adaptation at inference time through few-shot examples [56, 12], offering promising path toward flexible, training-free generalization. In this study, we explore how ICL can be extended to process rep- resentations from mopality-specific foundation models (FMs) during inference. To this end, we propose In- In-Context Representation Learning (ICRL)-a proof-of- designed to enhance LLM adaptability and enable a single model to generalize efficiently across diverse modalities (Fig. 1(c)). In contrast to ICL where text-label pairs (Fig. (a)) are included in the text prompt for the LLM to conduct few-shot prediction, ICRL replaces the sample text (e.g. SMILES of molecule") with the representation from an external FM (Fig. 1(c)). Note that our primary objective is not to outperform ICL but to investigate the feasibility of adaptively integrating non-text FM representations into text-based LLM in a training-free manner. We study ICRL and its mechanisms on a battery of molecular tasks and organize our findings around core research questions as follows.
RQ1 (Sec. & 3): How Map FM Representations into an LLM in a Training-free Manner?
We first study ICRL designs that integrate the FM representation vectors directly in the text prompt as few-shot examples. To fit these high- digh-dimeniionl vectors as text within LLM's limited context window, we employ dimensionality reduction (i.e., PCA) and find this simple strategy surprisingly effective. We also investigate using a projection module to interface between the FM representations and a text-hased L.M's embedding layer. To enable inference-time integration of the non-text modality, we also examine several training-free projection methods that map FM representations into the LLM's s embedding space without fine-tuning and compare their empirical performances. Notably, our analysis indicates that an approach based optimal transport theory, which aligns the distribution of FM representations with that of LLM embeddings, yields promising results.
RQ2 (Sec. 4): What Factors affect ICRL Performance?
We found that key parameters affecting standard ICL, such as the number of few-shot examples, similarly influence ICRL. Additionally, analyze various parameters in ICRL to gain deeper insights into their impact on performance.
"To ensure a fair comparison, we use molecular datasets where SMILES strings encode structural information in a textual format naturally compatible with LLMs, avoiding the need for architectural changes or fine-tuning. This allows us to study non-text modality integration into ICL pipelines in clean, training free setting. Related modalities (e.g., protein sequences) and preliminary results on vision and speech are discussed in Sec. 7.

---

RQ3 (Sec. 5): What are the Mechanisms behind ICRL?
Our findings show that ICRL performance improves when projected FM representations closely resemble their corresponding text embeddings, while larger deviations can harm performance. Addi- tionally, we observed an inverse correlation between the similarity of ICRL-projected representations across different few-shot examples and ICRL performance, implying that excessive uniformity among projected representations may hinder effectiveness. Lastly, we found that when traditional ICL is present, the mode ICRL representations shifts, leading them to be treated as pause tokens [17].
The main contributions of this paper are as follows: (i) To the best of our knowledge, the proposed in-context representation learning is the first training-free approach integrate non-text modalities into a text-based LLM. (ii) We explore various design choices and analyze their impact on ICRL performance across a range of molecular domain tasks. (iii) We present mechanistic insights behind ICRL to show how the distribution of projected representation affects performance.
2 How to Map FM Representations into an LLM in a Training-free Manner?
In this section, we first discuss the proposed ICRL framework, which encompasses two different loca- tions to integrate FM representations into a text-based LLM: (1) introduction of the representation as string in the prompt text and (2) injection of the FM features into the LLM embedding spaces. Then, we introduce several methods that constitute the proposed ICRL framework: detailed algorithmic descriptions of the overall pipeline and each injection strategy are provided in Appendix D.
2.1 Preliminaries
In-Context Learning & LLM. Consider the general ICL framework [12]: given a set of text inputs
X [xi,.... FT],, i.e., molecular SMILES sequences, and the task label E V, a pre-trained LLM & predicts by selecting the candidate with the highest score based on a demonstration set E. This set comprises the instruction I and & demonstration examples: E {I, (I1, ) ..... (Øk. yk)), where each (z,, y/) represents a few-shot learning exemplar. The final prediction ÿ is computed using a scoring function f over the entire input sequence:
arg max folus,F X). e,
where X represents set of text examples. Within this framework, the LLM e implements the above formulation through sequential token processing. Specifically, for each input sequence vy. the embedding layer of LLM the converts it into a sequence of token embeddings: D: = (2,) E Blxxdur. where 4; is the number of tokens corresponding to I;, and dL. is the dimensionality of each token embedding. For convenience, the distribution of LLM outputs is denoted as: DI..M T XdLLM where T = - to is the total number of tokens across all inputs.
Foundation Model. We define the FM as a mapping: 6 X + NxdeM, and the extracted representations are structured as: H R Mxdrs where M is the number of extracted feature vectors, and deM is the corresponding dimensionality. In this article, we focus on the sentence classification feature, i... M = 1, and denote the distribution of the extracted representations as DFM E R xdxu, Further details on feature extraction schemes can be found in Appendix F.6.
Projector: We employ simple multilayer linear model (MLM) to align the FM and LLM embedding spaces, defined as P Rdd -> RRdLLM The projected embedding distribution denoted as DD Drooj-
2.2 In-Context Representation Learning
Unlike standard ICL, which constructs examples (z,, y1) only from textual inputs, and Multi-modal Large Language Model (MLLM) methods requiring supervised training for modality alignment, ICRL directly injects adjusted FM representations into LLMs without training, i.e., (r., y.). bypassing raw input z. We classify our methods into two injection levels and discuss their design and theoretical foundations in the following subsections.
2.2.1 Text-Level Injection
PCA. A straightforward approach to integrating FM representations into L.MSs is to embed the high- dimensional vectors directly into the prompt as strings. However, these digh-dimensional embeddings
3

---

often surpass the context window limitations of most models and make them incompatible with ICL approaches. To address this challenge, we implement PCA for dimensionality reduction. This transformation maps the original embeddings to lower-dimensional space: 'PCAA E TR. xdReduced, where Reduced & drM- Let W rCA E RRd XdReduced, the reduced-dimension poo H x WPCA are subsequently converted into PCA strings Spca, enabling their seamless integration into the prompt while retaining the most critical features of the original representations.
embeddings
2.2.2 Embedding-LLevl Injection
While dimensionality reduction enables the FM representations to fit within the text prompt, the vector strings still occupy a substantial portion of the prompt and incur information loss, thereby limiting their effectiveness. To address these limitations and enhance the LLM's understanding of these representations, we propose several embedding-level injection methods that directly inject features into the LLM's text embedding layer:
Zero-Pad. The simplest approach applies zero padding FM representations to match the dimen- sionality of the I.LM embedding space, offering a straightforward solution that preserves the original representation components. Then, we apply normalization step (as detailed in Appendix G.1) to adjust the mean and variance of the padded representation to match the average mean and variance of the LLM's embeddings before feeding into the LLM. This prevents the generation of irrelevant or nonsensical outputs due to input embeddings with out-of-distribution statistics.
Random Projection. We employ randomly initialized MLM projector module to address the dimension mismatch problem. In contrast to previous studies [29], this projector is untrained and devoid of activation functions, which is mathematically equivalent to simple matrix multiplication, making it a lightweight and efficient solution. Then the projected features are directly concatenated with the embeddings of the rest of the example.
Optimal Transport Alignment. Directly using random projectors may result in a distribution mismatch between the LLM's embeddings and the mapped FM representations. As mathematical framework designed to align two distributions, optimal transport (OT) provides a viable solution to resolve this mismatch [53]. In this approach, we extract the tokens corresponding to the original input 2; or the representation string obtained by the PCA model. These token embeddings serve as the target distribution in OT to adjust the projected results Hproj P(H). The alignment process
formalized as follows: Let 'D Dpraj& E N XdLLM and Dtar E N XdLLM represent the source and target distributions, respectively, where N' denote the number of tokens in the target distribution. The final objective can be formulated as:
min [ *ETT(()))) Dproj DDar
c(u, v) @n((, v),
(2)
where y E 11,,, 1)) is the transport plan that defines how to map the points in the source distribution to the target one, μ and v are the marginal distributions over D Pproj and Ptar- respectively. c(u, v) is the function that measures the cost of moving an element u from Dproj to u in Dtar-
For practical implementation, align the mean and variance of each dimension of D Drroj match the corresponding dimension of Drar. i.e., for each dimension j. we have:
shift, = v, ū, and scale, Øt.j
Tn.i
where ü, Vj, Opj and ot,j represent the means and standard deviations of the j-th dimension of D Dproj and Dtar, respectively. The final aligned embeddings Haligned are obtained by applying the following transformation:
OT(Dprop Dror) = scale Hpros shift.
Given the differing target distributions, we propose two OT-based alignment methods:
Embed. To enhance the interpretability of the representation for the LLM, a suitable alignment target is the LLM embedding of input text features (e.g., SMILES). Specifically, for each input z, the target embedding (a;);; ) is computed as the mean of its token-level embeddings. The adjusted embeddings can then be represented as: Hatigned OT(H proj. (X)).
OT PCA. Another OT variant uses the embeddings of the stringified, dimensionally reduced FM representations SPCA the target distribution, as this approach more closely captures the

---

token embeddings associated with FM representations. This can be expressed as: Haligned OT(Hpro d (Spa))) Note that the computation of OT shift and scale parameters needs to be performed only once and takes negligible time, the subsequent use only requires quick adjustments. We describe the OT method in more detail in Alg.
Random Noise. To verify whether the model is learning from the FM representation, we conduct an ablation study by replacing informative FM features with random noise.
Theoretical support for linear projector. designing the projector, we initially experimented with commonly used two-layer MLP. However, the mapped representations show greater similarity, suggesting a noticeable loss of information [31]. Here, we conduct a theoretical analysis to examine how nonlinear activations in the projector layers affect the original embedding geometry.
Following [31], given a linear layer with random initialization. Let E RRddd be a random square weight matrix such that each entry WA.1 is an i.i.d. Gaussian random variable, i.e., Wk,1 ~ .N(0, 2)-
We consider affine transformations of the form Wx + b, where b E R is random bias vector. Our objective is to demonstrate that such random linear mappings preserve the norms and angles of high-dimensional vectors, thereby retaining the underlying variability of the original embeddings.
Theorem 1 (Concentration of Norm Under Random Linear Projector). Let u E R& be a fixed vector, and that E Rdxd has i..dd. entries WI,1 ~ (0, 1 /d), independent of a bias vector Rdd. Then for any ō1 E (0, 1), with probability at least - 81, there exists E1 = =0(Vlog(1/5)))
such that:
[Wu b11 (11b1
|m]]²] < 61 (11112 ||a|)²).
(5)
Proof Sketch. Each coordinate (Wu + b)s is a sum of Gaussian variables with variance on the order of ||u|l// /d, shifted by bg. By applying classical concentration inequalities (e.g., Chebyshev's s), the squared norm |[ + bi2² concentrates around its mean | ||ul². We provide the full derivation and proof for all theorems in Appendíx E.1 and E.2.
Theorem 2 (Preservation of Cosine Similarity). Let u.v E Rdd be any two fixed vectors, and W Raxd have ii.dd entries WK,1 (O, 1/d). Then for any 82 E (0.1). there exists small c = 0 (lolog(1//))) such that with high probability at least 1 - 82, we have:
cos Wu. Wr) - cos(u, v) S 62.
(6)
Remark. We set the bias term b our theoretical analysis and subsequent experiments. This choice enables exact matching between pre- and post-projection cosine similarities, any non-zero bias would introduce additional terms that obscure this relationship.
Corollary 1. Nonlinear activations may distort vector angles and inflate similarities. Formally, let a(-) be nonlinear activation (e.g., ReLU, sigmoid). Then with high probability:
|cos(a (Wv)) cos(u, v)
ccos| Wu, Wv) cos(u, v)
(7)
This effect arises because both norms and dot products tend to concentrate under random linear mappings x + Wx, thus preserving the geometric distinctions (i.e., angles) among points in high- dimensional space [31]. In contrast, nonlinear activations with range constraints sparsity sparsity-inducing induciig properties (c.g.. ReLU, sigmoid) may suppress variation and exaggerate alignment, leading to inflated cosine similarities even for originally dissimilar inputs. Empirical evidence and theoretical proof supporting this corollary are presented in Sec. 4 and Appendix E.3.
3 Experimental Results
In addition to constructing examples from representations, i.e. V:), ICRL can also utilize both the original textual input and its corresponding FM features form examples like (I,, r. y.). Extensive experiments show that adding representations can further enhance ICL performance.
3.1 Experiments Setup
Datasets. We evaluate our ICRL method on five molecular datasets: ESOL [11], Caco_wann [54], AqSolDB [52], LD50 Zhu [70], and AstraZeneca [58]. For larger datasets, following [2],
5

---

Table 1: RMSE (L) Comparison of ICRL Across Datasets. Bold/ Underline: hest/second-bed value among the Embedding Injection methods. Ran-Noi and Ran-Pro denote the Random Noise and Random Projection methods, respectively.
Text Injection
Embedding Injection
Dataset
ICL
PCA
Zero-Pad
Ran-Noi
Ran-Pro
OT-Embed
OT-PCA
ESOL
1.16 21.9e22
1.11 #2.3e4
1.73 #4.6-2
1,41 =3de3
1.69 s4.40-2
1.19 #1.8e-3
1.24 54.e33
Caco2_Wang
0.83 284e-4
0.95 32.4
1,04 14.1e
1.03 34.e
1.03 11.3e-3
0.89 32.4e
0.88 3
AqSolDB
1.92 =39ee
2.91 +1.502
4.01 +5.3e-2
3.95 =4Xed
4.02 #8.0e-2
4.06 #1.3e-1
3.25 =5.0e22
LD50_ZM -hu
0.99 =l 7e-7
1.06 a2.5e-
1.28 19.0e-4
1.21 al Be-3
1.29 s1.3e-3
1.18 ell26-3
1.14 =13344
AstraZeneca
1.37 42
1.39 15.1e5
1.55 42.0e-3
1.50 =13e22
1.54 19.2c-4
1.46 23.9e-4
1.47
1.10
Deco2
2.1
ESCL
1.27
Clcod
ESOL
5.01400
12.01000
0000
3.25
0.44
1.3
0.53
(a)
(b)
(c)
(d)
Figure 2: (a) and (b) present the performance with and without activation functions in the projector across two datasets, evaluated by RMSE (J), demonstrating that linear projectors can achieve superior results. (c) and (d) depict the impact of different projector initialization strategies, indicating that these choices have minimal influence on performance.
randomly select 1,000 samples for the test set. Additional experiments and discussions on protein and drug-target interaction datasets can be found in Appendix F.7.
Implementation & evaluation. Unless stated otherwise, raw text input features omitted in ICRL. We use Uni-Mol [68] to generate molecular representations and Llama-3.1-700 inference. During inference, we set the number of examples (shots), the PCA target dimensionality to 20, and the batch query size to 3. The projector is a two-layer MLM with 64 hidden units per layer. To ensure result stability, each method is evaluated using ten random seeds, with final results derived from the top three runs. Details about the dataset and implementation are in Appendices C and G.
[18] for
3.2 Which Injection Method is more effective?
This subsection evaluates various ICRL methods across different scenarios, analyzing their strengths and limitations 5. It is important to note that our aim is not claim state-of-the oo--th--tttttttt performance, but rather as an initial probe into training-free approaches for LLMs to leverage FM representations. Given that PCA and OT processes incur negligible fraction of computational cost (take less than two seconds, with only one additional token per sample in the input window) compared to training- based approaches, we focus comparing several representation injection strategies under similar conditions, and include ICL results as reference. A detailed discussion is provided in Sec. 7.
Comparison of representation injection approaches. As shown in Tables 1, 24 & 25, the text-level injection approach PCA outperforms other embedding-level methods on most datasets. These results demonstrate that LLMs can effectively interpret and utilize features injected in this manner, while its reliance on a large context window limits scalability. In contrast, the OT based approaches achieve comparable results with minimal context window usage, i.e., one token per FM representation. Conversely, the Zedo-Pad and Random Projection methods perform poorly, often falling below that of Random noise, indicating that these simplistic techniques fail generate suitable representations.
ICRL performance with original text features (SMILES)) Incorporating SMILES sequences with ICRL reveals that most embedding injection methods could improve performance over ICL with only SMILES strings (Table 2, 22 & 23). Notably, on the ESOL dataset, the OT-PCA method
'The conclusions we proposed are consistent across different metrics, i.e., Pearson's 's correlation coefficient (Pearson r) and Root Mean Square Error (RMSE). Additional details can be found in Appendix G.

---

(E) OT-PCA
(E) OT-Embed
(E) Ran-Pro
PCA
(T) ICL
(T-T)
(E+T) Ran-Pro-ICL.
Canodl
ESOL
Caasd
ESSL
0.41
0.50
60.30
Pemont
9-47
0.30
0:00
6:1
0.34
0.34
diremiors
Bakh overy
Batch auery
(8)
(b)
(c)
(d)
0.33
Caco?
0.98
Cano?
ESOL
0.24
0.20
2.41
Peament
3.30
0.45
3.25
0.00
2.16
4004
-008
10
20 3.10
0.12
10
Chote
Shors
(e)
(g)
(h)
Figure 3: (a) and (b): performance of various methods under different PCA dimensions. (c) and (d):
different methods behave similarly across batch query size settings. (e)-(h): Pearson correlation (f)
with increasing example count, demonstrating enhanced learning via OT-adjusted representations and
feature injection. E and T denote different injection levels.
achieves a significant 16.6% improvement compared to using textual input alone (measured by Pearson r). Surprisingly, while L.MMs exhibit strong ability interpret PCA strings and OT-based representations, their combined benefits with text features are less pronounced compared to simpler methods. Specifically, the Random Noise method consistently outperforms the baseline across all datasets when textual input is included, and Zero-Pad delivers superior results in most cases. In contrast, OT-based methods show inconsistent gains, and the PCA method-despit its prior strong cetrmannce-degradee performances across all datasets. Further analysis is provided in Sec. 5.
4 What Factors affect ICRL Performance?
Model Capability. The effectiveness of ICRL is closely linked to the capacity of the underlying pre-trained LLM. As shown Tables 20 and 21, larger models are generally better leveraging FM-derived representations and handling in-context prompts. In contrast, smaller models (i.e.. Mama--.223B--Instree exhibit noticeable performance degradation on both ICL and ICRL (see Appendix F.8). Despite this, ICRL achieves performance comparable to, and in some cases even surpassing ICL in these smaller models. This suggests that, under capacity constraints, OT-aligged FM embeddings serve strings (see Appendix F.9). Consequently, ICRL demonstrates strong potential as a lightweight and effective approach for enabling small-scale LLMs to leverage foundation model knowledge, providing promising alternative to costly supervised learning in resource-constraing settings.
informative input representation compared to the baseline SMILES
loiggnn--peffilllly.. the inclusion of activation functions and initialization choices- pplasspllssss a crucial role. We evaluate the effects of incorporating ReLU and GELU activation functions between two linear layers and ex- plore four common initialization methods: Glorot initialization [16], He initialization [20], Dirac whit structures the weight matrix to approximate an identity matrix, and standard normal initialization. Results in Figs. 2(a) and 2(b) demonstrate that activation functions consistently degrade performance, which aligns with our theoretical analysis. Additionally, Figs. 2(c) and 2(d) suggest that while alternative initialization methods may provide minor improvements, the standard normal initialization consistently yields the best overall performance. Furthermore, initialization choices have a relatively minor impact compared to activation function settings.
ICRL projector schemes. As our projector is untrained, its structural
PCA dimensions. As the dimensionality of PCA affects the length of text-level injected represen- tations, we conducted an ablation study to further investigate its impact. Interestingly, in scenarios without text input, the PCA method did not yield better performance with increasing representation length (Figs. 3(a) and 3(b).. most cases, performance even declined, suggesting that longer representations do not necessarily enhance the model's ability to interpret them. In contrast, when
7

---

Table 2: Pearson Correlation (1) Comparison Across Datasets. Bold/ Underline: best/second-best value compared with ICL.
Baseline
ICRL (Ours)
Dataset
Text
Text PCA+ICL
Embedding
ICL
Zero-Pad+ICL
Ran-Noi-ICL.
Ran-Pro+ICL
OT-Embod+ICL
OT-PCA+ICL
ESOL
0.465
0.455
0.526
0.540
0.525
0.508
0.542
Caco2_ Wang
k13w6
193e-4
0.410 a4ited
0.420 a1le-
0.405
0.429 a1
1576-1
AqSolDB
0.596 1s
0.549 252z-4
0,606 shiled
0,597 a1.ee
0.600 4246-5
0.569 25.
0.589 a1Me5
LD50 Zhu
0,378 212
0,356 21.544
0.393 siiied
0.379 #5.4o6
0.392 27.ed
0.361 21.2e-5
0,362 #7.65
AstraZeneca
0,266 22.00
0.227 abuted
0.272 2436-3
0.267 a2.1o3
0.269 a1.e
0.269 a2.1e-4
0,271 a6/03
Ran-Pro
OT-PCA
(E) OT-Embed
1.00
0842
1.00
034
ESOL
EPG
0:31
Cassi
0.33
ESCL
Coate Seitaty
Collne Smlknt
Oolite Fraato
2,50
0.75
0.15
000
.001
Michod ID (a)
Muthod ID
Mull oD
Mothod ID
(c)
(d)
Figure (a) and (b) illustrate the mean cosine similarity of ICRL representations of different molecules in different ICRL methods. (c) and (d) present the mean pairwise cosine similarity between the ICRL representations and their corresponding SMILES text embeddings.
both textual and representational inputs were considered, longer representations contributed deeper understanding of SMILES sequences to some extent. Additionally, for embedding-level injec- tion methods, the length of the injected item remains fixed, making this parameter only marginally influential on the target distribution. Detailed results and analyses are provided in Appendix F.10.
Do key ICL parameters affect ICRL similarly? ICL has been shown to benefit from more examples and batch query processing [23]. This raises a key question: does ICRL exhibit similar behavior? To investigate, conduct ablation experiments these parameters, leading the following findings:
(1) ICRL benefits from an increased number of examples. As illustrated in Figs. 3(c) and 3(f),
OT-based methods demonstrate a noticeable upward trend with an increasing number of examples, suggesting that LLMs effectively interpret injected representations and leverage additional examples to enhance the performance downstream tasks. In contrast, unadjusted representations show limited gains. Since ICRL requires significantly less context space, this finding suggests that embedding-level injection methods have the potential to achieve comparable performance to standard ICL by using a larger number of examples.
(2) ICRL behaves similarly to ICL. Figs. 3(c) to 3(h) illustrate the performance trends when incorporating both textual input and corresponding representations compared to using textual input alone under varying conditions. Specifically, both approaches demonstrate synchronized performance changes with increasing batch query sizes, and the number of examples grows, consistent improve- ments are observed across all methods. These results indicate that the inclusion of representations does not alter the overall trends observed in textual-only approaches. Instead, it generally enhances the LLM's capability to interpret SMILES sequences within the provided examples.
5 What are the Mechanisms Behind ICRL?
In this section, we further analyze and discuss ICRL's distinct behaviors based on several observations:
(I) High similarity between ICRL representations degrades performance. Specifically, the FM features are typically confined to a narrower space [31], leading to highly similar mapped embeddings (Figs. 4(a) and 4(b)). Thus, the LLM perceives minimal differences between samples, lcading to

---

random predictions based on the example label distribution, as further analyzed in Appendix F. However, representation similarity alone does not fully explain the results. For instance, methods like Random Noise exhibit poor performance despite their low similarity, whereas OTT-based methods, which demonstrate moderate diversity, achieve superior outcomes. This underscores the importance of representation-text alignment in effective utilization.
(2) Improved FM features do not necessarily lead to better improvement when text features are included. Interestingly, although the OT-based method shows the best ICRL performance, it does not consistently achieve optimal results when integrated with textual features. Instead, simpler methods often provide more effective enhancements when combined with text input, as demonstrated in Table 2. An analysis of attention weights (Fig. 5) reveals that the model predominantly focuses on the SMILES sequences. which are more familiar and occupy a significantly larger portion of the context window compared to the injected representations. This suggests that the L.LM tends to ignore the injected representations rather than actively learning from them.
The dual operational modes of ICRL representations in different scenarios. When prompts consist solely of ICRL representations as input features, the demonstrations differ significantly from the data encountered during pre-training. As result, the model operates primarily in a task learning mode [33]. relying heavily on the few-shot ICRL exemplars for prediction of test samples. This mode poses a challenge when the few-shot examples are insufficient, making both the prediction and the representation interpretation difficult. As shown in Figs, 3(e) and 3(f), when fewer than ten examples are provided, most methods yield nearly random predictions. However, as the number of examples increases, ICRI. performance improves significantly, indicating that the model gradually acquires the ability to interpret and leverage the injected representations when provided with sufficient contextual samples [1]. In this case, ICRL performance depends on two critical factors: (i) the diversity of representations, which enables meaningful mappings to labels, and (ii) the alignment of projected FM representations with the LLM's embedding distribution. Both conditions are essential for the model to effectively utilize the injected features and function optimally in the task learning mode.
In contrast, when examples incorporate both representations and text inputs, the model may shift towards a task retrieval mode [33] due to the presence of SMILES strings in the LLM's pretraining data. In this mode, predictions are guided by the interaction between the model's pre-training priors and the contextual examples, allowing it to effectively leverage prior knowledge when encountering familiar information [33]. This explains the inferior performance of the PCA+ICI method: the injected representations, being textual tokens, interfere with the model's s interpretation of other text inputs, such as SMILES strings [15]. Conversely, injecting more distinctive embedding-levee features, such as random noise, may function similarly to a pause token, allowing additional "thoughts" that improve performance [17]. These findings underscore the importance of the uniqueness of injected representations. By aligning more effectively with the model's retrieval mechanism, distinct representations enhance the model's 's ability to leverage its prior knowledge and improve performance.
Related Work
Representation-basee In-context Learning, Recent work shows that LLMs can reorganize internal representations to capture task semantics purely from in-context examples [40]. Most related to our work, Vector-ICL [71], designs pre-training and finetuning train projectors to align external models' representations into the LLM embedding space. In contrast, our proposed approach aims to study training-free free methods to derive the projector to avoid costly supervised training.
In-context Learning& & Multi-modal LLMs. ICL is a notable emergent property of LLMs [56], enabling them to perform tasks by conditioning on a few examples without requiring parameter updates [12]. MLLMS extend LLMs by integrating multiple data modalities [28, 65, 42], allowing for more comprehensive reasoning. The Appendix B provides more detailed related work.
7 Further Diseussions
Performance-Cost Trade-off. To clarify the trade-off between accuracy and efficiency, we compare ICRL with representative fine-tuning pipelines, including instruction-tuning (I-FT), supervised pretraining (S-PT)+ + finetuning, and unsupervised pretraining (PT) + finetuning. While these methods achieve good performance, they typically require hours to weeks of GPU training. contrast, ICRI. is training-free free and requires only a lightweight CPU alignment step of about 2 seconds.

---

Table 3: Performance- cost comparison between ICRL and recent fine-tuning pipelines.
Method
Type
Resource
Training Time
ESOL (RMSE)
Lipo (RMSE)
Avg.
MolecularGPT [36]
I-FT
4xA800-80G
<I day
1.471
1.157
1.314
GIMLET [67]
S-PT FT
2-4 GPUs
~1 day
1.132
1.345
1.239
SELFormer [64]
PT
2xA5000
~2 weeks
1.357
3.192
2.275
FT
2xA5000
~2 weeks
0.682
1,005
0.844
GPT-MolBERTa [5]
FT+
2-4 GPUs
2/2 weeks
0.477=10.01
0.758=0.001
0.612
OT-PCA (ours)
Training-free Training-free
only
~/2 see
1.140=0.001
1.349=0.01
1.245
OT-PCA ICL (ours)
CPU only
~2 sec
1.09410.01
1.277=0.01
1.186
As shown in Table 3, while large-scale PT+FT methods achieve the lowest RMSE, they require substantial training cost. In contrast, ICRL runs in only a few seconds on CPU and still matches or surpasses some lightweight tuning pipelines. (Detailed setup are provided in Appendix F.I.)
Lightweight Trainable Projectors. We also investigate whether lightweight training can enhance projector performance by exploring three strategies: caption-based pretraining on LPM24, contrastive alignment between SMILES and FM embeddings, and a combined multi-task variant. While these methods can improve captioning quality on the pretraining domain, they fail to transfer effectively to regression tasks such as ESOL and AqSolDB. some cases, training even destabilizes the ICL process, leading to incomplete outputs or degraded accuracy. These findings indicate that, under limited data and without domain-specific supervision, lightweight trainable projectors are less reliable than the training-fre free OT-PCA, underscoring the robustness and practicality of our approach. (Implementation details and results are given in Appendix F.2.)
Performance under Challenging Tasks. Tasks such drug-target interaction and protein- rreaaee property prediction are particularly challenging for ICI., where subtle sequence variations critically determine labels but remain hard for general-purpost LLMS to capture [2, 26], often leading to near-random performance. ICRL mitigates this limitation by injecting representation- based inputs, allowing the model to capture informative signals even when textual features are homogeneous.
Our additional experiments on molecular QA and caption generation confirm this trend: ICRL consistently outperforms ICL under difficult tasks, yet it still falls short of domain-specific expert models. This highlights ICRL as practical and lightweight alternative when ICL struggles, while leaving room for future integration with specialized approaches (Appendix F.3).
Generalizability. ICRL provides lightweight framework for enabling LLMs to process non-text modalities such as vision and audio without requiring paired data or modality- specific training. Even in the absence of supervision, OT-aligned embeddings from models like ViT and wav2vec2 support meaningful few-shot predictions (Appendix F.4). While ICRL does reach the performance of resource-intensive supervised methods domains where large-scale task-specific or multimodal LLMs already exist, it serves as practical alternative in scenarios where such models are scarce, costly, or difficult to deploy. Representative examples include molecular property prediction [59], sensor- based human activity recogmition [30], and biomedical applications such as protein -igein binding [45], where pretrained multimodal LLMs are not readily available.
In these settings, the lack of modality-speciinn pretrained models, limited labeled data, and domain complexity often make end-to-end fine-tuning infeasible. By directly injecting representation-llvel features, ICRL enables L.LMs to exploit modality-speciife information with no additional supervision or architectural modification. Importantly, our findings across molecular and protein datasets confirm that the link between representation diversity and downstream performance is modality- underscoring ICRL's potential general approach for extending LLMs beyond text (Appendix F.7).
Conclusion
We propose ICRL explore whether LLMs leverage moddaliy-sppeciff FM features without supervised training. Results show its feasibility and potential for generalizable multimodal reasoning.
Limitations and Future Work. Despite its efficiency and generalizability. ICRL underperforms compared to supervised methods due to the lack of task-specific training. In future work, we plan to explore lightweight training strategies to improve its performance while preserving efficiency.
10

---

Acknowledgement
This research is supported by the National Research Foundation, Singapore under its National Large Language Models Funding Initiative (AISG Award No: DSG-NNMLP20024-00 NTU Start Up Grant and by the Ministry of Education, Singapore, under its Academic Research Fund Tier (RG22/24). Any opinions, findings and conclusions recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore. The computational work for this article was partially performed on resources of the National Supercomputing Centre (NSCC), Singapore httpss//www.nssccss)
References
[1] Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning, 2024.
[2] Microsoft Research A14Science and Microsoft Azure Quantum. The impact of large language models on scientific discovery: a preliminary study using gpt-4, 2023.
[3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 11223716-23766 2022.
[4] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023.
[5] Suryanarayanan Balaji, Rishikesh Magar, Yayati Jadhav, and Amir Barati Farimani. Gpt- molberta: Gpt molecular features language model for molecular property prediction, 2023.
[6] Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Chemcrow: Augmenting large-language models with chemistry tools, 2023.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33: 18771971,,,,, 2020.
[8] Juan Manuel Zambrano Chaves, Eric Wang, Tao Tu, Eeshit Dhaval Vaishnav, Byron Lee, S. Sara Mahdavi, Christopher Semturs, David Fleet, Vivek Natarajan, and Shekoofeh Azizi. Tx-llm: A large language model for therapeutics, 2024.
[9] Jiuhai Chen, Jianwei Yang, Haiping Wu, Dianqi Li, Jianfeng Gao, Tianyi Zhou, and Bin Xiao. Florence-vit vl: Enhancing vision-language models with generative vision encoder and depth-breadth fusion, 2024.
[10] Wenliang Dai, Jumnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purposs vision-language models with instruction tuning, 2023.
[11] John S Delaney. Esol: estimating aqueous solubility directly from molecular structure. Journal of chemical information and computer sciences, 44(3):1000-1005, 2004.
[12] Qingxiu Dong. Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in-context learning, 2024.
[13] Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. Translation between molecules and natural language, 2022.

---

[14] Carl Edwards, Qingyun Wang. Lawrence Zhao, and Heng Ji. L+M-24: Building dataset for LanguggeMMoleclees @ ACL 2024. In Carl Edwards, Qingyun Wang, Manling Li, Lawrence 7hao, Tom Hope, and Heng Ji, editors, Proceedings of the 1st Workshop Language Molecules (L++M 2024), pages 1-9, Bangkok, Thailand, August 2024. Association for Computa- tional Linguistics.
[15] Hongfu Gao, Feipeng Zhang, Wenyu Jiang, Jun Shu, Feng Zheng, and Hongxin Wei. On the noise robustness of in-context learning for text generation, 2024.
[16] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics, 2010.
[17] Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens, 2024.
[18] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, and et al. The llaaa 3 herd of models, 2024.
[19] Michael Hahn and Navin Goyal. A theory of emergent in-context learning as implicit structure induction, 2023.
[20] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015.
[21] Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovee, Connor W. Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development, 2021.
[22] Kexin Huang, Tianfan Fu, Lucas M Glass, Marinka Zitnik, Cao Xiao, and Jimeng Sun. Deep- purpose: deep learning library for drug-target interaction prediction. Bioinformatics, 36(22- 23):5545-5547, 2020.
[23] Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan Chen, and Andrew Y. Ng. Many-shot in-context learning in multimodal foundation models, 2024.
[24] Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik Siu Chan, Xubai Xu, Daniel McDuff, Hyeon- hoon Lee, Marzych Ghassemi, Cynthia Breazeal, and Hae Won Park. Mdagents: An adaptive collaboration of Ilms for medical bocision-making. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[25] Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik Siu Chan, Xuhai Xu, Daniel McDuff, Hyeon- hoon Lee, Marzyeh Ghassemi, Cynthia Breazeal, and Hae Won Park. Mdagents: An adaptive collaboration of 11ms for medical decision-making 2024.
[26] Young Su Ko, Jonathan Parkinson, and Wei Wang. Benchmarking text-integrated protein language model embeddings and embedding fusion on diverse downstream tasks. bioRxiv, 2024.
[27] Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating images with multimodal language models, 2023.
[28] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and-vision assistant for biomedicine in one day, 2023.
[29] Zechen Li, Shohreh Deldari, Linyao Chen, Hao Xue, and Flora D. Salim. Sensorllm: Aligning large language models with motion sensors for human activity recognition, 2024.
[30] Zechen Li. Shohreh Deldari, Linyao Chen, Hao Xue, and Flora D. Salim. Sensorllm: Aligning large language models with motion sensors for human activity recognition, 2025.
[31] Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning, 2022.
12

---

[32] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi Tom Sereu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv, 2022:500902, 2022.
[33] Ziqian Lin and Kangwook Lee. Dual operating modes of in-context learning, 2024.
[34] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
[35] Tiqing Liu, Yuhmei Lin, Xin Wen, Robert N Jorissen, and Michael K Gilson. Bindingdb: web-accessible database of experimentally determined protein-ligand binding affinities. Nucleic 2007.
acids research, 3S(suppl_)))DD18-DD22)
[36] Yuyan Liu, Sirui Ding, Sheng Zhou, Wenqi Fan, and Qiaoyu Tan. Moleculargpt: Open large language model (Ilm) for few-shot molecular property prediction, 2024.
[37] Quanyu Long, Wenya Wang, and Sinno Jialin Pan. Adapt in contexts: Retrieval -ugmentte domain adaptation via in-context learning, 2023.
[38] Xingyu Lu, He Cao, Zijing Liu, Shengyuan Bai, Leqing Chen, Yuan Yao. Hai-Tao Zheng, and Yu Li. Moleculeqa: dataset to evaluate factual accuracy in molecular comprehension, 2024.
[39] Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, 2025.
[40] Core Francisco Park, Andrew Lee, Ekdeep Singh Lubana, Yongyi Yang. Maya Okawa, Kento Nishi, Martin Wattenberg, and Hidenori Tanaka. Iclr: In-context learning of representations. 2025.
[41] Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-rong Wen. Tool learning with large language models: survey. Frontiers of Computer Science, 19(8), January 2025.
[42] Owen Queen, Yepeng Huang, Robert Calef, Valentina Giunchiglia, Tianlong Chen, George Dasoulas, LeAnn Tai, Yasha Ektefaie, Ayush Noori, Joseph Brown, Tom Cobley, Karin Hrovatin, Tom Hartvigsen, Fabian J. Theis, Bradley Pentelute, Vikram Khurana, Manolis Kellis, and Marinka Zitmik. Procyon: A multimodal foundation model for protein phenotypes. bioRxiv, 2024.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language supervision, 2021.
[44] Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape. Advances in neural information processing systems, 32, 2019.
[45] Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, and Yun S. Song. Evaluating protein transfer learning with tape, 2019.
[46] Gabriel J Rocklin, Tamuka M Chidyausiku, Inna Goreshnik, Alex Ford, Scott Houliston, Alexander Lemak, Lauren Carter, Rashmi Ravichandran, Vikram K Mulligan, Aaron Chevalier, al. Global analysis of protein folding using massively parallel design, synthesis, and testing. 2017
Science, 357(6347):16--1
[47] Karen S Sarkisyan, Dmitry A Bolotin, Margarita Meer, Dinara R Usmanova, Alexander Mishin, George V Sharonov, Dmitry N Ivankov, Nina G Bozhanova, Mikhail S Baranov, Onuralp Soylemez, et al. Local fitness landscape of the green fluorescent protein. Nature, 533(7603).377-401, 2016.
[48] Shannon Zejiang Shen, Hunter Lang, Bailin Wang, Yoon Kim, and David Sontag. Learning to decode collaboratively with multiple language models, 2024.
[49] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023.
13

---

[50] Seongjin Shin, Sang- Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung-Wac Woo Ha, et al. On the effect of pretraining corpora on in-context learning by a large-scale language model. arXiv preprint arXiv:2204. 13509, 2022.
[51] Oscar Skean, Md Rifat Arefin, Yann LeCun, and Ravid Shwartz-Ziv. Does representation matter? exploring intermediate layers in large language models, 2024.
[52] Murat Cihan Sorkun, Abhishek Khetan, and Suleyman Er. Aqsoldb, a curated reference set of aqueous solubility and 2d descriptors for diverse set of compounds. Scientific data, 6(1):143, 2019.
[53] Luis Caicedo Torres, Luiz Manella Pereira, and M. Hadi Amini A survey on optimal transport for machine learning: Theory and applications. 2021.
[54] Ning-Ning Wang. Jie Dong, Yin-Hua Deng, Min-Feng Zhu, Ming Wen, Zhi-Jiang Yao, Ai-Ping Lu, Jian-Bing Wang, and Dong-Sheng Cao. Adme properties evaluation in drug discovery: prediction of caco-2 cell permeability using a combination of nsga-ii and boosting. Journal of chemical information and modeling, 56(4):763-773, 2016.
[55] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 2021.
[56] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022.
[57] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.
[58] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(2):113 530, 2018.
[59] Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg. Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chem. Sci., 9:513-530, 2018
[60] Tong Yang, Yu Huang, Yingbin Liang, and Yuejie Chi. In-context learning with representations: Contextual generalization of trained transformers. arXiv preprint arXiv:2408. 10147, 2024.
[61] Tong Yang, Yu Huang, Yingbin Liang, and Yuejie Chi. In-context learning with representations: Contextual generalization of trained transformers, 2024.
[62] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023.
[63] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 11(12), November 2024.
[64] Atakan Yüksel, Erva Ulusoy, Atabey Ünlü, and Tunca Dogan. Selformer: Molecular represen- tation learning via selfies language models, 2023.
[65] Ruohong Zhang, Liangke Gui, Zhiqing Sun. Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, and Yiming Yang- Direct preference optimization of video large multimodal models from language model reward, 2024.
[66] Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. arXiv preprint 203.06200.. 2022.
14

---

[67] Haiteng Zhao, Shengehao Liu, Chang Ma, Hannan Xu, Jie Fu, Zhi-Hong Deng, Lingpeng Kong, and Qi Liu. Gimlet: A unified graph-text model for instruction-base molecule zero-shot learning, 2023.
[68] Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. In International Conference on Learning Representations, 2023.
[69] Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. In The Eleventh International Conference Learning Representations, 2023.
[70] Hao Zhu, Todd M Martin, Lin Ye, Alexander Sedykh, Douglas M Young, and Alexander Tropsha. Quantitative structure- activity relationship modeling of rat acute toxicity by oral exposure. Chemical research in toxicology, 22(12):1913 1921, 2009.
[71] Yufan Zhuang, Chandan Singh, Liyuan Liu, Jingbo Shang. and Jianfeng Gao. Vector-icl: In-context learning with continuous vector representations, 2025.
15

---

Appendix Contents
1. Author Contributions
16
2. Detailed Related Work
17
3. Datasets Details
17
(a) Molecular Dataset Description
17
(b) Protein Dataset Description
4. Algorithmic Descriptions
19
(a) ICRL Framework
19
(b) PCA Method
19
(c) Zero-Pad
19
(d) Random Noise
20
(e) Optimal Transport Alignment
20
5. Theoretical Analysis
21
(a) Proof of Theorem
21
(b) Proof of Theorem 2
22
(c) Proof of Corollary
.23
6. Additional Experiments
25
(a) Performance- Effiiiiincee Tradc-offs
(b) Lightweight Learnable Projectors
(c) Cross-Tasks Generalizability
26
(d) Modal Generalizability
26
(e) Visualization Attentional Weights
27
(f) Feature Extraction from Different FM Layers
28
(g) Experiments on Protein and DTI Tasks
28
(h) Results with
29
(i Generalization Across LLMs
30
PCA Ablation Analysis
31
(K) Representation Similarity Analysis
7. More Experimental Details
32
(a) Experimental Setup
32
(b) In-Context Example Sampling
32
(c) Metric Discrepancies
A Author Contributions
Tianle Zhang:
Led the project; designed and implemented various representation-llvel ICRL variants; conducted most experiments and analysis; wrote the main manuscript.
Wanlong Fang: Assisted with vision-related experiments; contributed to project discussions and paper revisions.
Jonathan Woo: Designed and implemented text-level ICRL, variants; developed the initial project codebase; contributed to project discussions and paper revisions; assisted in designing the thcoretical analysis framework.
Paridhi Latawa: Prepared and processed protein-related datasets; contributed to project discussions and paper revisions.
Deepak A. Subramanian: Prepared and processed meall-molecule-relian datasets; contributed to project discussions and paper revisions.
Alvin Chan: Provided overall supervision and guidance throughout the project; advised on the ICRI. framework; contributed to manuscript writing.
16

---

B Detailed Related work
In-Context Learning (ICL): ICL remarkable emergent property of LLMs, enabling them to perform various tasks by conditioning on a few input-output examples, without requiring parameter updates or fine-tuning [7, 55]. It has been explored extensively in terms of studying factors that influence the performance of ICL such as prompting strategies, amount and order of few-shot exemplars[50, 23, 19, 12]. However, it remains unknown how we can apply in-context learning to integrate high-dimensionall representations from external non-LLM models. Our work is pioneering its focus on enabling LLMs to directly learn from and utilize representations derived from diverse modalities beyond text from external models. This novel approach has the potential to radically expand the utility of LLMs in domains requiring the integration of multiple modalities and knowledge, thus opening new avenues for the application of LLMs. In recent work [60], it has been shown that LLMs can shift from their original semantic representations to new ones that aligned with the graph's structure given as few-shot learning examples in the text prompt, suggesting that scaling context can enable LLMs to reorganize their knowledge to accommodate novel contexts and tasks. In contrast, our work here focuses on the potential to integrate representations from external models during inference time through few-shot exemplars.
Foundation Models (FMs): Foundation Models are large-scale, pre-trained deep learning models designed to generalize across diverse tasks by learning from massive datasets using self-supervised learning techniques. These models, such GPT3 [43, 7]. leverage billions of parameters develop general-purpose representations that can be fine-tuned for specific tasks or domains, revolutionizing fields like natural language processing, computer vision, and biomedical research [7]. For molecular and protein analysis, structure-based FMs, such as GearNet [66]. Uni-Mol [69], and ESMFold [32] extend this paradigm by incorporating dopain-specific 2D/3D molecular representations. Although user-fite friendle,lly L.MS typically lack specialized training in areas like protein sequences or chemical structures. Such specialized domains FMs offers a way to enable LLMs to efficiently acquire and apply specialized knowledge. Our work use ICL to integrate domain-speciff expertise from FMs pre-trained on specialized datasets, enhancing both usability and computational efficiency.
Tool Use in LLMs: Current research on tool use with LLMs, such as the implementation of APIs for external functionalities in applications like Chemcrow, demonstrates the capacity of LLMs to interface with external tools to perform specific tasks [6. 62, 41]. However, in the current approaches, LLMs utilize only the final outputs of external models the external tool. There is currently no work looking into using the richer, underlying representations from these external models for more sophisticated LLM inference. Our work aims to bridge this gap by enabling LLMs to access and leverage deep learning model representations, enhancing their inferential capabilities to address specialized tasks beyond what the LLMs are trained on.
Multi-modal Large Language Models are designed to process and integrate multiple data types, including images, text, audio, and more [63, 43). Existing research in the realm of MLLMs, such as Disiun-Languuge Models (VLM) [3]am [an Tx-LLM[8], has significantly advanced
Multi-modal Large Language Models (MLLMs):
understanding of how MLLMs can process and integrate information across different input modalities. However, these models typically require extensive supervised training for both the LLM and their projection layers to handle multi-modal tasks [27]. This process not only demands substantial computational resources but also necessitates expertise AI, making it less accessible to researchers without background in machine learning. The requirement for multi-task training means the entire model must be retrained with both old and new data to avoid catastrophic forgetting- significant limitation when only new data is ]9]roduced[[9]. Our work addresses these limitations by proposing novel in-context representation learning framework that simplifies the integration of multi-modal data, thereby reducing the need for extensive computational resources and specialized AI knowledge.
C Datasets Details
C.1 Molecular Dataset Description
Molecular Domain: In the molecular domain, we evaluate the performance of our ICRL framework on seven key molecular property prediction regression tasks. These tasks are taken from Therapeutic
17

---

Data Commons (TDC) Absorption, Distribution, Metabolism, and Excretion (ADME) property prediction benchmarks and the Toxicity property prediction benchmarks.
TDC is a platform consisting of Al-based datasets, benchmarks, and processing tools for therapeutic machine learning [2 The ADME tasks are set of single-instance prediction tasks in the molecular domain that examine the ADME properties of small-molecule drug chemicals. The Toxicity property prediction tasks aim predict drug toxicity properties. All molecular datasets are as follows.
ESOL [[]]: ESOL is made up of water solubility data (log solubility in mols per litre) for common organic small molecules. Input chemical compounds SMILES string, predict the solubility of chemical compounds.
Caco-2 [54]: The human colon epithelial cancer cell line, Caco-2. is used an in vitro model to simulate human intestinal tissue. Experimental measurements of the rate at which drugs pass through Caco-2 cells can approximate the rate of drug permeation through human intestinal tissue. Input a drug SMILES string, predict the Caco-2 cell effective permeability.
AqSolDB [52]: Aqueous solubility measures a drug's 's ability to dissolve in water. Poor water solubility can lead to slow drug absorption, inadequate bioavailability, and even toxicity. More than 40% of new chemical entities are poorly soluble. Input a drug SMILES string, predict the activity of solubility.
LD50Zhu [70]: Acute toxicity LD50 measures the most conservative dose that can lead to lethal adverse effects. The higher the dose, the more lethal of drug. Input a drug SMILES string, predict its acute toxicity.
AstraZeneca [58]: Lipophilicity measures drug's ability to dissolve in a lipid environment (e.g., fats, oils). High lipophilicity is often associated with high rate of metabolism, poor solubility, rapid turnover, and low absorption. Input drug SMILES string, predict the activity of lipophilicity.
C.2 Protein Dataset Description
In the protein domain, we evaluate the performance of the ICRL framework on four key protein property prediction regression tasks. These tasks are taken from two key protein representation learning benchmarks: TAPE (Tasks Assessing Protein Embeddings) and PEER (Protein Sequence Understanding).
Protein Domain:
The TAPE benchmark consists of five tasks spanning different domains of protein biology [44]. We focused on the Fluorescence and Stability tasks because they are whole protein-sequeenc--leell regression tasks, in contrast to amino acid-level regression, pairwise amino acid regression, or classification. The Fluorescence dataset and Stability dataset descriptions are as follows:
Fluorescence: The Fluorescence task is regression-based task that predicts the log- fluorescence intensity of an input protein [47]. The train set consists of a small neighborhood of the GFP protein, and the test set has distant GFP proteins.
Stability: The Stability task is regression task that measures the most extreme circum- stances in which protein maintains its folded state above a concentration threshold, consid- ered a proxy for stability [46]. The train set comprises a broad set of proteins, and the test set comprises one-mutation neighborhoods of sampled proteins.
Drug-Targer Interaction Domain:
In the Drug-Target Interaction (DTI) domain, we evaluate the performance of our framework on two key molecular-proter cross-domain property prediction regression tasks. These tasks are taken from the TDC Multi-Instance Prediction Problem. The drug-target interaction (DTI) prediction task evaluates the interaction activity of small-molecule drug compounds.
BindingDB[35, 22] is collection of various assays which is a part of the DTI prediction task, which takes in a target amino acid sequence and SMILES string and then predicts the binding affinity. Since different assays use different metrics, two separate BindingDB datasets in Ki and IC50 units were used.
BindingDB Ki is database containing experimental data on the binding affinity (Ki values) of small molecules to protein targets.
18

---

BindingDB IC30 is publicly available database containing experimental data on the binding affinities of small molecules to proteins, specifically in terms of IC50 values. IC50 measures the concentration of a substance required to inhibit 50% of biological target.
D Algorithmic descriptions
this section, we provide a detailed description of the overall pipeline and all representation injection algorithms.
Algorithm ICRL Framework
1: Input: Training and test dataset Qtrain and Qrest, FM Φ, LLM the, batch query size b, injection strategy Manj- number of examples k, empty example set E
2: Stage I: Construct demonstration set
Extract representations from FM: H ((Qtroén))
4: if' Many requires additional parameters then
5: Derive parameters from 1 _ Qtrain and H
6: end if
7: Sample {(c;. (4) )= from Qtrain
8: for each sampled (Ia, Vi, ha) do
9: Apply injection strategy: r, Minj (h+)
10: Add example (ri; M)) to E
end for
12: Stage 2: Batch Inference
for each test batch B, {( yj (1))) S Qtest do
14: Initialize empty batch queries Qj
15:
for each sample / = to do
16:
Extract representation: h
(c) 0 (z(z(()
Apply injection strategy: r Min (0)
17:
18: Add query (r)()) to Qj
19:
end for
20: Generate predictions: Y, = 10, 16 = (EE
end for
22:
Output: Predictions Y.
Algorithm 2 PCA method
1: Input: Extracted representations H, PCA model W PCA fitted by Qerain and string conversion
function S
2: Compute peduced-dimensioon embeddings: H = H x PPCA
3: Convent reduced embeddings into text strings: Spea S(H (ca)))
4: Output: S pca
Algorithm 3 Zero-Pad
Input: Extracted representations H. LLM embedding space dimension dLEM
2: Compute padding size: pad_size = dLLM - dFM
3: for each h; in H do
4: Apply zero padding to H to match the LLM embedding dimension: hipadded Pad(h; pad pizesize)
5: end for
6: Output: Hpadded
19

---

Algorithm 4 Random Noise
1: Input: Training and test dataset Qirain LLM dee. LLM embedding dimension dLLM
2: for each z; in Qtrain do
3: Compute random hash seed using d4e hese Hash(
4: Generate a unique random vector for each input based on the hash seed: h; Random(hisacaff dLLM)
5: end for
6: Output: Hnoise
Algorithm 5 Optimal Transport Alignment
1: Input: Extracted and projected representations H and H proj. input texts Qtrain LLM em- bedding function th. alignment mode mode, PCA model WWPC fitted by Qtrain and string conversion function S
2: Stage Get the OT parameters
3: if mode: "embed then
4: Compute input embeddings: Haar vhe (Qtrain)
5: else if mode "pea" then
6: Get PCA strings: Spea = S(H X WCC)
7: Compute PCA embeddings: Haar (Spoa)
8: end if
9: Initialize empty vectors shift, scale E RdI.
10: for each dimension j do
11: ū mean H proj [:. jl), v, mean He (:,5])
12: Jp.j = std
std H proj F. 21), ot.j = std(H (.))):
13: shiftt]j
UJ scale(j] Tt. /ppj
14: end for
15: Stage 2: Implement alignment
16: Haligned = scale Hproj + shift
return Haligned, shi ft, scale
20

---

E
Detailed Proofs
In this section, provide complete derivation of the concentration result stated in Theorem 1 and Theorem 2. For convenience, we restate the theorem and then present the step-by-step analysis.
E.1 Proof of Theorem
Given fixed vector E Rd, suppose W E Raxd has i.i.d. entries WE.1 (0, 1/d), and b E Rd is bias vector (fixed or random and independent of W). Then there exist small constants E1, 81 such that, with probability at least ò1.
theorem:
Wu - bll² [lbb] + [|u|])] < e [|b]]++||uu)
+ 11
(8)
E.1.1 Step 1: Coordinate-wise Distribution
Consider the affine transformation x + Wx + b. For each coordinate & f1,2..... .d],
(Wu b) = Σ Wxe ue + by.
(9)
Because Wa ~ (O0, 1/d) are ii.dd, the sum Et= Wa is Gaussian with mean 0 and variance
Hence each coordinate (Wu + b)д is distributed as
Y2 = (Wu- + b)2
N
b
1
(10)
E.1.2 Step 2: Expectation and First Moments of the Squared Norm
Let Z = Wu + b112 Eim Yg since Y ~ N(bs, [u]li" for each k; we have B[ =
b +
||l| summation over yields:
|u|
E Z =
b
d = -||l + ||u||
(11)
Denote this sum by
[bb| + ||u|?
(12)
our task is to show that Z remains close to M with high probability as d grows.
E.1.3 Step 3: Concentration of the Squared Norm (Refined Analysis)
We now give more precise argument about the variance of Z and how it influences the bound from Chebyshev's 's inequality.
3.1. Exact Variance Computation. Since z= Dd Y² and each Y, is independent across k:
Var(Z)= Σd Eb=1 (Yr For Y₁ NV 2²) with μt by and の² ||u||// /d, one has
Var(Y?) B|Y! (El 2 2 4 1 o²
(13)
Substituting μı by and -² ||ul///d, we get
Var(Y |1ا|
+
|
(bi)-
(14)
Summing over k from to d,
d
uu|
u
2
2 2|u||
4
||u||
|bb|
Var(Z)= Σ
4
by
(15)
d2
d
d
Thus we have:
2||u|/ 4
+
Var(Z)
.E[Z] M
||u|2
+ ||b|
(16)
21

---

3.2. Ratio of Variance to Square of the Mean.
To apply Chebyshev precisely, we look at
Var [Z)
2 |||| + | |b||
(17)
[EZ)
d d ||
Provided ||u|| and [|b]] do not scale with d, this ratio shrinks the order of 1/d. Thus, the relative standard deviation / (/rr())EE)) behaves like 1/vd. Hence, + oo, Z concentrates around its mean M.
3.3. Chebyshev's Inequality for c,, 6, From Chebyshev's inequality:
(Z)
Pr Pr( Z
E
a)
(18)
a
Set a = 61 (I|u|2 + |[b]|2] c1 M. Then
Pr(zz M 2 ci M)
Var(Z)
||
4 |
b
Cu.b
(19)
Ef M
u
|b]
dep
where
||||| + 4 |
G.
(20)
(||u 2 + bb|
Hence, for any fixed E,, the failure probability 8, satisfies
51 = P -M| 2 < M) v
Cu.b deñ
(21)
As d increases, 81 - 0. This establishes high probabbbbbyiiit guarantee of the form
Pr (llwu b1(2 (||u||2 |b| S a (1uul22 + [b](²) 2 1 - 1161,
(22)
with explicit constants that depend on ||u||. ||b]|. and d.
3.4. Uniform Dependence on ||u|| and [[b]]- [u| or ||b]] grows with d, the coefficient G. might also grow, thus weakening the rate at which 8 decays. However, if |uu| and ||b]] remain bounded independently of d, then G.,b is constant, and the probability failure vanishes on the order of //d. This completes our proof of Theorem
Remarks.
The key structural property is that W W Las similar statements hold under any isotropic, sub-Gaussian distribution for W. This also explains why the normal initialization achieves the best performance, since it is the that best fits the theoretical assumptions.
If b itself random and independent W, conditioning on one and integrating out the other yields nearly identical analysis; the main requirement is that b not be too large or adversarially correlated with W, e.g., in practice, we avoid this problem by setting directly to O.
E.2 Proof of Theorem 2
theorem: Let u, E R&d be any two fixed vectors, and W E Rdxd have i.i.d. entries We.r A/ 1/d). Define u' = Wu. V' = Wy. Then there exist constants 6 82 > O such that, with probability least 1 52.
coos(u', v²) cos(u, v) <
(23)
Equivalently,
cos(Wu. Wv) x cos(u, v) with high probability.
(24)
22

---

E.2.1 Step 1: Concentration of norms.
By Theorem 1, for each of the vectors u and v, their images under x + Wx satisfy
|Wul2
~
|ull
[[v]]2 ~ lIvll?
(25)
More precisely, there exist 61, 51
> such that with probability at least 81.
Wull ||u|
S < ||ll?
||v]]] ||v|9| S cillv11.
(26)
Taking square roots and using standard bounds yields
|ul| ||u]]
||v||| = [[v]| ~ [[v|l
(27)
E.2.2 Step 2: Concentration of the dot product.
Consider (Wu) = u² v², we have:
u v2 (Wu)
(28)
Since W has mean-zero i.i.d. Gaussian entries, we have:
B[(Wu) ((w)] v'v.
(29)
By concentration incqualitics for sums of Gaussians, there exist eg 8/ > such that with probability
at least 1 - 82.
(Wa) (Wv) u u'vl S e/ | [|]]-
(30)
E.2.3 Step 3: Combining the bounds to preserve cosine similarity.
T. For u² and v", we have cus" v²) ||||||||*||
From (27) and (30), with high probability, we have:
u' u'u v. |u'll
||u|]
||v'||
= ||
(31)
Thus we have:
uTv ||
cos(u' (u', v²)
cos(u, v).
(32)
||u'|l
Therefore,
|oos( Wu, Wv) cos(u, v) S for some 62 0, with probability least 1 52, where 82 = ōn 82. thus complete the proof.
(33)
E.3 Proof of Corollary 1
Below, we present a more formulaic argument showing that a random Gaussian linear map x + Wx followed by a standard nonlinear activation : IR IR (e.g.. ReLU, leaky ReLU, sigmoid, etc.) leads to larger angle distortion than the purely linear case. In other words, with high probability, the angle distortion 6 for nonlinearly transformed embeddings is larger than the purely linear 62,
Loordinate-Level Analysis. Consider a single coordinate, 21 (Wu) 22 (Wv).. The original product 21 22, whereas after applying a we have (21) (22). If a compresses negative values more strongly than positive ones (e.g., ReLU, sigmoid). otherwise saturates different portions of the real line, then (221) (22) can deviate significantly from 21 22.
Taking ReLU as example, the 22 and (=)) or (22) will differ by
A max 21, 0} max (22 0}
21 22-
(34)
If 21 and 2n have opposite signs, A > 0; this pushes the new vectors into the positive orthant and tends align them more.
Closed-Form for Correlated Gaussians. Due the properties of random Gaussian matrices, for any coordinate k. (21, 22) follows bivariate normal with correlation 0, i.c. (Z1, Z2) (O,
23

---

and p 1. Taking ReLU as an example, we can define Z1+ 1) ()
where
max 0} and Z2+ m 0}. thus we have:
B Z2+
Corr(Z1+:200 12+)
(35)
/B|Z B[Z2+]
A standard integral shows E[Z 2 so we can simplify the definition as: Corr(Z1+: Z2+) = 2 [B| Z1+ [1+22+] When 0, symmetry implies E[ [2+] 1. giving correlation of 1.2 As P - I, E[ 22+] + 2. the correlation approaches 1. For
as E|Z Z2+1 increases with p. staying above i and thus making: Corr(Z1+, Z2+) p. As correlation in Rd directly relates to cosine similarity, we conclude that
cos Z1, 0}, max ma, 01) cos (Z,, Z2),
(36)
Lee ReLU pushes eeeeady-lligned (positively correlated) coordinates even closer together, leading to "angle inflation."
Summary and Conclusion (for General a). Because generic activations (e.g. ReLUU, leaky ReLUU, sigmoid, tanh) discard or shrink certain parts of the real line (often negative values), they systematically inflate pairwise similarities in high-dimensional random setting. Therefore, we have
(ou)), (Wv)) cos u, v)] coos( Wu, Wv) - cos(u, u, v)
(37)
meaning the angle distortion 62.a for such non- non-linearities exceeds the purely linear case 62,
More Experiments
F.1 Performance-Effiiienn Trade-offs in ICRL
key motivation behind ICRL, is to explore the feasibility of enabling L.Mss to understand non-text modalities (e.g.. molecular, vision, audio) in training-free manner. This contrasts with prior methods such as Vector-ICL. [71], Florence-VL. [9], and ICL-Reps [61], J. which rely on supervised training or projection tuning. In this section, we provide comprehensive analysis of the trade-off's between performance and efficiency across model scale, input modalities, and computational cost.
Comparison with Similar Methods. As shown in Table 4, ICRL is unique in being both training-fiee and capable of accepting non-text inputs. In contrast, MLLMs (Florence- VL) and Vector-ICI require large-scale supervised training projection tuning. These approaches also consume substantially more resources in terms data and time.
Table 4: Comparative overview of multimodal adaptation methods.
Method
Non-text Input Supervised Training
Data Requirements
Time Cost
MILMM [9]
High
High
Vector-ICI ICL [71]
High
High
ICL-Reps [61]
High
High
DA-ICL [37]
High
High
ICL (text only)
Low
Low
ICRL (Ours)
Low
Low
Cost-Performance Comparison with Fine- Fine-Tuning Tuning Pipelines. We compare ICRL against recent molecular property prediction pipelines, including instruction-tuning via Q-LoRA (I-FT) [36], super- vised pretraining followed by finetuning (S-PT FT) [67], and unsupervised pretraining followed by finetuning (PT + FT) [64, 5]. Evaluation conducted on ESOL and Lipophilicity datasets, with RMSE as the metric. To ensure fairness, we implement ICRL using the LLaMA-3.1 1----3.------nttu-e model-compaarble
inference cost to those used in prior work and feasible on a single GPU. Evaluation is based on RMSE. Notably, ICRL achieves even better performance with larger models: e.g., on ESOL,, it reaches 0.839 RMSE using I.L.MAA3.1-70B. Cost information for baselines extracted from original papers when available.

---

Analysis. As shown in Table 3, under comparable inference cost, ICRL achieves performance comparable to or even superior to lightweight tuning, while requiring no training and only ~2 seconds of CPU computation. Although large-scale PT+FT pipelines (e.g., GPT-MoIBERTa) still achieve the lowest RMSE, their training requires weeks on multiple GPUs, in sharp contrast to the negligible overhead of ICRL. This result highlights ICRL's practicality for low-resource and rapid-deployment scenarios, where retraining is prohibitive.
Improved understanding of small molecules does not necessarily translate to better performance on downstream tasks. In analogy to LLM pretraining, the pretraining in baseline methods primarily intended to enhance molecular understanding. However, as shown in [64], the standalone pretrained model performs significantly worse than ICRL on downstream prediction, suggesting that pretraining alone mainly serves as a warm-up up for subsequent supervised tuning.
Context Window Usage. One of the primary advantages of ICRL is its low context window usage. As summarized in Table 5, standard ICL methods require dozens to hundreds of tokens to represent non-text samples such SMILES strings, amino acid sequences, images, audio. In contrast, all variants of representation--evell ICRL compress the input into a single embedding vector, which is injected as single dramattliyy reducing input length and improving inference efficiency.
Table Approximate per-sample context window usage across methods and modalities.
Method
SMILES (ICL) AA Sequence (ICI.)
Vision (MLLM)
Audio (MLLM)
ICRI. (Ours)
Tokens
6-21
103-214
29-576
75-1500
Inference Overhead. Although ICRL includes additional steps such as PCA and OT alignment, these operations are lightweight. Table 6 shows that the PCA and OT steps collectively take about 2 seconds on CPU-faster than a single forward pass. Importantly, OT alignment only needs to be computed once per domain and reused thereafter via simple matrix multiplication during inference.
Table 6: Comparison of computational cost across adaptation strategies.
Method
Pre-training
Fine-tuning
PCA Step
OT Step
Time Cost
Hours-Days
1.2 sec
0.99 sec None
GPU Requirement
High
Medium
None
Conclusion. While ICRL may not achieve state-of-the-art performance in all tasks, it provides promising direction for enabling lightweight, training-free multimodal reasoning. By drastically reducing context window usage and inference cost, it offers an efficient alternative conventional multimodal systems, especially under low-resource constraints.
F.2 Lightweight Learnable Projectors
To further examine whether lightweight training can improve projector performance, we conduct all projector training on the LPM24 dataset [14]. We design three loss variants: (1) caption- based pretraining, where the projector is optimized to generate molecular descriptions from FM embeddings; (2) contrustive learning, inspired by CLIP [43), which minimizes the distance between LLM hidden states when receiving molecular inputs in SMILES form versus projected FM embeddings; and (3) combined loss that integrates both caption and contrastive objectives. This setup ensures that the L.LM learns to treat projected embeddings semantically consistent with textual SMILES representations. while also testing whether multi-task training enhances generalization.
All experiments use LLaMA-3, 1-8B-Innstre as the base model, with single-layer linear projector of hidden size 4096, consistent with [71]. Performance is evaluated on two tasks: molecular captioning on the LPM24 test set using BLEU-4, ROUGE-1, and ROUGE-L; and molecular property prediction on ESOL and Solubility Suuubllit_AAqSlDBB using RMSE. Other hyperparameters and evaluation protocols follow the main paper to ensure comparability.
Analysis. All three projector-trainie strategies perform worse than the training-free OT-PCA baseline. In regression tasks, they yield higher RMSE, and in some cases also harm ICL behavior
25

---

Table Molecular captioning results with learnable projectors.
Method
BLEU-4
ROUGE-1
ROUGE-L
Caption-only
35.29
0.551
0.373
Contrastive-only
21.42
0.320
0.237
Caption + Contrastive
37.31
0.592
0.369
Table 8: Regression results with learnable projectors.
Method
ESOL (RMSE)
AqSolDB (RMSE)
Caption-only Contrastive-only
1.256
3.030
1.372
2.915
Caption + Contrastive
1.213
3.805
OT-PCA (ours)
1.140
2.411
OT-PCA + ICL (ours)
1.094
2.385
(e.g., incomplete outputs, overfitting). Even when captioning quality improves, this does not transfer to downstream prediction. These results indicate that, under limited data and lack of domain knowledge, lightweight training is unstable and cannot match the robustness of training-fit -frce alignment.
F.3 Cross-Tasks Generalizability ICRL
To examine whether ICRL generalizes beyond regression tasks, we extend evaluation to two additional task types: molecular QA and captioning. For QA. we adopt the MoleculeQA benchmark [38], which contains four categories (Structure, Source, Property, Application), and report accuracy in each as well as the average score. For captioning, we use the ChEBI-20 dataset [13], where performance is measured with BLEU-4, ROUGE-I, I. and ROUGE-L. All experiments are conducted with LLaMA-33 8B-Instruct as the hase model, baseline results the table are taken from [38] and [13], respectively.
Table 9: Molecular QA results on MoleculeQA benchmark.
Method
Source Structure
Property
Application
Avg
Llama-2-7B-chat (L-FT)
28.75
39.84
31.33
27.71
31.54
ICL
35.03
27.04
24.62
28.69
28.85
OT-PCA (ours)
51.32
37.66
33.71
31.02
38.43
OT-PCA ICL (ours)
50.60
43.52
23.47
29.97
36.89
MolT5-base (FT)
58.01
65.85
45.14
42.24
55.39
Analysis. On the QA benchmark, ICRL significantly outperforms both standard ICL and fine-tuned general-purpose LLMs, showing that OT-aligned embeddings can be directly interpreted and utilized by text-based LLM. Notably, embedding-only injection sometimes surpasses the combination with textual features, suggesting that the injected representations can provide clearer and more informative signal than raw SMILES strings for molecular reasoning.
On the captioning task, ICRL also improves semantically relevant outputs, despite the base LLM lacking molecular grounding. However, the performance gap with expert models such MolT5 remains, indicating that while ICRL extends to generative tasks, domain-specific pretraining still advantageous settings requiring highly specialized knowledge.
ICL by enhancing the model's ability generate
F.4 Cross-Modal Generalizability of ICRL
To further examine the generalizability ICRL beyond molecular and protein datasets, we conducted experiments vision and audio datasets, which are more distant from the natural language domain. These modalities do not natively conform sequence-based formats, making ICI, particularly challenging.
26

---

Table 10: Molecular captioning results on ChEBI-20 dataset.
Method
BLEU-4
ROUGE-1
ROUGE-L
ICL
0.133
0.393
0.310
OT-PCA (ours)
0.147
0.353
0.274
OT-PCA + ICL (ours)
0.196
0.407
0.353
MolT5- (FT)
0.457
0.634
0,578
We evaluated ICRL on four datasets: ImageNet and CIFAR-100 (vision), ESC-50 and VGGSound (audio). Representations were extracted using ViT-B/16 for vision and wav2vec2-bars fase2--bae-9001 for audio. Due to computational constraints, we randomly selected 50 classes per dataset and followed the default settings described in the main paper. Because LLaMA-3- 3-703- IOB-Instruct does not support direct image/audio input, standard ICL is not applicable. Hence, we use random guessing as a baseline for comparison. All experiments were repeated 10 times with different random seeds, and we report the top-3 classification accuracy averaged across runs.
As shown in Table 11, naive strategies such as random noise or zero padding performed worse than random guessing, suggesting that LLMs cannot interpret these representations. In contrast, OT- based embeddings consistently outperformed random guessing, demonstrating that such embeddings preserve enough structure be understood by LLMs in training- free manner.
Table 11: Top-3 classification accuracy (%) on vision and audio datasets using LLaMA-based ICRL. OT-based embeddings clearly outperform baselines such as random noise or zero padding, indicating effective cross-modal generalization.
Dataset
Random Guess
Random Noise
Zero Pad
OT-Embed
OT-PCA
ImageNet
2.00
1.01
0.32
14.21
17.21
CIFAR-100
2.00
0.63
0.91
13.73
15.62
ESC-50
2.00
1.39
1.26
17.65
16.75
VGGSound
2.00
0.76
1.44
12.39
18.23
F.5
Visualization of attentional weights
Figs. 5a and 5b present wetett heatmaps for the Ran-Pro+ICL. and ICL methods, using the 20th attention head of the final layer as an example. The results show that the model's attention is predominantly concentrated on the SMILES string, suggesting that the injected representations are the primary focus of the model's learning process. We set the shot count 5 and batch query size to in this experiment.
304
(a)
(b)
Figure 5: (a) and (b) are the attention- heatmaps for the Ran-Pro+ICI and ICL. methods on the ESOL dataset, respectively.
27

---

F.6 Feature extraction for different layers of FM
In this study, we primarily utilize CLS features derived from the FM. However, previous research has shown that using CLS features may not be the optimal choice for improving downstream task performance [5 As shown Table 12, under the OT-PCA method, most representations from other layers are difficult to interpret, possibly due to their higher similarity. Even the OT method struggles to effectively adjust these representations. However, when combined with ICI., using features from shallower layers typically yields better performance, suggesting that these features exhibit lower similarity to textual embeddings and can serve as more effective identity tokens.
In the ablation study of PCA, we observed that increasing the length of injected representations significantly impacts the performance of ICRL. Therefore, we further explored the effect of this factor non-CLS features. As shown in Table 13, increasing the length did not lead to significant performance improvements. This indicates that representation length not a critical factor for embedding- injection methods. Moreover, longer representations may weaken their role as identity tokens, leading to performance degradation when combined with ICI.
Table 12: OT-PCA and OT-PCA+ICL across different layers (one layer) on ESOL. Bold = best value (highest for Pearson/Spearman, lowest for RMSE); Underline second-best.
Method
Layer
Pearson
Spearman
RMSE
0
-0.017 + 1. 1.6e-2
0.010 2.1c-2
1,432 # 1,2c-2
1
0.051 = 4.0e-4
0.102 # 6.2e-3
1,350 # 1.9e-2
OT-PCA
5
0.078 + 3.2c-4
0.148 + 3.7c-4
1.235 + 4.1c-4
10
0.052 + 4.0e-3
0.101 5.9e-3
1.336 4 2.1e-2
-1
0.078 8.6e-4
0.122 2.8e-3
1.220 # 5.7e-3
cls
0.227 # 7.3e-4
0.235 1.5e-4
1,243 + 4.0c-3
0
0.609 + 1.0e-3
0.595 + 2.4e-3
0.889 # 1.3e-3
1
0.586 + 3.7e-3
0.588 = 4.7e-3
0.901 = 1.1c-3
OT-PCA+ICL
5
0.607 + 1.3e-3
0.663 3.9e-3
0.901 # 1.5c-3
10
0.602 # 1.4c-3
0.581 3.3e-3
0.898 # 2.8c-5
-1
0.606 + 3.4e-3
0.588 1.2e-3
0.898 + 6.8e-3
cls
0.542 5.4ee4
0.552 # 1.6e-3
1.135 # 2.8e-3
Table 13: OT-PCA and OT-PCA+ICL across different layers layer 3 repeat) on ESOL. Bold best value (highest for Pearson/Spearman, lowest for RMSE); Underline = second-best.
Method
Layer
Pearson
Spearman
RMSE
0
0.075 + 2.3e-3
0.117 # 4.4c-3
1.321 2.5e-2
0.053 + 5.7e-3
0.088 # 1.1c-2
1,331 # 2.2c-2
OT-PCA
10
0.029 = 9.6e-3
0.081 = 1.3e-2
1.336 2.1e-2
0.114 + 4.4
0.177 # 6.7e-4
1,268 2.4e-2
cls
0.223 +: 3.1e-4
0.232 : 8.4e-4
1.210 6.0e-4
0
0.602 + 1.8c-3
0.595 = 4.4c-3
0.894 + 6.6c-4
0.579 + 4.0e-3
0.582 = 6.0e-3
0,920 2 2.1e-3
OT-CA+ICL
10
0.600 + 1.9c-3
0.608 + 2.2e-3
0.898 + 1.3e-3
0.598 1.3c-3
0.603 1.9c-3
0.892 3.9c-3
cls
0.493 + 3.8e-3
0.527 1.5e-3
0.898 2.8c-5
F.7 Experiments on the protein task and the DTT task
We also conducted experiments on drug-target interaction (DTT) and protein-related tasks. However, due to the poor performance of standard ICI on these datasets, with Pearson correlation coefficients below 0.2 [2]. Specifically, when evaluating the models test set consisting of 1,000 samples,
28

---

observed that all methods performed close to random guessing. Regardless of whether representations or protein sequences were used, the models failed learn the corresponding tasks. This can be attributed to the fact that such datasets typically focus on protein sequences within fixed domain, resulting in high similarity between sequences and their FM-derived representations, with similarity scores approaching 0.99. To further investigate the performance of ICRL on such datasets, we conducted experiments using a randomly sampled test set of 90 samples for reference purposes.
For the DTI task, due to the limitations of the context window, the number of shots was set to 10, while the other experimental settings remained consistent with those described in the main text. When using Pearson as the evaluation metric, we observed that ICRL still demonstrates relatively strong performance across most datasets, as shown in Tables 14 to 17, sometimes even surpassing ICL. Additionally, we provide results based on other evaluation metrics for reference. However, it is important to note that these findings are based on small- small-sample experiments. In larger datasets, the performance of all methods tends to converge and show minimal differences.
V2 for protein. To address the issue of protein sequence similarity, we conducted a simple method by extracting tokens preceding the CLS token from the ESM model. These tokens, which contain specific information about the protein sequences, were used input features for ICRL. This approach demonstrated some improvement in small-sample datasets, as shown in Tables 16 and 17. Unfortu- nately, we found that it fails to resolve the fundamental issue of random guessing when applied to larger test sets.
Effect of different FMs. We also compared different protein FMs with varying levels of repre- sentation similarity. Using ESM2, which produces highly similar embeddings (average similarity ~0.98), both ICL and ICRL performed poorly and often degenerated random guessing. In contrast, ProtBert generates more diverse embeddings (average similarity ~0.92), leading to consistently better results on stability and fluorescence prediction (see Tables 18 and 19). These results highlight that the diversity of FM-derived representations is critical factor for ICRL effectiveness: when embeddings are overly homogeneous, the benefit of representation- more distinguishable embeddings allow ICRL to extract useful signals and surpass standard ICL. Importantly, this observation consistent with our analyses on amal-moleecus datasets, indicating that the conclusion is modality-agnone
inputs diminishes, whereas
Table 14: Pearson comparison 1 on the different DTI datasets without ICL. Bold/ Underline: best/second-best value among the Embedding Injection methods. *; 1000 test samples.
String Injection
Embedding Injection
Dataset
ICL
PCA
Random
Rep
Embed+OT
PCA+OT
BindingDB IC50
0.149 263e2
0.009 25.4e4
0.092 23.60-4
0.051 21.6
0.177 42/e
0,023 21.544
BindingDB _i
0.172 48.5e4
0.195 48.2c-4
0.170 44.60-4
0.182 43.3
0.113 45.1e
0.184 47.8e
BBndingDB Ki*
0.045 22100-
-0.004 3384
0.084 1.20-3
0.010 =2163
-0.006
0.037 +2,46-4
Table 15: Pearson comparison DTI task datasets. Bold/ Underline: best/second-best value
compared with (highes for Ser_corf), *; 1000 test samples.
Baseline
ICRL (Ours)
Dataset
Text ICL
Text PCA+ICL
Embedding
Ran-Noi+ICL
ICn-Pro+ICL
OT-Embed+ICL
OT-PCA+ICL
BindingDB_ JC50
0.149 269e-4
0.124 428e33
0.148 36.5e
0.081
0.164 45.le3
0.163 #28e-3
RindingDB_ Ki
0.172 4854
0.241 k1.
0.303 47.504
0.268 41.3
0.263 #1.163
0.333 +89.-1
BindingDB_KK Ki*
0.045 x2.10-3
0.039 493824
0.084 s1.603
0.088 23.10-3
0.078 #7003
0.074 +396-7
F.8
Experiments with -.ama-3.2-3--nnstruu
To further investigate the reasons behind the failure of the DTI task, we conducted experiments using smaller models, i.e. Llama-3.2-31 --nstuutt As shown in Table 20, when employing smaller models, even on simpler tasks such as solubility prediction the ESSL dataset, the standard ICI. approach performs poorly. Notably, under these conditions, ICRL outperforms ICL, suggesting that the poor
29

---

Table 16: Pearson comparison on the different ESM task datasets without ICL. Bold/ Underline: best/second-bot value among the Embedding Injection methods. OOM: out of memory.
String Injection
Embedding Injection Rep
Dataset
ICL
PCA
Random
Embed+OT
PCA+OT
Fluorescence
0.237 11.9e5
0.098 25.1e-4
0.114 48.4e-5
0.078 22.6e-4
0.051 24.7e-5
0.154 29.4e-5
Stability
0.130 17.3e4
0.057 18.7c-
0. =1.2c4
0.172 27.5e4
0.097 22.9-
0.143 24.9c-4
Stability_
0.130 64.1e4
OOM
0.187 +9.1e-7
0.212 +7.20-6
0.173 +5.60-6
0.180 =7.9e-6
Table 17: Pearson t comparison on ESM task datasets. Bold/ Underline: best/second-best value compared with ICL. *; 1000 test samples. OOM: out of memory.
Baseline
ICRL (Ours)
Dataset
Text ICI.
Text PCA+CL
Embedding
Ran-Noi+1CI.
Can-Pro+ICI.
OTT-Emhe+-ICI.
OTT-CA+ICL
Hluorescence Stability Stability v2
0.237 +1.9o-5
0.164 +2.1044
0.159 +7.5e5
0.129
+4.5o-5
0.083 +506-
0.151 +2.e-5
0.130 a7.3e-4
0.133 452e-3
0.131
14.e-4
0.138
0.141 12.ee
0.094 s7.ee5
0.130 s4.1e-
00M
0,111
#1.ee4
0.217
24.0-4
0.177 #1Ae3
0.194 =2905
Table 18: Protein results using ESM2 as FM encoder (high similarity).
ESM2 (sim ~0.98)
ICL
OT-Embed
OT-PCA
OT-Embed+ICL
OT-PCA+ICL
Stability
0.720
0.712 1.322
0.703
0.827
0.642
Fluorescence 0.995
1.222
0.997
0.987
Table 19: Protein results using ProtBert as FM encoder (more diverse embeddings).
ProtBert (sim ~0.92)
ICL
Om-embed
OT-PCA
OT-Embed+ICL
OT-PCA+ICL
Stability
0.720
0.631 (10.80))
0.644 (10.059)
0.673 (10.154)
0.577 (10.065)
Fluorescence 0.995
1,230 (10.092)
1.044 (10.178)
0.984 (10.013)
0.949 (10.038)
performance of ICRL the DTI task stems from the model's unfamiliarity with the task, primarily due to insufficient pre-training knowledge.
Table 20: Pearson, Spearman, and RMSE results under the Llama 3B model. The symbol indicates
that no normalization was applied. The results demonstrate a clear advantage of ICRL.
ICL
PCA+OT
PCA+OT_IC JCL
Rep*
Rep+ICL**
Pearson
0.048 29.96-5
0.100
29.4e-4
0.133 #2.0e-3
0.077 #1.4-3
0.140 26.20-3
Spearman
0.045 4
0.092 44.6e-4
0.117 36.9e
0.070 43.3e
0.126 3
RMSE
1.317 +2.26-3
1.398 45.5e-3
1.341 12.1e-2
1.303 46.3e-3
1.244 25.8e-3
F.9
Generalization Across LLMs
To further analyze the generalizability of ICRL across different language model capacities, evaluated three LLaMA models varying sizes: 3B, 8B, and 70B, on two molecular property repeated 10 times with different
prediction datasets ESOL and Caco2. Each experiment random seeds, and we report the average of the top runs for robustness.
As shown in Table 21, across both datasets and all model sizes, OT-based embeddings were con- sistently better interpreted and leveraged by the models, particularly by smaller models (e.g., 3B). This may be attributed to their more limited pretraining capacity, which makes external structured
30

---

representations more useful. Notably, ICRL even surpasses ICL in some scenarios, reinforcing its effectiveness and the dual-mode framework discussed in Sec. 5.
Table 21: RMSE(L) results across three LLaMA models on the ESOL and Caco2 datasets. OT-based methods shows consistent improvements over baseline methods, especially in smaller models.
ESOL
Caco2 Zero-Pad
Model Size
ICL
Ran-Noi
Zero-Pad
OT-Embed
OI-PCA
ICL
Ran-Noi
OT-Embed
OT-PCA
3B
1.313
1817
2.013
1.299
1.315
1.027
1.207
1.193
0.971
0.965
1.764
1.186
1.098
70B
1.158
1.412
1.727
1.199
1,243
0.832
1,026
1,037
0.899
0.898
These additional results confirm that OT-adjusted representations can be effectively interpreted and utilized across spectrum of LLMs, thereby reinforcing the general utility of our method.
PCA
(B) OT-PCA
(T-T)CCA
(E+T) OT-PCA+ICL
Caco2
ESOL
0.4
0.5
Pearson
Pearson
0.4
0.3
0.3
0.2
0.2
0.1
5
10
20
30
40
50
5
10
20
30
40
50
PCA dimensions
PCA dimensions
Figure 6: (a) and (b) present the performance of various methods under different PCA parameter settings, measured using Pearson correlation (higher is better).
(b)
F.10 More Analysis about PCA Ablation.
The experimental results in Figs. 6, demonstrate that the performance of the text-level injection method is significantly affected by this parameter, particularly in non-ICL. scenarios. In contrast, the embedding-leve. injection methods exhibit apparent stability, regardless of whether they combined with ICI.
This phenomenon arises because, in text-level injection methods, modifying the PCA dimension directly alters the length the injected strings, which can substantially impact the final inference results. On the other hand, in embedding- level injection methods, this change only introduces minor variations to the target distribution. Consequently, the OT-PCA method remains insensitive to this parameter.
F11 More Analysis about Representation Similarity.
In our experiments, we observed that when representation similarity is excessively high, the decoded representations exhibit nearly identical character compositions. Consequently, the model's output values are also highly similar, differing primarily in sequence and minor variations decimal places due to the specific representation values. As illustrated in Figs. 10 and which present examples of the Random Projection method, the model's responses to different queries show only sequential differences. The high degree of similarity in the representations can be attributed, in part, to the inherent characteristies of FM representations in chemical datasets. More about the visualization results of the similarity analysis in Sec. 5 are shown in Figs. 7 and 8.
This issue particularly prominent protein and drug-target interaction datasets, where the similarity values tend to cluster closely together, often reaching cosine similarity levels as high as 0.99.
G
More Details
Due to the page limitation, present more details about settings, results, and analysis here.
31

---

1.(E)
(E)
AqSoIDe
AstraZenncc
1.03
1.08
1.03
0.42
3393
0045
Cosine Sintaity
0.75
Smfarty
0.77
Casino Simlaity
0.78
0.52
0.51
0.52
0.26
0.25
0.20
0.00
00.01
9M
0135
0.00
2
2
4
00G2
6
7
Method
Method
Method ID
Figure 7: Internal similarity across datasets
Zoro
(E) Ram-Noi
3: Ram-Pro
(E) OT-PCA
(E)
(T)
LD50
A4SxDB
AstraZenoca
0.28
0.36
Cosine Simbontty
Cosine Simitely
Cooine Simiarty
0.14
AMC
0.06
0.03
3300
0130
.0088
-0.01
5
0.00
0..0
6
Figure 8: Similarity to text embedding
G.1 Experimental Setup
Normalization. First, we compute the overall average mean and variance from the non-zero elements of the embedding matrix of the LLM, ensuring that the statistics representative of meaningful data. Specifically, we calculate the mean and variance for each embedding vector, then take the average across all embeddings to obtain single reference mean and variance. Next, normalize the input embeddings by adjusting their mean and variance to align with the target values derived from the LLM's embeddings. This is done by: (1) Calculate the mean and variance of the input embeddings. (2) Shift the embeddings by subtracting their mean and scaling them to match the target variance. (3) Finally, add the target mean the scaled embeddings ensure statistical consistency with the LLM''s embeddings.. This normalization step ensures that the input embeddings are statistically aligned with the LLM's internal representations, improving compatibility and potentially enhancing model performance.
In experiments, we observed that applying normalization to representations obtained through random methods occasionally resulted in NaN or Inf values. To simplify the process, we opted to disable normalization across all datasets. For the AqSoIDB and LD50 Zhu datasets, similar issue was observed with the OT--Embed method. However, given its relatively low occurrence- appeerrng in only 2-3 out of 10 random seeds we conducted non-normalized experiments only for the OT-Emmbed method on these two datasets.
It is noteworthy that while normalization was consistently disabled across experiments, in non- ICL scenarios, omitting normalization generally led to improved performance, particularly for the OT+PCA method. In such cases, the Pearson correlation coefficient improved by approximately 0.1 to 0.2, likely because normalization disrupts the distribution adjustments made by the OT method. Conversely, in ICL-inclusive settings, normalization proved beneficial in mitigating noise in representations, thereby enhancing retrieval performance.
System prompt. In all experiments presented in the main text, we consistently utilize simple system prompt: You are a drug expert. The answer should be different from the examples; DO NOT COPYANY FLOAT VALUE. For illustration, Fig. 9 provides example of decoded prompt for the OT+PCA+ICL approach.
G.2 In-Context Example Sampling
For all regression and classification tasks, we adopt stratified sampling strategy to select in-context examples from the training set. Specifically, for a given number examples ki the training data are partitioned into & equal-sized bins according to the label distribution, and one example is randomly
32

---

drawn from each bin. This ensures that the selected demonstrations cover diverse range of labels and avoids werrrepresenting particular regions of the label space. For question answering and caption generation tasks, we randomly sample & examples uniformly from the training split at each run, following the original benchmark protocols. Beyond these default strategies, we also explored more informed selection methods. For example, using Tanimoto similarity to choose in-context examples that are closer to the test instance the molecular embedding space can further improve ICRL performance. As discussed in Sec. 4, many techniques that enhance ICL quality-$uch as similarity- sampling-alss benefit ICRL in a consistent manner.
Considering the computational cost of running inference with 10 repeated trials, we further adopt the following rule for constructing candidate pools: 1. when the training dataset contains fewer than 1000 samples, we directly use the entire training set as the candidate pool; 2. when the dataset contains more than 1000 samples, we uniformly down-sample to 1000 candidates before applying the sampling procedure. This ensures a consistent and tractable evaluation budget across datasets of different sizes.
G.2.1 Differences in different assessment indicators.
We employed three evaluation metrics: RMSE, Spearman correlation coefficient, and Pearson correlation coefficient. Among these, RMSE the most sensitive metric, whereas improving the Pearson coefficient poses the greatest challenge. For instance, PCA may achieve a lower RMSE than ICL, yet its Pearson coefficient is lower, indicating that PCA should not be considered superior to ICL. Therefore, we primarily use RMSE to accurately assess performance differences in ICRL, while Pearson correlation is employed to demonstrate ICRL's contribution to enhancing ICL performance. The complete evaluation results are provided in Tables 22-25 for reference.
Table 22: Spearman comparison the different datasets. Bold/ Underline: best/second-bed value compared with ICL.
Baseline
ICRL (Ours)
Dataset
Text ICI.
Text PCA+ICI.
Embedding Ran-Pro+1CI.
Zern Prrn-Pad+IC..
Man-NoisICI.
OT--Bmbed+ICI.
OT-PCA+ICI.
ESOL
0.464 s25e3
0,460 27.4c
0.554 25.04
0.562 25
0.526 a14c4
0.522 st.7c4
0.552 =l
Caco2_ Wang
0.416 42.ea
0.387 21.6c4
0.407 25ie-4
0.426 23
0.410 21.es
0.409 23.3--3
0.401 21.433
AqSoIDB
0.610 21.es
0.551
0.612
0.594
0.599 al.le
0.586
0.592 494
LD50_Zha
0.395 ataes
0.382 25.0e4
0.411 =2.e-5
0.399 23.66
0.408 21.2s
0.382 s2.36
0,380
AstraZeneca
0.233 12.344
0.189 41.46-5
0.239 42.6
0,235 16606
0.234 a1ees
0.250 12.3e-4
0.244 9e-4
Table 23; RMSE J comparison on the different datasets. Bold/ Underline: best/second-bes. value compared with ICL (lowest for RMSEJ).
Baseline
ICRL (Ours)
Dataset
Text ICI.
Embedding Ran Pro+ICI.
PCA+ICI
Pero-Pad+ICL.
Ran Noi+ICI.
OT OT-BeeedIICL.
OT-PCA+ICL
ESOL
1.158
1.135
1.085
1.152
1.037 #45e3
1.154
1.135
Caco2_ Wang
0.832 45444
0,842 s1.le3
0.830) So-d
0,841 47.5el
0.839 #13e3
0,815 el.ted
0.888 M8l
AqSoIDB
1.917 239c4
2,029 23.3e3
1,910 25.204
1,944 24.c44
1.941 230e
1,963 s1.8e3
1,986 21
LD50_Zha
+1.74
0.998 =2.5c-4
0.966 22.0e-
0.984 e1.e4
0.995 #77ee4
1.002 #4/6c-4
1,004 21764
AstraZenecu
1.366 =3.4e3
1.319
1.353
1.399
1.372 22.e6
1.372
1.395
33

---

Table 24: Pearson comparison on the different datasets without ICL. Bold/ Underline: best/second- best value among the Embedding Injection methods.
Text Injection
Embedding Injection Ran-Pro
Dataset
ICL.
PCA
7cro-Pad
Ran-Noi
OT-Embed
CT-PCCA
ESOL
0.465 19224
0.291 +1.o33
0,155 +3.703
0.123 =2.6e4
0,125 =1.303
0.270 =8.2o-3
0.227 +7.3c4
Caco2_ Wang
0.411 21.3e3
0.163 21.5e
0.122 21.2e
0.114 #1.3e
0.113 22.0e
0.226
=1.3e
0.245 #4.8e
AqSolDB
0.596 15.te5
0.075
0.027
-0.026
0.035
0.1
0.215
LD50 Zhu
0.378 21.205
0.128 21.6e4
0.047 2900-5
0.029
=1.204
0.026 =1.3e-5
0.064
=5.6e-4
0.079 29.9
AstraZeneca
0.266 13.565
0.027
0.041 60.8e55
0.046 76-5
0.049 #3.9e-5
0.018 #8.4e-5
0.043
Table 25: Spearman comparison f on the different datasets without ICL. Bold/ Underline: best/second- best value among the Embedding Injection methods.
String Injection
Embedding Injection
Dataset
ICI.
PCA
Zero-Pad
Ran-Noi
Ran-Pro
OT-FEbed
OT-PCA
ESOL
0.464 x2504
0.299 18.4-4
0,177 =6.8-3
0.127
=16644
0.151 =1.3e3
0.231
=6:0-3
0.235 =1.5:44
Caco2_ Wang
0.416 #2.363
0.175 282e-7
0,124 31.4e
0.113
=2,ee
0.107 =3.8e
0,245
2dile 4
0.240 =2.8e
AqSolDB
0.610 57.5e5
0.056 1388-3
0.025 23.2e-4
-0.026
#l 3e-
0.045
23.2e-4
24.9e-3
0.203 31.3e33
LD50 Zhu
0.395 2836-5
0.165 2.5e4
0.054 29.9c.6
0,039
=1.8e.4
0.027
=2.6e55
0,090
s1.6e
0.091 #1.4e-4
AstraZeneca
0.233 +2 3o44
0.028 +88
0.035 +1 43.0
0.041 -0.7e-4
0.040 =2.0e5
0.015
-7.5e-5
0.040 8o-4
Table 26: Spearman comparison 1 on the different DTI datasets without ICL. Bold/ Underline: best/second- best value among the Embedding Injection methods. *; 1000 test samples.
String Injection
Embedding Injection
Dataset
ICL
PCA
Random
Rep
Embed+OT
PCA+OT
BindingDB_ICSS
0.156
0.001
0.080
0.035 +6.60-3
0.156 +2.1e-3
0.035
BindingDB Ki
0.152 +3.5e-4
0.185
0.184
0.157 +1.70-4
0.104 =2.0e-4
0.172 +1.96-4
BindingDB Ki
0.030 1.7e-
-0.004 12.8o-4
0.064 1.2e-
0.007 1.76-3
0.002 a1.6e-4
0.031 13.2e-4
Table 27: RMSE comparison + on the different DTI datasets without ICL. Bold/ Underline: best/second best value among the Embedding Injection methods.** *; 1000 test samples.
String Injection
Embedding Injection
Dataset
ICL
PCA
Random
Rep
Embed+OT
PCA+OT
BindingDB ICS0
1.740
1.659 25.6e-
1.767 28.504
1.754 2244-3
1.726 s4.5e-4
1.743
BindingDB_ Ki
1.575 4335
1.555 +7.26-3
1.665 +4.26-4
1.578
1.678
1.655 +4.96-4
BindingDB Ki*
1.631 #2.663
1.646 +2.8-4
1.612 49.26-3
1.683 #7.2-33
1.716
1.675 +3.8e-4
Table 28: Spearman comparison on DTI task datasets. Bold/ Underline: best/second value compared with ICL, *; 1000 test samples.
Baseline
ICRL (Ours)
Dataset
Text ICI.
Text PCA+ICI.
Embedding
Ran-Noi+ICI.
Ran-Pro+ICI.
OT-Embed+ICL.
OT-PCA+ICI.
BindingDB_ICSK
0.156 41 Ae-4
0.120 42.8-
0.133 44.55
0.057 44.6-4
0.158 +48e4
0.163 +28e-5
BindingDB_ Ki
0.152 43.5.5
0.200 k13
0.267 18.5044
0.223 s1.764
0.229 #2004
0.310 +23645
BindingDB Ki*
0.030 21.7e-3
0.017 45.684
0.070 28.5c-4
0.069
0.064 24.6-55
0.055 4426-5
34

---

Table 29: RMSE 4 comparison on DTI task datasets. Bold/ Underline: compared with ICL, *: 1000 test samples.
best/second-best value
Baseline
ICRL (Ours)
Dataset
Text
Text PCA+ICL
Embedding
Ran-Noi+ICL
Ran- Pro+ICL
OT-Embed+ICL
OT-PCA+ICL
BindingDB_ICSS JC50
1,740 3177
1.739 428:
1.630 54.5
1.637 s4.e4
1.615 449e
1.748 #2Me
BindingDB_Li Ki
1,575
1,572
1.486 58.6-4
1.544
1.524 #25e4
1,465 #28e5
BindingDB_K Ki
1.631
1.679 4236-1
1.645
1.617
1.659
1.675 +2.50-3
Table 30: Spearman comparison the different ESM datasets without ICL. Bold/ Underline: best/second-bes. value among the Embedding Injection methods. OOM: out of memory.
String Injection
Embedding Injection
Dataset
ICL
PCA
Random
Rep
Embed+OT
PCA+OT
Fluorescence
0.204 21.7e4
0.046 25.4e-3
0.132 48.7e-4
0.083
24.1e-4
0.096 27.7e:
0.159 29.5e-4
Stability
0.133 17.063
0.056 14.70-4
0.170 =1.3e-3
0.199
28Ac-4
0.095128e-3
0.149 27.9c44
Stability_v2
0.133 64.26-5
OOM
0.163 +8.2e-6
0.187
+7.76-6
0.170 +4.60-6
0.166 #6.3e-6
Table 31: RMSE comparison., the different ESM datasets without ICL. Bold/ Underline: best/second-bert value among the Embedding Injection methods. OOM: out of memory.
String Injection
Embedding Injection Rep
Dataset
ICL
PCA
Random
Embed+OT
PCA+OT
Fluorescence
1.213 +2,7e4
0.046 +7.4-33
1.429 +8.1e-4
1.575 +1.6044
1.613 +3.2e-3
1,523 #6.7e-4
Stability
0.735 17.963
0.649 12.704
1.009 =1.6e-3
0.973
#8.8c-4
0.894 16.5033
0,952 14.1c-4
Stability_v2
0.735 +7.26.5
OOM
0.894 +7.7e-6
0.878
+7.7e-6
0.898 +4.96-6
0.862 =5.8e-6
Table 32: Spearman comparison ESM task datasets. Bold/ Underline: best/second-best value compared with ICL. 1000 test samples. OOM: out of memory.
Baseline
ICRL (Ours)
Dataset
Text ICI.
Text PCA+ICL
Embedding
Ran-Noi+ICL.
Ran-Pro+ICL
OT-Embed+IIL.
OT-PCA+ICL
Fluorescence
0.204 21.764
0.137 44.e
0.129 27.05
0,061 26.5e
0.110 47
0.138 13.7e
Stability Stability_v2 v2
0.133 #7.9-3
0.146 45765
0.125 14.1e-4
0.153 46776-
0.142 12.10-4
0.086 +7.ee5
0.133 #4.6-5
OOM
0.111
0.192 14.1e-4
0.1 44
0.184 189e-
Table 33: RMSE comparison J on the different ESM datasets without ICL. Bold/ Underline: best/second- best value compared with ICL. OOM: out memory
Baseline
ICRL (Ours)
Dataset
Text ICL
Text PCA+ICL
Embedding
Ran-NoisICL
Ran-Pro+ICL
OT-Embed+IIL
OT-PCA+ICL
Fluorescence
1.213 x2764
1.462 47.6-3
1.407 47/0-3
1.467 45.5e-4
1.479 4700-4
1.370 16.7e-4
Stability Stability v2
0.735 47.9
0.848 47365
0.770 +4.e--6
0.780 1o-4
0.716 +2.3
0.848 +7.004
0.735 #7.26-5
OOM
0.719 +2.8e-5
0.720 45.1045
0.738 a4.7o-5
0.780 +8.4c-5
35

---

Prompt
<beginn (ttt:t//ssbrttteeeeee heesserrreeetrrr_rrerr.e poystemmdedd header,
You drug eceet. The arswer shoul be diflrrent fro the examples, DO NOT PD ANY FLOAT VALUE.
<teat, djmestart, heacer, disusardennd header, Mj--
N-C1CroCC3
Given SMILES
the
Quastion: Wharis the Souubigg ofee melecule abowe?
Molecular recprreeeeettttttoo (REP
Anower -0.55
Drug SMILES
the
specitle
Question: Whatls the Solub By ofte drug molecule abovo?
Molecular vae worooreppee tereppeetiiion (2)(PP)
Ancwer: 2.48
<CCDCC-0XC 1-ci CCCCCCC0O 300 CCCO ccterrooe the
answer question speciiee Quastion: Whatis the Solubilly ofti drug melecule abow??
Given SMILES
Molaculor socerrpppaaittooo (38(JE3))
Anewrer -4.5
diven SMILLS
of the drgg leuee above, ansver the follwing question the specilled format. Question: Whatls Solub oftec drug molecule abeve?
Molecular vor ieppsstttiioo (520AAE)
Anower: 5.85
SMILES: <CCcoooclC
Question: Whatis the Sorih diility ofhhe drug melecule above? the
Given SMILES
the
speciled
Molacular voctorrpp vocerrepesattaaooo (AER)SS](EEP))
Annarer -3.07
SMILES: -CNICCCC-O//cciccCiicc Given SWIl
of the drug mo lenie above, ansver the following question the specilled format. Question 1: What a the Souuiiity the moletule aboue?
Molecular voc teprppaatiiioo JM (EEB-A)AAP))
resond
folowinng QUESTION
quector:
Question
..END FORMAT TEMPLATE FOR Q KTN
Drug SMILES: (CC.NN/NN/ooIccccioo000 (S11) CCCCCCCC--/3c2CCC
Giveen SWiLL sequence the drig melectle abowe, ansuer the follwing question using the specilled
Molecular receerrepresoiiaaaonn [42IEE))
Plaas wapond with the Balowingg formalfrr each queelar:
. BEGRN FOR WAT TEMPLATE FD QUESTION 2
Answer 2. Your Answer here Quaction 2]
QESTION
<CC (Cc-o)00clocc.aa (ececc2ccc
Given the SWILES sequence of the drug mo boule above, answer the follwing question using the specifled format.
Question 3: What is the Soub iit of drug molecale above?
Molecular vecerrppesriaaion (EP8)7E)AEE)))
falowing resood TEMPLATE .-BEE
queslars
Your Answer
Question
..END FORMATTEMPLAT TEMPLATE FOR Q UE 3--t dipaaartt header sspassitarr-gnnn, header idp>>
Output
BEGIN FORMAT TEMPLATE FOR QUESTION 1.-
Answer 2::293
-IEGIN EDAMAT TEMPLATE QUESTION 2--
Answer 2 24
END GERMATEEMPLATE FOR QORQUETIIONO
FON TEMPLATE FOR QUESTION 3..
Figure Decoded prompt example and output for ESOL task.
36

---

Prompt
-pegin of topp-qsart header, yetemmedde header,
should diferent
NOT FLOAT VALUE.
-fpot, Ministtrr, header, disusersfedd header, di>
Drug MMILES: <0 acio cacccio o2ttct/00 20C OiCIO/CI Icoc22 > sequince
fillowing question using specifed
C orug Question: What's perection Jdss((EPP)
Answer: 46.3
Drig SMILES 5: Ocio-O0cccce I-cle 2ecc 2oc2or Mi 12
Glvzn the SMILES sequence the drgg malecule above, and wer the tollowing question us ng the specifed fomat permeaiinn through Caco- ee drug melecule
Molecular vecbrresssseeeebie
[dss)REP))
5..44
Drig SMILES: CCcooNN2C CMC (-C @ HJC cacccli (ocCC iCc C2ccC HHINE (-OICCNICIIIIIC CCCCCCCCCCCCCCCCCCC
Given the SMILES sequence the drug molecule above. answer the followng question using the specifed formst Question What's the sermesiin rate trough Coco co of the ong motezule abous? Moleculer representation jine RI
Answer:
SAIje iesooicc30jc HRRceeco0 K10) BHOOKC Given the SMILES sequence of moliccule above, answer the following qusstion specified famat. Queston: What's the permcatin rate thro Caco the orug moletule above?
MMILES: (C--H400
Moleculanvee rewreeeeerettoooo fatonn [REP (dasjREP))
Answer: 4.82
NCCICCO Question What the permaaton rale through Caco> the melecole abows? Moletularvectru vectarrreeecrrrooo repreentation |REP [dss[(EEP) Answor--..
SMILES
molecule
specifed forat.
Drug CCCclooCC
sequence
ng specifed melezule above?
What's the
Moleculer representation IREP (dms(AEP))
Answer: 4.63
Drug SMILES: -CCNCCICCCCCC
SMILES sequenca #eloonc2cdiClooot
following question using specified fomat. drug molecule above?
Question What permaatin through
following formattor
--BEGR Ni FORMAT EPLATT FOR QUESTION I..
Answer [Tour Answer for Question 1) --f ND FORMAT MTTTT TERPLATE FOR QURSTION 1-
Drug
SMILES clecc2gc1e wel.cencccccccccccccccccc 2CC
following suestion us ng specifed molecule above?
Moleculer representaiin tadion: [RLP dess(ALP))
Picase respond with the following formatfor queston:
--Eid Ii FORMAT MAT TE WPI LATE F OUFS KON 2--
Answer [Your Answer Here for Question
QUESTION
SMILES: ES:
Inc2ccedC [-O)c3ccooc Ser2]/K]
Given the SMILES
molecule
the following question specifed fommt.
Question What is the permeatio rabe through Caco cells the molocule above?
Molecular peecullrveee vector represer eattoon [RLP (dee(iEP))
Piease respond with the following formatfor each queston:
MAT
Answer
FORMAT TEMPLLA
QUESTION 4-ot
apassisarttseed
Output
FORMAT
QUESTION
FORMT TEMPLATE OUESTIONN
-BGG R ORMMA TEMPLATE FOR QUESTION 2--
Answer 2- 66511 ND TEMPLATE QUESTION
BFG TEMPLATE
- FORMAT TEMPLATE FOR QUESTION 3..
Figure 10: Decoded prompt example and output for Caco2 task.
37

---

Prompt
<(belh lexti=sstar, header dljsystammjend header,
The should déferent exmplees: NOT COPY FLOAT
<eot, bl alsasttrrt heaser, plpuser serdeedd heacer, kip
2cceeo/ccteecce D//CC/////////// 30
Given
using
Queston: Whatts
trrough Caco
molecule above?
Mokecular voclorrip apsartattii J&En Pd Ansswar -.36
CCCC aec.coeccccccceoooocccoo 2accccieeecccceeeeeeeee
drgg
question
Question: Molecularr veciorrsp oeceraipeeeraaaio [REP] des
trrough
bolecale
Ansaeer -5.90
Given CcllcciNCC (-oji da c3ronc (ocaCiD c2w ip -ONCCC (CC)) motecile above?
molecule
Quesbom: Whattis Mokecular wostorripsssraaaaii JREP BBB BjREP]
trrough Caro
Aneaeer -4.8
<CIC @4DOjC HDO @ (DCHHO je300 capoC @ BcZecct0
sing HJ0/CCH0011C0H00
Question:
trruugh
the aboue?
Mebecular vae weprrppeeeriiii J&EP [
Ancwer: 6.82
SMILES <CNEXCCC Nic.cccccccccccccoooooo
Question. sequence drig
lloingg question ssing
Meleculer eppesrreeestnnn JREP (ees/Me Aneseer -4.00
Drug CC be coceloood 2e1
Quesbon: sequence drug malecule
question using
thruugh
becule
Mekecular vae veclor vechrrappeettaaii J&EP (EEP)deghEP))
Ancwerr 4.83
Drug SMILES: <CScUiCa HJCC g--i-t clccece JONc ji (i a Iik INC (sDiIC igrn Moccccoo0ooc////ayyy Question What
drig
using
through
moleccue
Meleculer vee pepreppeerrriiio J& Binn (IFP))
Please respond whth the follaring formatfor queston: ..EE & FOR MAT TEMPL ATE FOR QJIFSTION 1-- I: Ansoed
SMILES: ecci2e 3IOICIC @HDON CoceC/Ccclllll (ccooc344 IC ICt-OlOC
HiBic JOYCI
-0 DOCCCC DID IC DXC ICCCCN CVC (a 20
Givan the SMILES sequerie the drug meleculé above and wer the lo looking question using the specified fornat Guestion What the parmetton trough Cacodl cels of drug molecule above? Molecular rctarrepprserraaioon
des//REP)
Cl FORMA TEMPLA TP OO QUESTIIN J..
Ancwerr (Your Answerticre Question 2]
.-END FORMAT TEMPLATE FO QUSTION 2..
Given <CIC HING #3CCclcccoccCC (CCCCIIIICCCCCIICCIICCCCCCCC ii HILC((000
Queston What
thrugh Cacodl of molecule above?
Molecular veclorrapeesarneeee ropresariaation (REP)assRRE?
Ploase respond with the follerng format each queston:
..REG
QUEST
3] 3 Answer QUFSSINN
Hlpessisartcpedd
Output
BEGR FORMAT TEMPLATE FOR QUESTION 1.-
--BIG N FORNAT MPLATE FOR QUESTION 2--
Answer 2--6.11
-END FOMMAT TEMPLATE FOR QUESTION 2--
.-NDD FORMAT TEMPLATE FOR QUESTION 3...
Figure 11: Decoded prompt example and output for Caco2 task.
38

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. They provide clear overview of what the paper aims to achieve and the methodologies used, aligning well with the detailed findings presented in the subsequent sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, the paper does discuss the limitations of the work performed by the authors. It helps the stage for future work and encourages ongoing dialogue the field.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
39

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The paper clearly states all necessary assumptions prior each theoretical
result. Each theorem or proposition is accompanied by complete and logically sound proof, either in the main text in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Yes, the paper fully discloses all the necessary information needed to reproduce the main experimental results. The authors have been meticulous in detailing the methodol- ogy, settings, and parameters used in their experiments, ensuring that other researchers can replicate the study accurately and validate the findings.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
40

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code and data released at submission time to preserve anonymity. However, the authors state that all necessary code, data, and instructions will be made publicly available upon paper acceptance and publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
aublic/guides// Saaiddee/ooeSuumisseee comiiissiiooicyyyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/nips.
/CodeSubmi cesssummissiooni.. ssionpolicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, the paper specifies all the training and test details, including data splits, hyperparameters, the rationale behind their selection, and the type of optimizer used.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, the paper reports appropriate information about the statistical significance of the experiments.
41

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer:[Yor Yes]
Justification: Yes, for each experiment, the paper provides sufficient information on the computer resources required.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't 'tttt make it into the paper).
paper (e.g., preliminary or failed experiments that NeurlPS Code of Ethics https://neunt cc htte//ublcc/EEtiiieeeeee de del
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the
Answer: [Yes]
Justification: Yes, the research conducted in the paper conforms in every respect with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
42

---

Answer: [Yes]
Justification: Yes, the paper discusses both potential positive and negative societal impacts of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Yes, the paper describes safeguards that have been put in place for the respon- sible release of data models that have a high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, the creators original owners of assets used in the paper properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
43

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, new assets introduced in the paper are well documented, and the docu- mentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does involve any crowdsoureing experiments or research with human subjects. All results are derived from computational experiments using publicly available datasets and models.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
44

---

Answer: [NA]
This work does not involve human subjects or any form of user study. All experiments conducted using machine-generated data publicly available datasets, and therefore do not require IRB approval.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
45


Reviews:
[{"review_id": "ndU5RR2Psq", "reviewer": "Reviewer_S7er", "summary": "This paper investigates the feasibility of enabling Large Language Models (LLMs) to reason over non-text modalities in a training-free manner. Unlike traditional in-context learning, which incorporates text-label pairs, the proposed method replaces text inputs with FM representations, enabling the LLM to perform multi-modal inference without fine-tuning. Experiments show its feasibility and potential for generalizable multimodal reasoning.", "questions": "Below are some sub-topics that I'd like clarity on and included in the paper. \n\n1. While the paper provides theoretical support for linear projectors, could a lightweight, learnable projection module (perhaps with minimal training) further improve performance without negating the \"training-free\" promise of ICRL?\n\n2. How does the computational cost and latency of the optimal transport alignment method scale with increasingly larger LLMs and higher-dimensional FM representations during inference?\n\n3. Are there specific types of non-text modalities or tasks where ICRL's training-free approach might inherently struggle or be less effective compared to others, even within its target domains (e.g., molecular, sensor-based)?", "limitations": "Please refer the weakness section above.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is decently motivated, well written and easy to understand. The problem statement is clear and the requirements driven table (Table 13) in appendix makes it easier to understand the contributions w.r.t existing baselines.\n\n2. The paper successfully demonstrates the feasibility of LLMs performing multi-modal inference by leveraging FM representations with few-shot learning. This opens up new avenues for LLM application in diverse domains without needing the resources for additional training or fine-tuning of pre-trained LLMs. \n\n3. The experimental section is thorough, and discusses various relevant questions from different angles aside from the traditional benchmarking with baselines which makes the paper more insightful and takeaways more concrete. \n\nWeaknesses:\n\n1. As acknowledged in the paper, ICRL underperforms compared to supervised methods due to the absence of task-specific training. This suggests a trade-off between the training-free advantage and peak performance in well-established domains. I'd like to see this comparison to make the takeaway more concrete. The paper is insightful in its current form, however, as a reader i am not sure about the trade-off b/w accuarcy and cost when comparing training-free method vs the traditional route. \n\n2. The reliability of the method on carefully tuning PCA parameters makes its non-trivial to use, particularly for non-ICL scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer S7er for the valuable comments. For the concerns and questions, we provide the following responses.\n\n**W1: Unclear Trade-off Between Accuracy and Cost**\n\nThank you for the insightful comment.\n\nTo clarify the trade-off between performance and cost, we conducted a comparative analysis of ICRL against four recent \nLLM-based methods for molecular property prediction and found that \n**ICRL achieves comparable or even superior performance to lightweight fine-tuning methods—while requiring no training and only ~2 seconds of CPU computation**.\n\nSpecifically, we compare ICRL with instruction tuning via Q-LoRA (I-FT) [1], supervised full-parameter pretraining followed by \nsupervised fine-tuning (S-PT + FT) [2], and unsupervised full-parameter pretraining followed \nby supervised fine-tuning (PT + FT) [3,4]. The comparison spans training paradigms, resource consumption, \ntraining duration, and performance on two benchmark datasets.\n**_We will incorporate this comparative analysis into the main text in the next revision to make the trade-off more transparent to readers._**\n\n\n**Setup:** \n\n-- To ensure fairness, we implement ICRL using the LLaMA-3.1-8B-Instruct model—comparable in inference cost to those used in prior work and feasible on a single GPU. Evaluation is based on RMSE. Notably, ICRL achieves even better performance with larger models; e.g., on ESOL, it reaches 0.839 RMSE using LLaMA-3.1-70B.  \n\n-- Few-shot examples in ICRL are selected via stratified sampling from the training set, with 20 samples for ESOL and 50 for Lipophilicity, following the same protocol as in our main paper.  \n\n-- Regarding computational cost, we extract information directly from the original papers whenever possible. For example, [1] reports using 4× A800-80G GPUs for I-FT, and [3] states that pretraining took 11 days. When GPU specifications or training times are not provided, we estimate them based on similar work. Performance metrics are collected from: [4] Table 3; [3] Table 8; [1] Table 2; and [2] Table 3. Standard deviations are reported only when available in the original texts.\n\n|     |     |     |     |     |     |     |\n| --- | --- | --- | --- | --- | --- | --- |\n|     | Type | Resource | Time | Performance (ESOL) | Performance (Lipophilicity) | Performance (Avg) |\n| MolecularGPT [1] | I-FT | 4 Tesla A800-80G GPUs | < 1 day | 1.471 | 1.157 | 1.314 |\n| Gimlet [2] | S-PT + FT | 2-4 GPUs | ~ 1 day | 1.132 | 1.345 | 1.239 |\n| SELFormer [3] | PT  | 2 NVIDIA A5000 GPUs | ~ 2 weeks | 1.357 | 3.192 | 2.275 |\n| | PT + FT | 2 NVIDIA A5000 GPUs |~ 2 weeks | 0.682 | 1.005 | 0.844 |\n| GPT-MolBERTa [4] | PT + FT | 2-4 GPUs | ~ 2 weeks | 0.477±0.01 | 0.758±0.01 | 0.612 |\n| **OT-PCA (ours)** | **Traing-free** | **CPUs** | **~ 2 seconds** | **1.140±0.01** | **1.349±0.01** | **1.245** |\n| **OT-PCA + ICL (ours)** | **Traing-free** | **CPUs** | **~ 2 seconds** | **1.094±0.01** | **1.277±0.01** | **1.186** |\n\n**Analysis:** \n\n-- Under comparable inference cost, **ICRL achieves performance on par with or even superior to that of lightweight fine-tuning and short-duration pretraining + fine-tuning approaches, all while remaining entirely training-free**. While lightweight tuning can improve downstream performance to some extent, its effect is limited when the base model has not been pre-trained on small-molecule corpora. In contrast, ICRL allows a general-purpose LLM to achieve competitive or better results with only **~2 seconds** of overhead, highlighting both the method's effectiveness and practicality.  \n\n-- Improved understanding of small molecules does not necessarily translate to better performance on downstream tasks. In analogy to LLM pretraining, the pretraining in baseline methods is primarily intended to enhance molecular understanding. However, as shown in [3], the standalone pretrained model performs significantly worse than ICRL on downstream prediction, suggesting that pretraining alone mainly serves as a warm-up for subsequent supervised tuning.\n\n**Conclusion:** \n\nWhile a performance gap still exists between ICRL and full PT + FT pipelines, ICRL offers **comparable or even superior results** to supervised lightweight tuning—**without any training**—underscoring its practical value in low-resource or rapid-deployment scenarios.\n\n\n---\n\n**W2: Method Requires Careful PCA Tuning**\n\nThank you for the comment.\n\nWe would like to clarify that the analysis presented in Figure 3(a) and 3(b) focuses specifically on _text-level injected representations_, aiming to show that more expressive representations do not always help the model better utilize FM features. In contrast, _embedding-level injection methods_ such as OT-PCA are much less sensitive to the PCA dimension—particularly when combined with ICL, as demonstrated in Appendix F.2.2 and Figure 6.\n\nIn fact, across all modalities and datasets used in both the main paper and the rebuttal, we consistently set the PCA dimension to 20. This parameter **was not tuned during method implementation**; the figures simply illustrate a special **case analysis** rather than a requirement for manual adjustment.\n\n---\n\n**Q1: Potential of Lightweight Learnable Projectors**\n\nThank you for your insightful comment.\n\nAs shown in W1, while lightweight fine-tuning can improve performance to some extent, it still underperforms compared to the training-free ICRL method—largely due to the inherent difficulty of molecular property prediction tasks for general-purpose LLMs.\n\nTo further investigate this question, we explored two lightweight training strategies for the random projectors used in ICRL: one based on contrastive learning (inspired by CLIP), and another based on molecular captioning, following concurrent work. However, as shown in **Reviewer J6ma Q2**, both approaches perform worse than our OT-PCA method.\n\nWe conducted additional experiments by directly training the projector on the downstream task, and found that this approach led to lower predictive performance and impaired generation quality, especially in few-shot settings. Specifically, due to the limited scale of the datasets and the lack of domain-specific knowledge in the LLM, this setup often degraded the model's in-context learning ability—sometimes even preventing it from producing complete outputs. \n\nThese findings suggest that for these small molecular datasets, **lightweight training** of the projector—while feasible in principle—tends to result in **worse** performance compared to our **training-free** design.\n\n---\n\n**Q2: Scalability of OT Alignment with Model Size**\n\nThank you for the comment.\n\nWe would like to clarify that the OT alignment process does **not** scale significantly with the size of the LLM or the dimensionality of the FM features. The table below summarizes the runtime of OT under different LLaMA model sizes and FM feature dimensions. Importantly, like traditional pretraining or fine-tuning, OT alignment is a **one-time computation**. \n\nDuring inference, applying OT requires only a single matrix operation per sample, which takes **under 1 millisecond** in our setup—adding negligible overhead to the overall generation process.\nWe provide a more detailed discussion of ICRL's inference cost in Appendix E.7 and Table 12.\n\n|     |     |     |     |\n| --- | --- | --- | --- |\n|     | 3 B | 8 B | 70 B |\n| Unimol (512) | ~ 0.842 sec | ~ 0.843 sec | ~ 0.846 sec |\n| ESM2 (640) | ~ 0.844 sec | ~ 0.844 sec | ~ 0.849 sec |\n\n---\n\n**Q3: Limitations of ICRL on Certain Modalities or Tasks**\n\nThank you for your insightful comment.\n\nIndeed, our extensive experiments suggest that while ICRL can enhance LLM performance by leveraging FM representations, the degree of improvement is closely tied to the **\"difficulty\"** of the downstream task for the LLM—roughly indicated by its ICL performance.\n\nSpecifically, although ICRL outperforms lightweight training approaches on molecular property prediction and molecular QA tasks, it performs less effectively on caption generation tasks **(see Reviewer aUho W2)**. In these cases, the base ICL performance is already poor, suggesting that general-purpose LLMs struggle to learn such complex mappings from a few examples alone. While ICRL still outperforms ICL in this setting, the gap between ICRL and task-specific expert models remains substantial.\n\nA similar limitation appears in protein property prediction tasks. Due to the high sequence similarity within protein datasets, it becomes difficult for the model to distinguish between samples, leading to weak ICL and ICRL performance (see Appendix E.3). This issue, however, can be mitigated by adopting stronger FMs, as discussed in **Reviewer J6ma W1**.\n\n\n---\n\n[1] MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Prediction. Arxiv 2023.\n\n[2] GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning. NeurIPS 2023.\n\n[3] SELFormer: Molecular Representation Learning via SELFIES Language Models. Machine Learning: Science and Technology 2023.\n\n[4] GPT-MolBERTa: GPT Molecular Features Language Model for molecular property prediction. Arxiv 2023.\n\n---\n\n**_As reflected in our title, ICRL serves as a case study in our broader exploration of how LLMs can reason over non-text modalities in a training-free manner. We sincerely appreciate all the valuable feedback provided by the reviewers and will incorporate the insights and suggestions from this rebuttal into the revised version of the paper or pursue them in future work._**"}, {"review_id": "JZSZ8eX6yp", "reviewer": "Reviewer_J6ma", "summary": "This paper explores feasibility of integrating representations from non-text FMs into text LLMs without having to rely on supervised training (costly). The eventual goal is to make progress towards multi-modal generation by being able to adapt LLMs to new domains and modalities on the fly. They propose a framework (“ICLR”) meant to be a proof of concept to allow LLMs to adaptively utilize non-text modality representations with few shot learning (training free). Main contribution is a theoretical backed framework to evaluate two main strategies for injecting non-text representations: text-level injection and embedding-level injection.", "questions": "(1) I expected to see a more diverse set of evaluations – I see some experiments on vision and audio, but most of the work is done primarily in molecular properties (at least this is where the strongest results are demonstrated) \n\n(2) concern on practical feasibility since ICLR (particularly when applied on its own) seems to underperform relative to the baselines. Overall, I found the results inconsistent. There are some interesting potential explanations (eg: injection being used for thoughts), but more discussion on understanding the result is needed", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: (1) well written, with clearly defined research questions. Easy to follow intuition and motivation with the discussion starting at the success of test time scaling. (2) novel and important work in multi-modality and low-resource domains – challenges the current paradigm which is expensive and restrictive (2) comprehensive exploration of injection strategies (3) good to see theoretic grounding of the empirical findings – the linear mappings better preserve the geometry of the original embeddings compared to non-linear mappings (4) good discussion on task learning vs retrieval \nWeaknesses: (1) I expected to see a more diverse set of evaluations – I see some experiments on vision and audio, but most of the work is done primarily in molecular properties (at least this is where the strongest results are demonstrated) (2) concern on practical feasibility since ICLR (particularly when applied on its own) seems to underperform relative to the baselines. Overall, I found the results inconsistent. There are some interesting potential explanations (eg: injection being used for thoughts), but more discussion on understanding the result is needed", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer J6ma for the valuable comments. For the concerns and questions, we provide the following responses.\n\n**Q1: More Diverse Set of Evaluations**\n\nThank you for your insightful comment.\n\nWhile the experiments in the main paper focus on molecular tasks, this choice was made to enable deeper analysis of ICRL's underlying mechanisms under well-controlled conditions. Other modalities—including protein, vision, and audio—were also explored, but due to space limitations, these results were placed in the appendix. \nIn addition, we have included more types of tasks in our evaluations, such as QA and generation (see **Reviewer aUho W2**), as well as experiments on lightweight projector training strategies (see Q2). We plan to further extend ICRL to a broader range of modalities and task types in future work.\n\nNotably, the key conclusions drawn from the molecular experiments—such as the impact of representation similarity—are **modality-agnostic**.\nSpecifically, we observed that when using embedding-level representations alone, the intra-dataset similarity of FM representations significantly affects downstream performance. \nIn our earlier appendix experiments, ICL and ICRL performed poorly on protein property prediction tasks, largely because protein datasets often focus on a specific protein function and thus contain sequences that differ by only a few amino acids—leading to extremely high representation similarity.\n\nIn our latest experiments, we addressed this issue by replacing the FM encoder with one that produces more diverse representations, and the results below further validate the **generalizability of our findings across different modalities.**\n\n**Setup:**\n\n-- Both tables use LLaMA-3.1-8B-Instruct for ICRL implementation, with RMSE as the evaluation metric.  \n\n-- Few-shot examples are selected via stratified sampling (20 per dataset); all other settings follow the main paper.  \n\n-- Table 1 reports results using ESM2 [3] as the FM encoder, which yields highly similar protein representations (average similarity ~0.98).  \n\n-- Table 2 uses ProtBert [4], which produces more distinguishable embeddings (average similarity ~0.92), leading to improved downstream performance.\n\n|     |     |     |     |     |     |\n| --- | --- | --- | --- | --- | --- |\n| _ESM(sim ~ 0.98)_ | ICL | OT-Embed | OT-PCA | OT-Embed + ICL | OT-PCA + ICL |\n| Stability | 0.720 | 0.712 | 0.703 | 0.827 | 0.642 |\n| Fluorescence | 0.995 | 1.322 | 1.222 | 0.997 | 0.987 |\n\n\n|     |     |     |     |     |     |\n| --- | --- | --- | --- | --- | --- |\n| _ProtBert(sim ~ 0.92)_ | ICL | OT-Embed | OT-PCA | OT-Embed + ICL | OT-PCA + ICL |\n| Stability           | 0.720 | 0.631 **(↓0.081)** | 0.644 **(↓0.059)** | 0.673 **(↓0.154)** | 0.577 **(↓0.065)** |\n| Fluorescence        | 0.995 | 1.230 **(↓0.092)** | 1.044 **(↓0.178)** | 0.984 **(↓0.013)** | 0.949 **(↓0.038)** |\n\n\n\n**Analysis:**\n\n-- As shown in Table 1, using ESM as the FM yields limited performance improvements—occasionally surpassing ICL but generally underperforming. In contrast, with ProtBert, which provides more diverse embeddings, even using embedding-level representations alone can achieve performance comparable to or significantly better than ICL.\n\n**Conclusion:**\n\nWhile most experiments in the main paper focus on small-molecule datasets, the key insights derived from them generalize well across modalities and datasets.\n\n\n---\n\n**Q2: Discussion on understanding Inconsistent Results**\n\nThank you for your comment.\n\nTo better understand the inconsistency in ICRL performance, we explored whether training-based strategies could mitigate the model's unfamiliarity with representation-based inputs. \nWhile concurrent work [1] demonstrates that projector pretraining can help LLMs interpret such inputs in language tasks, we found that even with explicit projector pretraining, these strategies still **underperform** in molecular property prediction compared to our training-free approach.\n\nAs analyzed in Section 4 of the paper, ICRL performance is influenced by several factors, with the most critical being the characteristics of the LLM used during inference. Specifically, the inconsistency in performance—such as the inability of embedding-level representations to consistently outperform ICL or the varying effects depending on whether textual examples are included—stems from **the model's unfamiliarity with representation-based inputs**, which are absent during pretraining. While the LLM also lacks domain knowledge about small molecules, it remains highly attuned to _textual_ inputs, making it easier to interpret and utilize them.\n\nOur work focuses on helping the model _understand and utilize unseen input types_ in a training-free manner, allowing it to benefit from expert FM representations without requiring fine-tuning. Traditional methods can achieve similar goals but often rely on extensive pretraining on domain-specific datasets (see Reviewer S7er W1). For example, concurrent work [1] pretrains a projector using molecular captioning data to help LLMs interpret embedding-type inputs. However, this approach requires **~15 hours** of training and does not generalize well to downstream tasks—ultimately performing worse than our training-free method.\n\nTo further explore this issue, we designed a contrastive training strategy inspired by CLIP [2]. It aligns LLM hidden states by minimizing the distance between the final token representations when receiving molecular inputs either in SMILES form or as projected embeddings. This strategy can also be combined with pretraining to enhance the model's ability to interpret representation-based inputs.\n\n**Setup:**\n\n-- All methods use LLaMA-3.1-8B-Instruct as the base model, with a single-layer linear projector of hidden size 4096, consistent with [1].  \n\n-- The pretraining strategy from [1] uses the Language + Molecules-24 (LPM24) dataset [5]. Our contrastive learning strategy is trained on the training split of each evaluation dataset and evaluated on the LPM24 test set for molecular caption generation.  \n\n-- Regression tasks are evaluated on Solubility_AqSolDB and ESOL using RMSE. Other experimental settings follow those in the main paper.\n\n|     |     |     |     |\n| --- | --- | --- | --- |\n|     | Caption | Contrastive | Caption + contrastive |\n| BLEU-4 | 35.29 | 21.42 | 37.31 |\n| rouge1 | 0.551 | 0.320 | 0.592 |\n| rougeL | 0.373 | 0.237 | 0.369 |\n\n|     |     |     |     |     |     |\n| --- | --- | --- | --- | --- | --- |\n|     | Caption | Contrastive | Caption + contrastive | **OT-PCA** | **OT-PCA + ICL** |\n| ESOL | 1.256 | 1.372 | 1.213 | **1.140** | **1.094** |\n| Solubility_AqSolDB | 3.030 | 2.915 | 3.805 | **2.411** | **2.385** |\n\n**Analysis:**\n\n-- As shown in the table, even with projector training designed to help the LLM better interpret representation-based inputs, the performance on regression tasks remains poor—**significantly worse** than that of the training-free ICRL approach.\n\n-- Improved molecular understanding or caption generation ability in LLMs does **not** necessarily translate to better performance on regression tasks, which is consistent with our findings in Reviewer S7er W1.\n\n**Conclusion:**\n\nThe inconsistent results of ICRL across settings stem from the LLM's inherent difficulty in interpreting and utilizing representation-based examples. While traditional methods rely on time- and resource-intensive training to mitigate this, such familiarity **does not guarantee improved downstream performance**. In contrast, ICRL enables the LLM to acquire a certain level of understanding of FM representations **at test time**, offering a lightweight and effective alternative.\n\n---\n\n[1] Vector-ICL: In-context Learning with Continuous Vector Representations. ICLR 2025.\n\n[2] Learning Transferable Visual Models From Natural Language Supervision. ICML 2021.\n\n[3] Evolutionary-scale prediction of atomic-level protein structure with a language model. Proceedings of the National Academy of Sciences 2021.\n\n[4] ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 2022.\n\n[5] L+M-24: Building a Dataset for Language + Molecules. ACL 2024.\n\n---\n**_As reflected in our title, ICRL serves as a case study in our broader effort to explore how LLMs can reason over non-text modalities in a training-free manner. We sincerely thank all reviewers for their thoughtful feedback and will incorporate the suggestions and insights from this rebuttal into the next revision and future work._**"}, {"review_id": "eNgYcoqFCE", "reviewer": "Reviewer_AQ11", "summary": "The paper presents in-context representation learning (ICRL), a training-free method to integrate non-text modalities into text LLMs. Empirical validation focuses on tasks in the molecular domain.", "questions": "1. \\[Abstract\\] Major findings are missing in the abstract. The current abstract only presents questions, but not answers.   \n2. \\[Line 44\\] If it does not outperform ICL, why should we bother? Shouldn’t we expect performance gain by leveraging modality-specific FM?   \n3. It’s unclear how those projected vectors serve as input to the LLM. Do you just str(vec) and use it as text input to the LLM? Or use that as the embedding for a particular token?", "limitations": "yes", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper explores a new setting: leverage tokens from other modalities without training.\n\nWeaknesses\n1. Despite the new setting (training-free), I’m concerned about the actual value of the training-free approach. Most cross-modality adaptors are lightweight (such as a simple MLP), so compute is probably not a major concern here. If we assume API-access only, we probably don’t have the ability to pass in embeddings directly either.  \n2. The method is only validated in the chemistry domain, so I don’t think it’s a good fit for a general technical venue like NeurIPS.   \n3. The caption for Figure 1 is missing, which makes it harder to understand.  \n4. Figure 1 can be simplified, now it shows too many things without clear clue guidance. For example, what’s the relationship between the two sections (text level & embedding level)?", "was_revised": true, "final_justification": "I've read the rebuttal and other reviewers' comments. My major concerns are still not addressed, so I'll keep my rating.", "rebuttal": "We sincerely thank Reviewer AQ11 for the valuable comments. For the concerns and questions, we provide the following responses.\n\n**W1: Questionable Value of Training-Free Setting**\n\nThank you for your comment.\n\nWhile many cross-modality adapters are indeed lightweight, their effectiveness heavily depends on both the base LLM and the downstream task. For example, in molecular property prediction, **ICRL achieves comparable or even better performance** than lightweight fine-tuning or short-duration pretraining + fine-tuning methods—all at **similar inference cost** and in a **training-free** manner (see Reviewer S7er W2, Reviewer J6ma Q2).\n\nAlthough simple lightweight tuning can improve downstream performance, its effect is limited when the base model has not been pretrained on molecular corpora. In contrast, ICRL enables a general-purpose LLM to surpass these methods with only ~2 seconds of test-time computation per sample, showcasing both the **efficiency** and **effectiveness** of the approach.\n\nMore importantly, while fine-tuning is relatively inexpensive compared to full pretraining, it still requires **substantial GPU resources** and **hours to weeks of training time** whenever a new downstream task, FM, base LLM, or dataset is introduced. In contrast, **ICRL can be seamlessly applied to any new modality** and benefits directly from improvements in both the FM and LLM (see Reviewer J6ma Q1, Appendix E.5). The only additional cost is a **CPU-level** alignment step, which takes **under 2 seconds**, yet yields performance comparable to or even exceeding many traditional approaches.\n\nAdditionally, for closed-source LLMs accessed via API, we can still use open-source tokenizers to decode pre-aligned embeddings into text form and feed them as inputs. We plan to include experiments in this direction in future work.\n\n\n---\n\n**W2: Chemistry-Only Validation isn't a Good Fit for NeurIPS**\n\nThank you for your comment.\n\nOur experiments go beyond small molecules and include multiple modalities such as **proteins** (Lines 334–340, Appendix E.3), **vision**, and **audio** (Lines 341–353, Appendix E.6). We have also added **question answering** and **generation tasks** (see Reviewer aUho W2), demonstrating both the **effectiveness** and **generality** of our approach across domains.\n\nMoreover, evaluating a method within the chemistry domain **does not inherently disqualify** it from NeurIPS. Numerous impactful works have been accepted at NeurIPS while focusing exclusively on molecular or chemistry-related problems [1, 2, 3].\n\n---\n\n**W3: Issues about Figure 1 Caption and two Level injection methods.**\n\nThank you for your comment.\n\nWe would like to clarify that Figure 1 illustrates the differences between (i) traditional ICL methods, (ii) time-consuming, projector-trained multimodal LLMs, and (iii) our training-free pipeline. Due to space limitations and the intuitive nature of the comparison, we placed this figure in the Introduction (Lines 36–47), where it is explained in detail.\n\nAs reflected in the paper title, our goal is to explore how LLMs can reason over non-text modalities in a training-free manner. The proposed _text-level_ and _embedding-level_ strategies represent two distinct approaches to this problem. Text-level injection is more intuitive but suffers from limitations such as context window constraints. As a result, we started our investigation with text-level methods and subsequently extended our analysis to embedding-level injection (see Lines 48–57 and 116–120 for detailed discussion).\n\n---\n\n**Q1: Issues about Abstract.**\n\nThank you for pointing this out. We will revise the abstract in the next version to include a clearer articulation of the research problem and the corresponding conclusions.\n\n---\n\n**Q2: Issues about performance.**\n\nThank you for your comment.\n\nWe would like to clarify that the sentence in the paper—\"our primary objective is not to outperform ICL but to investigate the feasibility of adaptively integrating non-text FM representations into a text-based LLM in a training-free manner\"—is **not** meant to suggest that performance improvements are out of reach. Rather, it emphasizes our focus on methodological contribution.\n\nIn practice, LLMs are not inherently capable of interpreting FM-derived embeddings, and even domain-specific fine-tuning of projectors often fails to bridge this gap effectively (see Reviewer J6ma Q2). As a result, embedding-only injection can occasionally underperform compared to ICL. However, when used in conjunction with textual examples, ICRL **consistently outperforms** standard ICL, with improvements reaching up to **16.6%** (Lines 223–233).\n\nMoreover, in scenarios where the LLM lacks sufficient internal knowledge (Lines 235–244, Appendix E.5), and in certain domains such as **proteins** (Reviewer J6ma Q1) and **molecular QA** (Reviewer aUho W2), even embedding-only injection achieves **better performance than ICL**, further underscoring the **effectiveness** and **practical value** of our method.\n\n---\n\n**Q3: Issues about how those projected vectors serve as input to the LLM**\n\nThank you for your comment.\n\nAs indicated by our title, the goal of this work is to explore **training-free methods** for enabling text-based LLMs to understand information from non-text modalities through FM-derived representations. To this end, we propose two complementary strategies for injecting FM features into LLMs:\n\n**Text-level injection**: The projected FM representation is serialized into a string and fed into the model as part of the text input (Lines 48–52, 107–114).\n\n**Embedding-level injection**: The projected FM representation is directly concatenated with the token embeddings of other text inputs (Lines 52–57, 116–159).\n\nWe further compare these two strategies under different conditions—e.g., whether the FM representations are used alone or together with original textual features such as SMILES (Lines 208–280)—and analyze the underlying mechanisms behind their performance differences (Lines 281–322).\n\nIn summary, this work investigates **whether there exist lower-cost alternatives** to the conventional pretraining–fine-tuning paradigm that can still allow LLMs to interpret and utilize non-text FM representations. To this end, we present both text-level and embedding-level injection strategies. The embedding-level approach, in particular, not only enhances ICL when combined with textual examples, but also achieves performance **comparable to or better** than time-consuming trained models when used on its own (see Reviewer S7er W1, Reviewer J6ma Q1, Reviewer aUho W2).\n\n---\n\n[1] GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning. NeurIPS 2023.\n\n[2] Self-Supervised Graph Transformer on Large-Scale Molecular Data. NeurIPS 2020.\n\n[3] DrugCLIP: Contrastive Protein-Molecule Representation Learning for Virtual Screening. NeurIPS 2023.\n\n---\n**_As reflected in our title, ICRL serves as a case study in our broader effort to explore how LLMs can reason over non-text modalities in a training-free manner. We sincerely thank all reviewers for their thoughtful feedback and will incorporate the suggestions and insights from this rebuttal into the next revision and future work._**"}, {"review_id": "CvgOMrivCY", "reviewer": "Reviewer_aUho", "summary": "This work explores methods to integrate the multimodal representations into LLMs in a training-free manner. To this end, the authors propose In-Context Representation Learning (ICRL) by mainly exploring diverse transformations of multimodal embeddings based on the in-context learning framework. To transform the multimodal embeddings, two main categories are devised.\n- Text-level Injection: Directly represent the embeddings in a textual form (i.e., a sequence of real values) after reducing the dimension using PCA, then let LLMs tokenize the textual inputs by themselves.\n- Embedding-level Injection: Forwarding the multimodal embeddings as token embeddings with four methods.\n   - Zero-Pad: Apply zero padding to match the dimensionality of the LLM embedding space.\n   - Random Projection: Randomly initialized projector\n   - Optimal Transport: Adjust the embedding from randomly initialized projectors to a target distribution. In this paper, the mean of token embeddings of SMILES (OT-Embed) or reduced embeddings by PCA (OT-PCA).\n   - Random Noise: A baseline to check whether the model actually understands the multimodal embeddings.\n\nThe authors provide a theoretical analysis of the information loss of nonlinear activations in the projector layers, demonstrating that nonlinear activations could lead to a restricted embedding space.\n\nProposed methods are evaluated for the molecular modality on the tasks related to absorption and toxicity of molecules. The authors investigate diverse aspects of ICRL, uncovering the effective method, showing the effect of the model size, PCA dimensions, nonlinear activation functions, number of examples, and so on.", "questions": "- How to select the in-context examples? Are they sampled randomly for each run?\n- Could the authors elaborate on why the Pearson's correlation coefficient is used for metrics?", "limitations": "As mentioned in the weaknesses section, I suggest evaluating with other MLLMs on molecular captioning and structural understanding tasks to verify whether LLMs leverage the multimodal embeddings.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper explores a novel and interesting point of view.\n- This work explores diverse and reasonable methods for transforming the multimodal representations in a training-free manner.\n\n**Weaknesses**\n\nI list my initial concerns below:\n- The baseline is limited to ICL. It would be better to compare with the molecular LLMs that are directly fine-tuned with the molecular embeddings [1, 2, 3], as an upper bound of the proposed method. Additionally, the embedding-level analysis between the proposed training-free methods and the fine-tuned embeddings would be helpful to show the technical bottleneck of the training-free methods and provide insights to develop future works.\n- The target tasks are limited to specific molecular properties, hindering understanding of the molecular modality. To verify whether LLMs understand and leverage the molecular modality, I recommend evaluating on molecular captioning such as ChEBI-20 [4] or QA benchmark, especially for structural understanding, such as MoleculeQA [5].\n- For metrics, I am concerned that using Pearson's correlation coefficient does not show the performance of the proposed methods. I suggest reporting RMSE values for all experiments.\n\n[1] Li, Sihang, et al. \"Towards 3d molecule-text interpretation in language models.\" ICLR, 2024.\n\n[2] Park, Jinyoung, et al. \"LLaMo: Large Language Model-based Molecular Graph Assistant.\" NeurIPS, 2024.\n\n[3] Kim, Dongki et al. \"Mol-llama: Towards general understanding of molecules in large molecular language model.\" ArXiv, 2025.\n\n[4] Edwards, Carl, et al. \"Translation between molecules and natural language.\" EMNLP, 2022.\n\n[5] Lu, Xingyu, et al. \"Moleculeqa: A dataset to evaluate factual accuracy in molecular comprehension.\" EMNLP, 2024.", "was_revised": true, "final_justification": "In terms of originality, this work presents a new perspective, exploring the training-free method with in-context learning schemes.\n\nWhile it does not fully compare with fine-tuned baselines, I acknowledge that the proposed method offers originality and efficiency, making it a valuable contribution to the field.\n\nAccordingly, I will increase my rating.", "rebuttal": "We sincerely thank Reviewer aUho for the valuable comments. For the concerns and questions, we provide the following responses.\n\n---\n\n**W1: Using Fine-Tuned Molecular LLMs as an upper bound**\n\nThank you for your insightful comment.\n\nAfter carefully reviewing the three referenced papers [1, 2, 3], we would like to clarify that their core methodology involves appending **2D or 3D** molecular representations—via trainable modules—to **1D SMILES** embeddings, thereby enhancing the LLM's ability to interpret molecules. In contrast, our current work focuses specifically on analyzing the **1D SMILES input scenario**, and thus the pretrained molecular LLM embeddings from these models are not directly comparable to our setup.\n\nThat said, we truly appreciate the reviewer's suggestion of using trained molecular LLMs capable of processing higher-dimensional inputs as an upper-bound reference for ICRL. In this spirit, we have designed two experimental strategies:\n\n-- **Concatenated embeddings**: Combining embeddings from multiple molecular views (1D/2D/3D) into a single vector as the FM representation.\n\n-- **Separated embeddings**: Treating the additional dimensions as supplementary context to the original 1D SMILES-based input.\n\nDue to time and computational resource constraints, we were unable to fully implement and analyze these experiments during the rebuttal period. **_However, we are committed to incorporating these directions—including references to related work—in a future version of the paper._**\n\nIt is also worth noting that while recent molecular LLMs are trained to better understand molecular structures, this does **not** imply that they directly perform well on downstream tasks such as property prediction. For example, we experimented with training a projector to connect FM embeddings with the LLM, which resulted in **stronger molecular captioning performance**—but significantly **worse performance on property prediction** compared to our training-free OT-PCA approach (see Reviewer J6ma Q2). Similarly, our literature review shows that pretraining strategies focused on molecular understanding, and even lightweight fine-tuning methods, often **underperform compared to ICRL** in downstream tasks (see Reviewer S7er Q1).\n\n---\n\n**W2: Limited Task Diversity such as molecular captioning and QA benchmark.**\n\nThank you for your insightful comment.\n\nWe agree that broader task coverage is important for evaluating the generality of ICRL. \nIn response, we have incorporated additional evaluations on the molecular QA benchmark [4] and molecular captioning [5] tasks. \nThese new results help us better understand the capabilities and behavior of ICRL under different task types. **_We are committed to including these findings and discussions in a future version of the paper._**\n\n**Setup:**\n\n-- All methods use LLaMA-3.1-8B-Instruct as the base model. For the ChEBI-20 dataset, in-context examples are randomly sampled from the training set at each run. For MoleculeQA, the examples are uniformly sampled from the training set during each evaluation. To ensure fairness, we disabled batch querying in our implementation, as it was not used in baseline methods. Other settings follow the original paper.  \n\n-- For the captioning task, we report BLEU-4, ROUGE-1, and ROUGE-L as evaluation metrics. For the QA task, we use accuracy. Baseline results in the table are taken from [4] Table 8 and [5] Table 1, respectively.\n\n|     |     |     |     |     |     |\n| --- | --- | --- | --- | --- | --- |\n| _MoleculeQA_ | Structure | Source | Property | Application | Avg |\n| Llama-2-7B-chat (L-FT) | 28.75 | 39.84 | 31.33 | 27.71 | 31.54 |\n| ICL | 35.03 | 27.04 | 24.62 | 28.69 | 28.85 |\n| **OT-PCA (ours)** | **51.32** | **37.66** | **33.71** | **31.02** | **38.43** |\n| **OT-PCA + ICL (ours)** | **50.60** | **43.52** | **23.47** | **29.97** | **36.89** |\n| MolT5-base (FT) | 58.01 | 65.85 | 45.14 | 42.24 | 55.39 |\n\n|     |     |     |     |     |     |\n| --- | --- | --- | --- | --- | --- |\n| _ChEBI-20_ | Transformer | ICL | **OT-PCA (ours)** | **OT-PCA + ICL (ours)** | MolT5-Base (FT) |\n| BLEU-4 | 0.027 | 0.133 | **0.147** | **0.196** | 0.457 |\n| rouge1 | 0.204 | 0.393 | **0.353** | **0.407** | 0.634 |\n| rougeL | 0.186 | 0.310 | **0.274** | **0.353** | 0.578 |\n\n**Analysis:**\n\n-- As shown in Table 1, the training-free ICRL approach outperforms both standard ICL and fine-tuned general-purpose LLMs on the QA task. This result demonstrates that, although there remains a performance gap compared to expert models, the processed FM representations **can indeed be interpreted and partially utilized** by a **text-based LLM**, even one that has never encountered representation-type inputs. Interestingly, in this task, embedding-level injection alone performs better than when combined with text features, suggesting that the injected representation serves as a more effective signal than the original SMILES strings in helping the LLM understand molecular structure.\n\n-- As shown in Table 2, despite the general LLM lacking inherent capabilities in molecular captioning, the embedding-level OT-PCA approach still **outperforms standard ICL** and effectively enhances the model's ability to generate captions from text features. This again supports the effectiveness and generalizability of our method, even on generative tasks.\n\n**Conclusion:**\n\nOverall, while ICRL performance is inherently influenced by the capabilities of the underlying FM and LLM on specific tasks, our experiments on the QA benchmark and molecular captioning clearly show that the injected representations can be **meaningfully used by general-purpose LLMs**. Moreover, ICRL significantly **outperforms lightweight tuning baselines**, which is consistent with our findings on regression tasks (see Reviewer S7er Q1).\n\n---\n\n**W3: Inadequate Metrics — Recommend Reporting RMSE instead of Pearson's r**\n\nThank you for your comment.\n\nWe would like to clarify that for all regression experiments in the paper, we evaluated performance using RMSE, Pearson's _r_, and Spearman's ρ. The conclusions we reported are consistent across all three metrics (see Line 210 and Footnote 2). Due to space limitations and the growing trend in regression evaluation to include correlation coefficients for a **more comprehensive view** [6,7], we chose to report Pearson's _r_ in some cases. Full results using all metrics, along with relevant discussions, are provided in Appendix F.2.4.\n\n---\n\n**Q1: How to select the in-context examples**\n\nThank you for pointing this out.\n\nIn all our regression experiments, we selected in-context examples via **stratified sampling** from the training set. Specifically, for each run, we partition the training set into equal-sized bins based on the number of examples required, and then randomly sample one example from each bin. **_We will include this implementation detail in the revised version of the paper._**\n\n---\n\n[1] Towards 3d molecule-text interpretation in language models. ICLR 2024.\n\n[2] LLaMo: Large Language Model-based Molecular Graph Assistant. NeurIPS 2024.\n\n[3] Mol-llama: Towards general understanding of molecules in large molecular language model. ArXiv 2025.\n\n[4] Moleculeqa: A dataset to evaluate factual accuracy in molecular comprehension. EMNLP 2024.\n\n[5] Translation between molecules and natural language. EMNLP 2022.\n\n[6] Regression Transformer enables concurrent sequence regression and generation for molecular language modelling. Nature Machine Intelligence 2023.\n\n[7] Gradient Aligned Regression via Pairwise Losses. ICML 2025.\n\n---\n**_As reflected in our title, ICRL serves as a case study in our broader effort to explore how LLMs can reason over non-text modalities in a training-free manner. We sincerely thank all reviewers for their thoughtful feedback and will incorporate the suggestions and insights from this rebuttal into the next revision and future work._**"}, {"review_id": "TcAiMYn3DD", "reviewer": "Reviewer_nfkW", "summary": "The authors introduce In-Context Representation Learning (ICRL), a training-free strategy that lets a text-only LLM consume continuous embeddings from a non-text foundation model (FM) at inference time. They treat the merged prompt as an extended few-shot context, rather than adding learned projection heads or fine-tuning the LLM. Experiments focus on molecular property regression where Uni-Mol supplies the embeddings and Llama-3.1-70B does the prediction and in-context learning.\n\nThrough out the paper, the author proposed and tested various ways to align the FM's output with the LLM. PCA then text injection is considered as means to shorten the input length, embedding space engineering (including optimal transport, Zero-Pad, Random Projection) are tested to align the 2 spaces with geometric intuitions.", "questions": "Besides the questions I raised about weaknesses, I have this additional question:\n\nThe studies conducted are nice and thorough, I really appreciate that. But my main concern would be on the motivation: suppose one can do a lightweight training (either with the projector or a LoRA on base model), it is a one-time cost, how does that compare to doing optimal transport for each sample everytime in inference? Can ICRL provide some advantage that fine-tuning cannot offer? for example prevent overfitting and outperform it with many-shot ICL? I wonder what the authors think about this.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengh**\n1. This paper is highly insightful, providing plenty of useful guidance for training-free representation alignment.\n2. The main question, \"Can LLMs Reason Over Non-Text Modalities in a Training-Free Manner?\" is nicely laid out and examined thorough out the paper. I enjoyed reading it.\n3. The ablations are extensive: almost all configurations are examined and evaluated. The empirical effect is clearly analyzed.\n\n**Weakness**\n1. Lack a baseline where the projector (or the models + LoRA) is slightly trained, so we might know the trade-off from training-free versus with training.\n2. The task / modality is limited. Molecules are encoded as text (SMILES). Vision/audio claims are relegated to appendix and lack quantitative depth. And on this point, evaluation focuses on regression RMSE; no classification or generative tasks.", "was_revised": true, "final_justification": "My questions were on the motivation of the paper: why do traininig-free when light finetuning is avaliable?\n\nI think the authors have answered my questions in that perspective, it's an early explorative work, I find the case studies interesting and can be of great insights to the community exploring multi-modality infusion with LMs.\n\nHence I maintain my score of 4 and leaning towards accepting this paper.", "rebuttal": "We sincerely thank Reviewer nfkW for the valuable comments. For the concerns and questions, we provide the following responses.\n\n---\n\n**W1: Missing Lightly-Trained Baseline for Trade-off Analysis**\n\nThank you for the insightful comment. To better highlight the differences between **training-free** and **conventional training-based** methods, we designed experiments from two perspectives and conducted a comprehensive analysis.\n\nFirst, considering that ICRL's performance bottleneck largely stems from the LLM's unfamiliarity with embedding-style inputs and its lack of molecular knowledge, we implemented two strategies for training the projector connecting the FM and the LLM. The first is inspired by CLIP, aligning the FM-projected embedding with the LLM's intermediate hidden states after processing the corresponding SMILES input. The second strategy involves pretraining the projector on a molecular captioning dataset. \n\nWhile both methods improved the LLM's ability to interpret FM-derived inputs, they **still underperformed compared to our training-free ICRL approach** on downstream molecular prediction tasks (detailed in Reviewer J6ma Q2).\nFurthermore, we surveyed relevant literature and included comparisons with a diverse set of training-based methods:  \n-- instruction finetuning with Q-LoRA,  \n-- fully supervised pretraining + finetuning, and  \n-- unsupervised pretraining followed by supervised finetuning.\n\nWe conducted a detailed comparison in terms of training paradigms, required computational resources, training time, and performance on two benchmark datasets. We found that **ICRL achieves comparable or even better results than these methods at the same inference cost—all without any training** (see Reviewer S7er W1 for details).\n\nIn other words, while lightweight tuning can indeed improve performance to some degree, its effectiveness is limited when the base model is not pretrained on molecular data. In contrast, ICRL allows a general-purpose LLM to surpass these training-based methods with only **~2 seconds of test-time cost**, demonstrating both the **practicality and effectiveness** of our approach.\n\n---\n\n**W2: Narrow Modalities and Task Types**\n\nThank you for the insightful comment.\n\nDue to space constraints, we focused our main analysis on **molecular** modality. However, we have also included additional results for **proteins** (Appendix E.3) as well as **vision** and **audio** (Appendix E.6), and **_we are committed to providing the same level of in-depth analysis for these modalities in future versions._**\n\nImportantly, the insights gained from our analysis on small molecules **generalize well to other modalities.** For example, as discussed in Section 5, high intra-set similarity in FM representations can hinder ICRL performance. Our follow-up experiments on the protein modality confirm this: by replacing the FM encoder with one that produces **more distinguishable representations**, we observe **significant improvements** in ICRL performance (see Reviewer J6ma Q1).\n\nWe also acknowledge that our initial evaluation primarily involved **regression tasks** and a few **classification tasks**. To address this, we have added more classification tasks (e.g., molecular QA) and generation tasks (e.g., molecular captioning) to more comprehensively evaluate ICRL's effectiveness. Specifically, on the QA benchmark, even embedding-level injection alone significantly **outperforms both fine-tuned general LLMs and ICL**. For captioning, although there is still a performance gap relative to expert models, representations injected via OT-PCA consistently outperform SMILES-based text examples and help the LLM better interpret unfamiliar inputs (detailed in **Reviewer aUho W2**).\n\n---\n\n**Q1: Unclear Motivation Over Fine-Tuning**\n\nThank you for pointing this out. **_We acknowledge that the original manuscript did not elaborate on the motivation in sufficient depth, and we will revise the discussion accordingly in the next version._**\n\nFirst, we would like to clarify that embedding-level injection methods such as OT-PCA, similar to lightweight training, require **only a one-time computation** of alignment parameters. At inference time, applying the projection amounts to a single matrix multiplication—even when using over a hundred examples, the total cost is **well under one second**, and can be ignored in practice. In fact, computing the OT alignment parameters for ICRL takes **less than 2 seconds on CPU**, which is orders of magnitude faster than even the most lightweight fine-tuning setups that typically require **at least several hours on 2–4 GPUs**.\n\nMore importantly, our latest experiments demonstrate that **ICRL achieves performance comparable to, or even exceeding, lightweight training approaches** across multiple datasets, modalities, and task types (see Reviewer S7er W2, Reviewer J6ma Q2). While large-scale pretraining + fine-tuning paradigms can yield strong performance, they often demand days or weeks of training.\n\nCrucially, both lightweight and large-scale training methods are inherently **task-, FM-, and LLM-specific**—any update in task formulation, foundation model, or dataset requires costly re-training or re-tuning from scratch. In contrast, **ICRL can be seamlessly applied to new modalities**, and **benefits directly from improvements in both FM and LLM backbones** (see Reviewer J6ma Q1, Appendix E.5), all with a negligible CPU-only alignment step. Despite its minimal cost, ICRL achieves **comparable or better results** than many training-based approaches.\n\nCompared to traditional **many-shot ICL**, ICRL also offers superior generalization. Even when the LLM lacks domain-specific knowledge, ICRL can **leverage FM-derived signals to outperform standard ICL** (see Reviewer aUho W2). More importantly, for modalities like **vision** and **audio**, general-purpose text-based LLMs **cannot handle the inputs at all**, while ICRL remains effective—bridging these gaps (see Appendix E.6).\n\nIn summary, ICRL offers a **highly cost-effective, training-free alternative** to fine-tuning, providing a practical way to leverage the growing pool of open-source LLM and FM weights in the AI community. We believe it opens a promising direction for multimodal research that challenges the conventional reliance on large-scale training.\n\n---\n**_As reflected in our title, ICRL serves as a case study in our broader effort to explore how LLMs can reason over non-text modalities in a training-free manner. We sincerely thank all reviewers for their thoughtful feedback and will incorporate the suggestions and insights from this rebuttal into the next revision and future work._**"}]

Author rebuttal:

Meta-review: This work introduces the training-free strategy of In-Context Representation Learning that lets a text-only LLM consume continuous embeddings from a non-text foundation model at inference time. The merged prompt is treated as an extended few-shot context. Results focus on molecular property regression but also show results for vision and audio. Uni-Mol is used as the foundation model and Llama-3.1-70B is used as the LLM.

Strengths:
* The paper is very insightful and novel.
* The main question is well laid out and motivated and easy to understand.
* The experimental setup is extensive and the effect well analyzed.

Weaknesses:
* The work would benefit from more baselines especially with various projectors.
* The modalities analyzed are limited. The focus is on molecules in SMILES format with light analysis on vision/audio generalization. It would be good to have stronger vision/audio evaluations with 2D/3D molecular formats.

The idea of adapting to other modalities without any training is novel and important as large open-weights models are generally text only and adapting them via fine-tuning to new modalities is prohibitively expensive for most users. Given the successful experimental results in this paper (on chemistry and other modalities) compared to other projector-based approaches I expect it will be impactful to the field by enabling the use and understanding of large models for more researchers working on non-text modalities. I recommend this paper be accepted.

In the discussion, there was debate around ICL results being better than the proposed approach. If the authors moved table 16 into the main part of the paper (where they show the approach is better combined with ICL than ICL alone), that would have resolved the discussion earlier. There was also discussion around various projectors and if they could improve the results (it seems they mostly don't but evaluation there could be more comprehensive). Reviewer AQ11 argued for rejection, however, there were factual inaccuracies in the review (claiming that only the chemistry modality was validated), the review was mostly superficial and the reviewer didn't engage in the discussion.

Finally, there was discussion around additional task types and modalities, the evaluation with other modalities could be more thorough and the authors ran additional experiments on new task types. I appreciate the authors putting in the additional time and effort to get additional results for the rebuttals, these seem to have resolved most of the concerns from the authors.
Decision: Accept (poster)