Boltzmann Semantic Score: A Semantic Metric for Evaluating Large Vision Models Using Large Language Models
Authors: Ali Khajegili Mirabadi
Katherine Rich
Hossein Farahani
Ali Bashashati
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Large Vision Models
Semantic Evaluation
Computational Pathology
Medical Imaging
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: A Semantic Metric for Evaluating Large Vision Models Using Large Language Models and A Large Collective Database of Paired Image and Pathology Reports

Abstract: Do Large Vision Models (LVMs) extract medically and semantically relevant features similar to those identified by human experts? Currently, only biased, qualitative approaches with limited, small-scale expert evaluations are available to answer this question. In this study, we propose the Boltzmann Semantic Score (BSS), a novel method inspired by state space modeling, to evaluate the encoding space of LVMs from medical images using the encoding space of Large Language Models (LLMs) from medical reports. Through extensive experimentation on 32 datasets from The Cancer Genome Atlas collection using five state-of-the-art LLMs, we first establish a baseline of LLMs' performance in digital pathology and show that LLMs' encoding can be linked to patient outcomes. Then, we compared seven LVMs with BSS and showed that LVMs suffer from poor semantic capability when compared with encoded expert knowledge from pathology reports.
We also found statistically significant correlations between BSS (as a measure of structural similarity) and performance in two downstream tasks: information retrieval and survival prediction tasks. Our study also investigates the consensus among LLMs in evaluating LVMs using BSS, indicating that LLMs generally reach substantial consensus in rating LVMs, with some variation dependant on the cancer type. We believe the BSS metric proposed here holds significant potential for application in other domains with similar contexts. Data and code can be found in \footnotesize \url{ https://github.com/AIMLab-UBC/Boltzmann}

Published as a conference paper at ICLR 2025
BOLTZMANN SEMANTIC SCORE: A SEMANTIC METRIC FOR EVALUATING LARGE VISION MODELS USING LARGE LANGUAGE MODELS
Ali Khajegili Mirabadi, Katherine Rich, Hossein Farahani, & Ali Bashashati School of Biomedical Engineering & Department of Pathology and Laboratory Medicine The University of British Columbia (ali mimmmaraaa.. ali. www.aahaataaia i @uube.c. ca.
ABSTRACT
Do Large Vision Models (LVMs) extract medically and semantically relevant features similar to those identified by human experts? Currently, only biased, qualitative approaches with limited, small-scale expert evaluations are available to answer this question. In this study, we propose the Boltzmann Semantic Score (BSS), a novel method inspired by state space modeling, to evaluate the encoding space of LVMs from medical images using the encoding space of Large Language Models (LLMs) from medical reports. Through extensive experimentation on 32 datasets from The Cancer Genome Atlas collection using five state-of-the-ar LLMs, we first establish a baseline of LLLM'' performance in digital pathology and show that LLMs' encoding can be linked to patient outcomes. Then, we compared seven LVMs with BSS and showed that LVMs suffer from poor semantic capability when compared with encoded expert knowledge from pathology reports. We also found statistically significant correlations between BSS (as a measure of structural similarity) and performance in two downstream tasks: information retrieval and survival prediction tasks. Our study also investigates the consensus among LLMs in evaluating LVMs using BSS, indicating that LLMs generally reach substantial consensus in rating LVMs, with some variation dependant on the cancer type. We believe the BSS metric proposed here holds significant potential for application in other domains with similar contexts. Data and code can be found in https://gittuut hom.ttt..ocoooooooooooommoooottt MMMabb- JBC/ /ool 3zman
INTRODUCTION
To even casual observer, Large Vision Models (LVMs), and Large Language Models (LLMs) seem poised to revolutionize medicine. However, despite their rapid adoption in other domains, they have yet to make substantial impact within the medical field (Omiye al.] (024). Currently, they are notably absent from any direct clinical applications. This is due to a variety of distinct challenges inherent to medical data: privacy, lack of large and diverse datasets, ethical considerations, licensing, and the inherent complexity of the domain (Bouderhem) 2024 Jiang et al. (021). This complexity makes interpreting the results of large models incredibly challenging, with researchers often resorting to qualitative assessment of the model's 's response to specific and limited inputs.
The standard method used in research studies is to recruit medical experts to examine the model's attention maps or activation layers as way to determine if they are capturing clinically relevant information. However, this approach has inherent variation due to the limited number of medical experts, and inter//intra-obseveer variability. When using these approaches, evaluations can also be biased as studies are often limited to small sample sizes. Variability can also originate from the medical expert's 's training, specialty, and experience. Additionally, the diagnostic guidelines for various diseases that medical experts rely on are constantly being updated (Alonso-Coell) et al. 2011). These changes impact the relevancy of older models that were trained on labels produced using older guidelines, and evaluated using those same guidelines.
In standard medical practice, imaging data is typically analyzed by single specialist, except in rare cases that require a collective assessment, In the field of pathology, experts examine the tumor tissue under the mieroscope their corresponding images and document their observations in text-based

---

Published as a conference paper at ICLR 2025
pathology reports.These pathology reports serve as concise summaries of the most significant findings.
Clinicians routinely consult these reports to review patient's ' history or diagnosis, especially when investigating rare cases, where one might be interested in identifying historical patients with similar
profiles to inform care for the patient hand Borowsky & et al., 2020) Evans & et al. 2022 & et al. 202T)
Farooq
With the recent rise in popularity of LVMs in digital pathology, an important question arises: do LVMs extract medically and se- mantically relevant features similar to those identified by human experts? This question remains largely unanswered. However, the rapid advancement of large language mod- els (LLMs) has made it feasible to repre- sent long and complex contexts, such detailed pathology reports as illustrated Figure I using UMAPs (Melnnes et al. 2018). This opens up the possibility of cre- ating large database of medical reports, encoding them with LLMs, and develop- ing a comprehensive resource of encoded medical information. By leveraging this re- source, we can address the question posed earlier. Evaluating LVMs on this extensive can mitigate biases inherent in smaller sample sizes and incorporate input from multiple observers (pathologists), thereby reducing observer variability.
Figure I: The UMAP plot of the encodings of Command upen-source L.I.M. from roughly 9,500 patients' patology reports over 32 different datasets in The Cancer Genome Atlas. Clusters indicate clear differentiation between cancer types in the cacoding space of Command-8. &8, showing the represeutation capability L.M..
To this end, our primary contribution is to propose systematic methodology, independent of the choice of I.I.M or IVM, to evaluate LSVs from a medically semantic perspective. Our methodology leverages L.Ms and large and collective database of medical reports across more than 30 cancer types that represent more than 9,500 patients. We also establish a baseline of LLMs' performance in two large-scale digital pathology tasks.
2 BACKGROUND & RELATED WORKS
The term foundation model was first coined by Bommasani et al (Bommasani et al. 2021) to describe models trained on huge and broad datasets that can be easily adapted to variety of downstream tasks. Since then, multiple foundational models have been introduced for variety of domains. For text,
GPT-3 (Brown et al. 2020 Jamba (Lieber et al. 2024). Gemma-7b (Team et al. [0044), Llama3- 8b Touvron et al. [023] AT Meta 2024). Bio-Llama3-83 (Ankit Pal] (2024). and Command-R For 2024) are a few examples to name. Within computer vision and vision-language,
general examples include CLIP (Radford et al. 2021) and VL-BERT (Su et al. 2019). However, in recent years, beyond generic models such as VIT (Dosovitskiy et al.) 2020) and Swin (Liu et al.
2021), ,, wave of domain-specifiic vision LVMs have been introduced. Within the medical field and more specifically in the histopathology imaging space, examples of LVMs include CTransPath (Wang al. 2022). PLIP (Huang al. 2002), Phikon (Filiot et al. 2023). UNI (Chen al. 2023). and
Virchow ( (Vorontsov et al. 2023).
To evaluate the performance of these models, they are often scored on an ensemble of downstream medical tasks. Common examples of these tasks include tumor grading, tumor subtype classification, image segmentation, and disease classification. While the performance of model on these tasks can give us an idea of how it performs in comparison to other machine learning models, it falls short of being conclusive evidence that the model will achieve similar performance clinical setting. Thus, in order to offer further insight into how the model is 'thinking', researchers will use qualitative techniques such as sitentionmmapping (Bahdanau et al. 20144 or GradCAM (Selvaraju et al. 2017) to highlight which sections of the input data the model finds most important for the relevant task. From there, one can attempt to draw direct comparisons to how a clinician will parse the same data, which is subject to bias.
In the context of medicine, where reports are typically written according to specific clinical guidelines, variability persists due to diverse writing styles, training backgrounds, professional experiences, and
2

---

Published as a conference paper at ICLR 2025
centtuutioo-spccifffe protocols. However, LLMs, owing to their exposure to diverse training text data, are adept at tackling these variabilities (Singhal et al., 2023; Ankit Pal, 2024; Luo et al. 2022] Achiam et al. 2023 Lieber et al. 2024). As result, LLMs demonstrate the potential to encode medical information effectively, providing nuanced representations that encapsulate expert knowledge. Consequently, the encoded medical report representations generated by L.MSs serve as valuable source of medical assessments for each paired image.
Accordingly, we propose the Boltzmann Semantic Score (BSS) where we utilize LLMs to encode medical reports accompanying digital images and then pair the feature spaces of LLM and LVM to measure the similarity of visual representations indicates patient-to- patient relationship, where we expect the same relationship between patients captured by experts (through medical reports) to be captured by LVMs.
the textual representations. This similarity
Here, we argue that BSS is a measure of the structural similarity between the visual embedding space and the text embedding space, where the text embedding space represents medical semantics as the reference. It can be interpreted as a measure of semantic similarity for LVMs within the broader medical domain.
METHOD
This section begins by introducing the intuition behind our work, followed by the problem formulation, and then expands the theory behind the Boltzmann Semantic Score.
3.1 INTUITION
We start with example illustrate the intuition behind our method: consider only three cases- A, B, and C--each comprising paired text and image data in our dataset. Expert annotations through an LLM indicate that case A is semantically closer B than C. reflecting a stronger clinical correspondence between A and B, with a weaker but non-negligible similarity between A and C.
An ideal LVM should reflect these relationships by embedding closer to than to C within its latent space. When this alignment is achieved, the semantic similarity for A (BA) reaches its maximum value (BA- 1), signifying strong semantic agreement with expert-provided annotations. Conversely, if the LVM incorrectly positions A closer to C' than to B, the semantic score for A should decrease. Importantly, due to inherent baseline similarities among cases, the score should not sharply drop to zero but rather be proportional to their energy distribution. With this mechanism, by averaging the semantic score values across the entire dataset, we obtain an overall score that quantifies the LVM's 's alignment with the reference LLM. Using an LLM as the fixed reference, the semantic score obtained allows us to compare different LVMs, highlighting which model best aligns its latent space with the semantic structure of the L.M and most effectively preserves semantic information.
3.2 PROBLEM FORMULATION
For a paired set of n texts and images in a dataset, can represent the set of the text data as T (Te l is an integer, 1 < L S n),, and the set of the image data as I (I, r an integer, 15 r S n.. where T; and I, are the matching pairs for any given r. With this definition, let's denote any given LLM as A(T:) representing a function of input text Tr, and any given LVM as (IZ.) representing a function of input image I,.. Therefore, A and & are
Reforace Shas
Obsever Say Spoct V
snertiiyy Aljhat
defined as,
.N- in-lonniatboiing saae of
Al weightdding
A: L RdixI
Higure state space representation
k-mearest neigh- burs in (left) text space (C) and in (right) the vision space (V). observer is estimating states matches to referece..
(1)
: I V V I
where C and V are the set of dense representations of the input text and image tensors, re- spectively. Similar to T, L {Le is an integer, t < n); and similar to I, y
3

---

Published as a conference paper at ICLR 2025
(V, is integer, I S n). Rd and Rd are the set of d1- and d2 dimensional vector space, respectively. And, d1 and d2 are the number features of the dense representation of and Φ respectively. Note that most often dj # d2-
The end goal here is to find a function F such that assigns a relative semantic score to & using on the paired dataset premised upon their dense representations. Given instance of query 9. expect B, = F(V;: f). where By is the desired score measuring the semantic worth of information extracted from the query image using & compared the query medical text written by human experts embedded by A.
3.3 BOLTZMANN SEMANTIC SCORE
To come with score function F, for the query q, we define the set of the k-nearest neighbors of Lg as pear(Cg: k) [Cy, < Nr S k), and the set of the k-nearest neighbors of V2 as near(V, g. k) [VNy | 1 N S k}. Hereafter, for the sake of simplicity in our notation, we refer to CN, as L, (the i-th nearest neighbor Cg in L), and to VNY as V (the j-th nearest neighbor to Vg in V).
Now, we can define two Star graphs corresponding to each query node in the text and vision spaces: Reference and Observer. For the text, which is the reference, Gr.g is defined with the central node of f, where the leaves are members of near(C,, k). Similarly, the vision space, which is the observer, Gv., is defined with central node of V, and leaves as members of near(V, k). It is worth mentioning that the k-neighbors of query 9 in the text and vision spaces are not essentially equal they are calculated using different text and vision encoders. Example graphs are illustrated in Figure [
Here, Gc,y is a state graph representing the relationship between the text encodings written by human experts after visually inspecting the corresponding images. The individual nodes can be viewed as a quantum state that ties in medical semantics; thus, it is expected that A can properly encode them. That is to say, Gc., represents the relationship between states sharing common information. Thus, if two states/nodes are in close proximity in the state space compared other states/nodes, the semantic similarity between the two states/nodes is more meaningful than the others. Here, Ge.q represents this for the k-nearest states/nodes in the text state space as the reference.
On the other hand, Gv.g is a state graph representing the relationship between the encoding of images produced by 4. If we expect an LVM to perform equivalently human experts, should ideally match Gc.q- In other words, the same relationship between states/nodes captured in Ge.g should be captured by & Gv. meaning that the observer aligns with the reference.
Reference Stote Space:
In statistical mechanics, the quantity repre- senting the variety of states associated with a specific energy level is termed the degree of degeneracy, often referred to simply the level's degeneracy (Shankar 1994). Us- ing this concept, want compare Gv. to G..,, as well as measure their similarity. When comparing these state spaces, if there is a matching state, we consider that as one degree of degeneracy. However, if there is no matching state (a state is in Gy., that is not present in Ge.g) we need to have two degrees of degeneracy to model that. To clarify, let's say C, a non motchingning state in G..q. corresponding with V,, where L; is the true matching state in Gr.q- Thus, there should be mechanism incorporate the randomness between the states C, and f. when measuring the total similarity of the two state spaces, as shown in Figure
Observer Stnte Space:
Pi Pj
Pi Pg
C. reference matching
Cg corresponding non-matching estimated state by
V the estimated state j by 1b in Gy.
Figure 3: The Second order Boltzmann Factor elucidates the resemblance hetween states and g. from i. Essentially, Py serves as metric indicating the degree of similarity in energy between these states, thereby causing the model 4 displace them accordingly.
incorporating
stochastic displacement of states
4

---

Published as a conference paper at ICLR 2025
Having introduced the concept above, we model the distribution of the states with Boltzmann Distribution in the reference state space Gc.q, which dates back to the famous Boltzmann's 's paper in 1877 (Bolizmann 1877) with simplified derivation introduced in (Mäller 2014). This expresses that fractions of energy are distributed over the discrete energy levels of an assembly of states. Therefore, for the central state/node Ly and given neighhor L; in Ge,g. the Boltzmann Factor indicates the similarity of the states (in terms of energy) within the state space, As the Boltzmann factor is solely dependent on the energy difference between states, we can formulate and adopt it for our problem as follows,
Pi exp Pq
AE KT
[[: Coll2 ddz
Boltzmann Factor:
exp
where P; is the probability of state/node Li, and Py is the probability of state/node Lg. For the energy level, AE = [| full2 represents the energy difference of the two states/nodes, and vdi is a constant equivalent to KT in the original formula, where d1, as previously introduced, is the reference space dimensionality.
To clarify this concept of modeling, when 2t approaches 1, it means the two states have similar Py energy levels and can appear in place of each other in the state space. For the reference state space, which is the textual representation, it means that the texts for two patients are characteristically similar. Conversely, when Pi approaches indicates that the two states are not characteristically similar.
However, the Boltzmann Factor in only represents one degree of degeneracy/simillrity between two states. Thus, we add the second order degeneracy for the states that are in near(V,. k) but not present in near(fy, k)) In other words, those states that the observer mistakenly estimated that present reference. Therefore, for a given state j such that V/ E near(V;; k) but L, E nnea (Cys k). and the true matching state C, in the reference, we define the second order Boltzmann Factor as follows.
Second-order Boltzmann Factor: badla = Pi Pi Pj
P4
where Pt shows the similarity between the two state L. and Ly: and P: shows the the degener- Pj acy/similarity between the two state L, and f.g- Consequently, burly includes the degeneracy between L, and Lj when comparing L; with Lgg
Now, since only the presence of states and their energy matters for modeling the energy distributions of the states, we ignore the orders of the states between the two state spaces if their presence correctly
is estimated by the observer. Hence, we define A the set of all the matching states in Gc.,, and Gv.g here:
((i,)) VV E near(Vy, k) & Cy E near(Lq, k), Bi: C, E near(Cq. k) & C== Cz}
(4)
Nevertheless, we define the set of the non-matching states between Gé,g and Gv.g as D:
DD ((1,5) VVV, E near(V k) & Lj & near(fq. k), 3i: V & near(V, k) & La nnar( k))
(5)
For any (i, A, essentially L; Lj. the second order Boltzmann Factors bugle pi Pr
Pg
1 P. 2:, which means that there is only degeneracy between states and q. On the other hand, for Pg P."
any (i,j) ID, see birjle Ps pi Pe which is showing the stochasticity between states and j can affect the presence of the states in proximity to q. With this concept, we introduce the normalized Boltzmann Semantic Score B, for the query g as follows,
Σ (EJ)EA
besla
Σ
bigile
Σ basla
Σ
basla
(IJ))ALD
(EJ)DD
(K,))D
B,
(6)
Σ
busll
bastle
Σ
bisla
(AD)
(A.))AUD
(A.))AUD
Finally, the average Boltzmann Semantic Score is calculated as,
B
L
B
(7)
5

---

Published as a conference paper at ICLR 2025
where |C] is the cardinality of L, and B denotes the average Boltzmann Semantic Score on the entire set. With the formulation above, we expect that B not only represents the semantic capability of using A. but also presents a predictive capability for downstream tasks that uses visual encodings.
EXPERIMENTS
4.1
DATASET & COMPUTE
The largest publicly available pathology archive of datasets, The Cancer Genome Atlas Project (CCGA)(Weinster) et al. 2013). was utilized in this study. TCGA archives include 32 different cancer datasets and roughly 12, 000 patients across more than 160 different centers, including matched Whole Slide Images (WSIs) and pathology reports. For pathology reports, we used the cleaned TCGA-Reports dataset Keffll & Tatonetti 2024). which is the largest publicly available dataset of cleaned pathology reports to date (more details in section Diin Appendix). All experiments were performed on nodes of Intel Xeon E5-4640 with 503GB RAM.
4.2 EVALUATION METRICS
For the evaluation metries. we followed the standard metrics used in (Chen et al.) 2022] Kalra et al. 2020: Wang et al. 2023), including top-1, top-3, and top-5 majority vote accuracy, top-1. top-3, and top-5 FT Score, as well top-1, top-3, and top-5 Average Precision (AP@k) for the retrieval tasks. Additionally, we employed the concordance index (C-index) to assess performance on the survival prediction task.
4.3 LLM REPRESENTATIONS
Recent studies have highlighted the promise of LLMs in various pathology tasks (Wiest et al. 2024 Lammert et al. 2024 Huang et al. 2024). Building on these insights and the findings from the original work on the TCGA reports on the quality of parsed pathology reports and the high performance of language models (Kefeli & Tatonetti 2024), we have designed two additional large-scale experiments focused on content-based information retrieval and survival prediction using only textual data.
In this regard, first encode the original text reports without any perturbation and conduct retrieval test on them within the database. Next, we perturb the input text reports by removing essential keywords, such cancer names subtypes, and then encode the perturbed reports with A and conduct the same retrieval test. By comparing the results of these two experiments, we can determine whether LLMs are capable of properly encoding the medical text reports, focusing on the ability to capture the overall context, not just specific keywords. Finally, to assess whether the LLM representations contain clinically relevant features, we evaluated the original text representations on survival prediction task and studied if they can be linked to patient outcomes.
4.3.
INFORMATION RETRIEVAL
Command R: Areraçe Top Metric for 32 Dataseta ofTOGA
1.0
08
ta Sta
mam
eva
sac
sna
crap
Dalsse
Figure Performance comparison Command-R, the best performing LLM, organ independenn setting content-bused information retrieval using the original text reports.
In this experiment, we used the LLM representations to conduct a retrieval test. The search was conducted in two settings: organ-specific search and organ rrgaaniindepeneen search.
In the organ- specifcc search, we used the cancer type as the query, searching among all cancer types associated with the organ related to the query's cancer type. In the argan-independen search, the cancer type was used as the query across all cancer types in the database (The detailed information on the different cancers and their organ allocations is provided in Table 4 and section D in the

---

Published as a conference paper at ICLR 2025
Table Information Retrieval with 5 LLMs over 32 Datasets of TCGA
(a) Average Performance of Organ- Specifle Search with Original Text Reparts
LLM
Accuracy
F1 Score
AP
top
lop
top
top
top 3
top
lop
tup
top 5
Jamba
4.838
1.823
a
0.900.azz
3.88
0,87
6.838
0. a.7
Gemmaa7h
0.782.019
0.722am2
0.863-044
0.835
n 0.800amm
0.782
a.738
0.708-u21
Llama3-88 Sb
0.8
1.865-317
0
0.911-cn
0.904.4
n
0.888.01
0.862...
0.827
0.300-01
Bio- Noo-Llama--5b
0.800
1,850,,37
0.810
0.909 0.917.ca
0.800.01
0.839
0.819_o.0
Command-R
1.87
1.868
0.839
0.929.ca
0.919aa32
0.912.nts
0.87
a.85)
0 0.835_nus
(b) Average Performance SpecifSpecific Search with Perturhed Text Reports
Cancer
Accuracy
FI Score
AP8
3op
top
top
3op
top
top
top
Jamba
0,827
0.811a33s
L.79
0,808.m.1
0.875
0.858.ai
0.827axas
0,800ua
0. 0.778.nos
Gemmu-7b
0,762
0.726
0.630uxz
a.848
a.81S
0.700uam:
0.762axee
0.736
0.71
Llama3-8h
2.84
0.833nz2
2.906
0.892
0.87
0.877 2.84
0 0.800
Bio-I Jamnal-h
0.838 2.852
0.50amm
2.911
0.897
a.577
2.862
0.832aus
0 20907
Command-R RR
0.846.) 0.848
0.15.m0
0.908.om1
0.900
0.883.01
0.830
0 0.804_
Average Performance Organ- Independent Search
Original Text Reports
LLM
Accuracy
FI Score
AP
top
3
top
top
top 3
top 5
top
tup 3
5
Jamba
0.830
0.782
0.773
0.835 1.805
0.877ana)
0.830
0.788
0 0.76
Gemman77
2019
0.637a323
a.82
0.750aam
0.70au
8.731
0.655
0.61 3.az
Llama3-8h
0.800
0.786
0.300am:
0.850 2.880
35
0.842.0a)
6:810
0.778
Bio-Llama3-Sb
0.8151m1
0.7922121
0.778an2:
0.8842ma
0.865amm
0.838una
0.815asa
0.7835mm
21 0 0.700
Command-RR -R
0,825ara
118177332
0.868
0,879
0,806
0.825
0. 0,787 02805
(d) Average Performance of Organ-Inndependet Search with Perturbed Text Reports
LLM
Accuracy
F1 Scure
AP
top
top
top 5
top
top
5
top
3
5
Jamba
0,778aam
1.7822111
0.72%unas
a.807ara
1,834am
0.812unz
0.778ases
0.748
0. 0.721
Gemma-77
0.638
0.575
0.515
0,7602aa7
0.703
0.612am
6.638
0.579
0. 0.534
Llama3-8h
0,792
a.7
0.736ars
0,868.0u
0.849aa3s
0.500.aa
0.778 6.80
755
Bio-Llamms--8h
0.7914em
8.767-355
0.74
0.867-6xx
0.845.a 1s
0.87-4030
0.757 0.790.4.0
731-mmm
Command-R
0.801
0.784.153
0.708.035
0.87346ar
0.8564a is
0.844.030
0.801.23:
0.775 s1
0. 0.755-00
Appendix, respectively.). For both settings, we utilized both original and perturbed representations to compare the contextual understanding capabilities of L.MSs. We report the average results for both organ-specific and organ- indeeedden searches across 32 different datasets in Table [] To illustrate this further, Figure 4 highlights the performance of the best-performing LLM across various datasets in the indeeenedd setting, which is the harder task. The detailed cancer-by-cance results for all 5 LLMs across different settings are available in section in the Appendix.
As Table la shows, LLMs perform well with Command-R R achieving a Top-1 accuracy of 0.871 and a Top-1 FT Score of 0.919 on average across all the cancer datasets for organ-specific search. This indicates that I.LMs' representations, although not perfect, can properly encode the text reports. However, one might question if this is due to the presence of medical keywords providing "hints' to the model. To evaluate this, removed the technical keywords such cancer names and subtypes from the original text reports and then re-generated the representations with LLMs. The results can be found in Table 1b As shown, the performance remains within the range of the organ-specific search. With this, we can empirically confirm that LLMs' representation is not only a function of the medical keywords but rather encoding of the medical context. The same conclusion is held after investigating the results from the organ-independent search in Table Tc [d We can potentially attribute this ability to the training scheme of language models as they are trained for next-token prediction for language modeling (Gloeckle et al. [224]] Achiamm et al. 2023). meaning that they are not solely dependent on keywords or specific tokens in the text.
Furthermore, comparing Ta and Table Ic we observe that LLMS maintain their performance when transitioning from organ- to wngan-independen searches. This suggests that they are capable of encoding beyond specific information and semantically understanding the differences between reports from various organs and body parts. Among the different LLMs, it is particularly noteworthy that Bio-Llamaa-88. almost always ranks among the top two best-performing models across various metrics, despite being significantly smaller in size compared to Command-R and Jamba. This highlights that smaller, bio-specific LLMs trained on medical data can encode pathology reports as effectively as larger,
L.MSs.
4.3.2 SURVIVAL PREDICTION
For survival prediction, we used an ensemble Random Survival Forest (RSF) based on the imple- mentation by (Ishwaran al. 2008). This decision was due the relative ease of implementation
7

---

Published as a conference paper at ICLR 2025
Table The C-Index of RSF for Survival Prediction on Eight Cancer Datasets using LLMs
LLM
BRCA
GBM
KIRC
KIRP
LGG
LUAD
LUSC
UCEC
Command-R
0.622.ao
0.722
0.743....
0.613.uu
0.547_am
u.602.u
Gemma-7h
0.008
0.512amu
0.707amcz
0.034aau
0.611
0.570
0.583-00
2.000
Jamba
0.625_ax
2,50)
0.6894a0s
0.745_am
0.630
0.5064424
0.545_am..
0.617
Lluma3-8b
0.629
0.521.0o:
0.713
0.750-.mm
0.607
0.585
0.520
0.580
Bio-Llama3-8b
0.627=a07
0.537
0.7009mes
0.726
0.587
0.583
0.518_
0.621
Average
0.522
0. 0.708
0.617
4.580 =.541
2.60
and training, as well as comparative robustness for higher input dimensionality. The RSF we used was from the scikit-surviaa python package (Pölsterl 2020) and initialized with 1000 estimators. Afterward, it was trained using 5-fold cross- vessdavaliationn and 10 random seeds per fold. The evaluation metric used was the right-censored concordance index, also known as the C-index.
We used the pathology report encodings generated by LLMs to train Random Survival Forests (RSFs) and report the C-index in Table 2 This clinically relevant task helps assess whether the LLM encodings capture features pertinent to patient outcomes. From a clinical perspective, cancer stage can help identify low- and high-risk patients for certain cancers. However, pathology reports contain far more detailed information, and this experiment demonstrates that LLMs are able to encode clinically relevant information in cancers such as BRCA, KIRC, KIRP, and UCEC, linking the encoded features to patient outcomes. To the best of our knowledge, this is the first time such phenomenon has been observed. We have also compared LLMs with LVMs in this task in section in Appendix.
Noerage DSS for 1334s with congardeed pocking Sor 201
Bataent
Figure 5: BSS compurison of seven L.Ms using the average BSS across five L.MMs on eight cancer datasets, with uncupervised pooling method, applicd all the cases in the dataset.
858 with sxpecvied
Bollssan Sommatic Seegu (600
TOANGEM
TOMNLGS
TOCNUCE:
Datamt
Figure 6: Average BSS comparison of seven LVMs using the average BSS across five LLMs on eight cancer datasets, with AbMIL as the supervised pooling method, applied to the cases in the held-out lest sel.
4.4 BOLTZMANN SEMANTIC SCORE
We have established a baseline for the current performance of LLMs in two tasks: information retrieval and survival prediction. Now, we experimentally evaluate and compare LVMs using the BSS, which leverages L.Mss. In this study, we employ two widely used settings in digital pathology: unsupervised pooling and supervised pooling. For unsupervised pooling, we use mean pooling to represent each WSI based on its patches, while for supervised pooling, we trained a Multiple Instance Learning (MIL) model, specifically AbMIL [Ise et al., 2018), following a 3-fold cross-validatte scheme represent the WSIs. We report the average BSS for &o 5 across five LLMs eight TCGA datasets for both settings in Figure [5) and Figure 66 When comparing the two settings, we observe that supervised pooling improves the semantic quality of the features some extent, but
8

---

Published as a conference paper at ICLR 2025
it remains below 0.4 for the majority of cancer datasets. More details regarding each setting provided in section 0.2 in Appendix.
Furthermore, LVMs achieve a fairly low BSS which indicates their encoding space may not be reflective of the information within a pathology report. The LVMs used were generally trained in self-supervised manner. This means non-medically relevant features may be extracted, and contrastively clinically relevant features may be missed. Among mathology-pretriing has not seen TCGA data, yet it outperforms the majority of the has been trained on 100,000 WSIs, compared to the TCGA-pretrained model with 12,000 WSIs, our observation suggests that models trained on more data may perform better semantically. As expected. ViT, which was trained on the non- non-medical ImageNet dataset, performed the worst overall. More details are provided in section D.3 and in Appendix.
models, UNI LVMs. As UNI
4.4.1 CORRELATION WITH DOWNSTREAM TASKS
Table 3: Two-sided Pearson Correlation Test between BSS and downstream task metrics
Correlation between BSS and top-1 Accuracy in Information Retrieval
LLM
GBM
KIRC
KIRP
LAGG
LUAD
LUSC
p-valoe
p-talue]
p-talee
p-value
p-tale
p-ralor
C Command--Rt
0.53
L.Ge-t
0.665
2.5--
1.267
25
6.251
2.2-12
0.550
6.6m 0.677
Gemma-7b 7b
2,300
9.te-k
0.433
2.t--
33--1e
6.am
9.20-1
0,8.9
1.te22 0.21
Jamba
0.30
1.3e
0.585
1:3e
Saw
0.838
1:se
0517
Llamuu Sb
0,20
1.4e
0.550
8.2c
1.226
6.227
5.5e-12
0.807
4.26 0.64
1.4e
Bio 1lo-L 8b
0,505
2.5e
0,372
2.4e
0.233
6.201
5.50
0.854
2.5c
0.73
1.0g
(b) Coreeation between BSS and C- index in Survival Prediction
BRCA
GBM
KIRC
KIRP
LGG
LUAD
LUSC
UCEC
L.M
p-value
p-value
p-value
p-value
p-value
p-value
p-value
p-value
Command-R
1.344
L.5e
0.081
9:6e
0.273
L.e
0.120
2.4c
0.027
G.9e11
0.233
5.46 10
0.150
3.2.-0
01.351
G.Ge"
Gemma-7b 7b
3.307
2.3e-99
7.1e-91
1.8c-0
0.200
7.1--5
0.223
1.3e-
666--1
0.352
6-4e-12
JJmbha
0.344
2.c
9:3c
0.257
6.
0.144
0.01
0.209
3.5c
4.
Llama3-80
0.350
7.86
-0.065
9.4e
0,272
1.1--1
0.151
24
-0.000
7.5e-1t
0:218
1.0g
0.096
3.6e
0.570
4.4e-23
Bio-Lluma3-Sb
0.317
1.2e 71
-0.67
9.5e
41
0.257
2.3c
0.151
7.16
-0.01
7:4--t
0.224
1,1e
0.112
1.8c
18
0.373
2.9e-1
Beyond the average BSS and the insights it provides into understanding semantic capability, also explored whether the score correlates with other metrics in downstream vision tasks, such as information retrieval and survival prediction.
Table and 3b pressent in information retrieval and BSS, well as between the C-index for survival prediction and BSS. As shown, the correlation in information retrieval is higher than that in survival prediction. We attribute this to the fact that, in information retrieval, the representations remain unaltered for the task, whereas in survival prediction, the representations undergo RSP-supervised training, which adapts them to the task (in this case, patient risk), emphasizing certain features. Our findings show that, in certain cancer types, there significant correlations between BSS and performance in both survival prediction and information retrieval. To the best of our knowledge, this is the first time has been observed that structural similarity in one modality is correlated with the performance of a downstream task in another modality. For further explanations, please refer to section K in the Appendix.
present the results of our hypothesis testing for the correlation between top-1 accuracy
4.4.2 CONSENSUS
One may argue that LLMs may not rank different LVMs similarly. However, to inspect this and measure agreement among various LLMs in ranking different LVMs based on BSS, Cohen's Kappa Score is utilized for pairwise com- parison of rankings. According to this metric, a score ranging from 0.21 to 0.40 is considered fair, 0.41 to 0.60 is moderate, 0.61 to 0.80 is substantial, and 0.81 to 1.00 represents almost perfect agreement. In this test, LVM rankings produced by any L.LM are compared across the 10 different auniguuaaiions employed for the information retrieval task. As depicted in Figure 7 on average, a sig- nificant agreement is observed between Command-R and LLama3-8b; Command-R R and Bio-Llama3 L Sb, and Jamba; and Llams3-8b and Bio- Llama3-8b, despite their complete independence and training on distinct data sources.
Average Coher''s Kaspa Scoras on TCGA
Oentema tang 10/4 te derecce
1:60
0.c5
0.62
0.85
0.75
0.65
0.48
0.65
0.55
0.05
0.77
1:50
0.75
0.75
0.63
0.65
6.75
Jange Long Maiel
Figure 7: The average pairwise Coben's 's Kappa Score
hetween two L.Mss different cancer types in TCGA.

---

Published as a conference paper at ICLR 2025
It is worth mentioning that moderate agreement, when present, depends the dataset. For instance, on datasets such as GBM, we observe moderate agreement between LLMs, while on datasets such as KIRC and BRCA, there is significant agreement among the majority of LLMs. We attribute this inter-LI. variability on some datasets to the training data sources of these models. The pairwise Kappa matrix plots for each dataset are available in section N in the Appendix.
As a result, we observe a substantial overall agreement between LLMs, which we attribute to their use of large dataset of reports calculating the BSS, leading to greater consistency. Moreover, unlike qualitative approaches where agreement is difficult to measure, we can quantify it by comparing different LLMs and assessing the reliability of various sources. With the introduction of domain- specific LLMs, inter-LLM agreement has the potential to increase further, and BSS can provide a deeper and more specialized semantic measure.
5 DISCUSSION
The goal of this research was to introduce systematic approach measuring the semantic capability of LVMs. To achieve this, we first demonstrated the capability of LLMs in two key tasks in digital pathology: information retrieval and survival prediction. We then proposed the Boltzmann Semantic Score (BSS), metric that measures structural similarity between two modalities, where we leverage LLMs evaluate the semantic capability of LVMs. This metric provides quantifiable and collectively derived evidence of an LVM's performance, offering more objective alternative to traditional small-scale qualitative assessments.
For the information retrieval task, we found that all LLMs achieved notable Top-1 performance, with Command-R R attaining an average accuracy of 0.871 and an Fl score of 0.919 across 32 pathology cancer datasets, encompassing 9, 523 patients. The results for the survival prediction task were more varied, with performance depending largely on the datasets and the clinical manifestations of the discases. However, the C-index remained relatively consistent across the LLMs, with Command- R again showing the best average C-index of 0.628. Overall, we demonstrated, for the first time in the field, that LLr--derieed features can be linked to patient outcomes, in future research.
ripe for further exploration
Our next step was assess the feasibility of the proposed Boltzmann Semantic Score for evaluating LVM performance. We observed that supervised pooling outperforms unsupervised pooling in WSI representation, largely due to the training step involved in supervised approaches. However, the semantic score for LVMs across different cancer datasets remains relatively low, suggesting there is considerable room for improvement in this area. We found that the BSS mostly correlated with the downstream task's 'ss performance metric, more so with retrieval than survival prediction. We attribute this difference to the methodological difference in the two tasks. In information retrieval, we use the exact same visual representations used for calculating BSS that were used for the task. However, in survival prediction, the visual representations undergo a regression step in the training phase that manipulates the representations, so more variations between BSS and C-index were expected.
We also conducted an experiment to investigate the consensus in L.MSs when reporting their BSS for each LVM. We show that, first, LLMs are substantially in agreement on ranking the LVMs. Second, unlike qualitative approaches where agreement is not easily measurable; we can quantify it here by comparing different LLMs. With this, we expect with the rise of domain-specific LLMs, inter-LLM agreement even increases more.
CONCLUSION
Inspired by state space modeling, we have introduced the Boltzmann Semantic Score (BSS) as a novel method for leveraging the encoding space of LLMs to evaluate the encoding space of LVMs. We first evaluated five state-of-art LLMs in clinically inspired digital pathology tasks and showed (for the first time) their capability in predicting patient outcomes. We then measured the BSS associated with seven LVMs using each of the five LLMs and a large collective database of pathology reports. Overall. LVMs showed poor BSS, highlighting their low semantic capability. We also found that the BSS for the seven evaluated LVMs is highly correlated with performance in two clinical tasks: information retrieval and survival prediction.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENT
We would like to thank Nikolay Alabi for validating the qualitative findings presented in this work. We also acknowledge the use of computational resources provided by Canada's 's Michael Smith Genome Sciences Centre (GSC) at BC Cancer, which supported our experiments and analyses. Additionally, we appreciate the insightful discussions and feedback from our colleagues and mentors, which have contributed to refining our methodology and interpretations.
8 REPRODUCIBILITY STATEMENT
We believe that the theory proposed here for the Boltzmann Semantic Score not limited to the medical field and can be applied to other domains with similar contexts. Therefore, the code developed and the processed data will be publicly released in our GitHub repository. We have also listed all the packages that have been used in scction the Appendix. Besides the theoretical discussion in the paper, we have provided the algorithms in Algorithm[ and For all the experiments, the results or the dataset description has been written in detail in the corresponding section in the Appendix.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877: 2023.
AI@ @ Meta. Llama 3 model card. 2024. /tt -atesttalllmmme 11ama/ / lama3
bl 3lob/main/00DEL. mm
Alonso Coeloo, L. Martfnez Garefa, J. M. Carrasco, et al. The updating of clinical practice
guidelines: insights from an international survey. Implementation Sci, 6:107, 2011. doi: 10.1186 1188-5908-6-077
Malaikannan Sankarasubbu Ankit Pal. Openbiollms: Advancing open-source large lan-
guage models for healthcare and life sciences. https: https: http://ggggaa.c./aaaaaaaaaaaaeaa
OpenBioL11--Laaag333
2024.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint
2014.
L. Boltzmann. Über die Beziehung zwischen dem zweiten Hauptsatze des mechanischen Wärmetheo- rie und der Wahrscheimlichkettsecemme respective den Sätzen übber das Wärmegleichgewicht. Kk Hof-und Staatsdruckerei, 1877.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.07255 2021.
Alexander D. Borowsky and et Digital whole slide imaging compared with light microscopy for primary diagnosis in surgical pathology: A multicenter, double-blinded randomized study of 2045 cases. Arch Pathol Lab Med, 1245-1233, October 2020.
R. Bouderhem. Shaping the future of ai in healthcare through ethics and governance. Humanit Soc Sci Commun, 11:16, 2024. htt.htts:: htt:://oo...o/oo
057 /117//4119999 0000---244---999
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
C. Chen, M.Y. Lu, D.F.K. Williamson, and et al. Fast and scalable search of whole-slide images via self-supervised deep learning. Nat. Biomed. Eng, 6: 6:1400-4434, 2022.
Richard Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Bowen Chen, Andrew Zhang, Daniel Shao, Andrew H Song. Muhammad Shaban, et al. A general-purpose self-supervised model for computational pathology. arXiv preprint ar0iv:2308. 15474, 2023.
11

---

Published as a conference paper at ICLR 2025
CohLreForAI Command-r. 64ai-command--v001 2024.
httee://aapiingfacccco//oorrreee
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Andrew J. Evans and al. Validating whole slide imaging systems for diagnostic purposes in pathology: Guideline update from the college of american pathologists in collaboration with the american society for clinical pathology and the association for pathology informatics. Arch Pathol Lab Med, 146(4):440-450, April 2022.
Ayesha Farooq and et al. Assessing the value of second opinion pathology review. International Journal for Quality in Health Care, 33(1), 2021. doi: mzab032.
Alexandre Filiot, Ridouane Ghermi, Antoine Olivier, Paul Jacob, Lucas Fidon, Alice Mac Kain, Charlie Saillard, and Can-ttttiste Schiratti. Scaling self-supervised learning for histopathology with masked image modeling. medRxiiv, 2023. doi: 10. http:///22222202111222222222 URL https?
wonteetteetttnnonn 7/ //ww. xii http:///w.....aoo///////////////
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. arXiv preprint arXiv:2404. 19737, 2024.
J. Huang, D.M. Yang, R. Rong, and et al. critical assessment using chatgpt for extracting struc- tured data from clinical notes. npj Digit. Med., 7:106, 2024. doi: 10.1088/4414466004-0000099
Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas Montine, and James Zou. Leveraging medical twitter build visual visual-language foundation model for pathology ai. bioRxiv, pp. 2023-03, 2023.
Maximilian Ilse, Jakub Tomezak, and Max Welling. Attention-based deep multiple instance learning. In International Conference on Machine Learning, pp. 2127-2136. PMLR, 2018
Hemant Ishwaran, Udaya B Kogalur, Eugene H Blackstone, and Michael S Lauer. Random survival forests. 2008.
L. Jiang. Z. Wu, X. Xu, et al. Opportunities and challenges of artificial intelligence in the medical field: current application, emerging problems, and problem-solving strategies. Journal of International Medical Research, 49(3), 2021. doi: 10. 10/1177003000001 1000157.
S. Kalra, H.R. Tizhoosh, C. Choi, S. Shah, P. Diamandis, C.J. Campbell, and L. Pantanowitz. Yottiel-- image search engine for large archives of histopathology whole slide images. Medical Image Analysis, 65:101757, October 2020.
Jenna Kefeli and Nicholas Tatonetti. TCGA-Reports: A machine-readable pathology report resource for benchmarking text-based AI models. Patterns, 5(3), March 2024. ISSN 2666-3899 doi: S2666- (24) 00024-2
URL hattps://wwe www.wwwwwww......ooooo...w.w.www cell. wtt.e//m//paa.e. htt.:///a/atteee/atttttaaaaaaaae Publisher: Elsevier.
J. Lammert, N. Pfarr, L. Kuligin, S. Mathes, T. Dreyer, L. Modersohn, P. Metzger, D. Ferber, J.N. Kather, D. Truhn, and L.C. Adams. Large language enabled digital twins for precision medicine in rare gynecological tumors. arXiv preprin! arXiv:2409. (tt:::24000055555 00544, 2024.
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shaaey-Shwartz, al. Jamba: A hybrid transformer- mamba language model. arXiv preprint arXiv:2403. /9887, 2024.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IFFF/CVF international conference computer vision, pp. 10012-10022, 2021.
12

---

Published as a conference paper at ICLR 2025
Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie- Yan Liu. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in bioinformatics, 23(6):bbac409, 2022.
L. Mclnnes, J. Healy, and J. Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 2xiiv:8822.03266 2018.
Rainer Müller. The boltzmann factor: a simplified derivation. European Journal of Physics, 35(5): 055002, 2014.
Jesutofummi A Omiye, Haiwen Gui. Shawheen J Rezaci, James Zou, and Roxana Daneshjou. Large language models in medicine: the potentials and pitfalls. Ann Intern Med, doi: 1of:10.7726/233-272. 2024.
Sebastian Pölsterl. scikit-survival: A library for time-to-evens analysis built on of scikit-leam, scittaae Jour- nal of Machine Learning Research, 21(212):1-6, 2020. URL http:://mmr..rgg/e/e/ee
v21/20- 72..ht9......
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp- PMLR, 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
R. Shankar. Principles of Quantum Mechanics. Springer Science & Business Media, 1994.
K. Singhal, S. Azizi, T. Tu, et al. Large language models encode clinical knowledge. Nature, 620: 172-180, 2023. doi: 10. 00M444188-002306222
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv: / 1908.08530 2019.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprint arXiv:2403.08295, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 2023.
Eugene Vorontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Siqi Liu, Philippe Mathieu, Alexander van Eck, Donghun Lee, Julian Viret, et al. Virchow: A million-slide digital pathology foundation model. arXiv preprint S.Vv:2309.07778 2023.
X. Wang, Y. Du, Yang, Zhang, M. Wang, Zhang, W. Yang, Huang, and X. Han. Retecl: Clustering-uuider contrastive learning for whole-slide image retrieval. Medical Image Analysis, 83:102645, January 2023.
Xiyue Wang. Sen Yang, Jun Zhang, Minghui Wang. Jing Zhang, Wei Yang. Junzhou Huang, and Xiao Han. Transformer-based based unsupervised contrastive learning for histopathological image classification. Medical image analysís, 81: 81:102559, 2022.
John N. Weinstein, Eric A. Collisson, Gordon B. Mills, Kenna Mills Shaw, Brad A. Ozenberger, Kyle Ellrotr, Ilya Shmulevich, Chris Sander, and Joshua M. Stuart. The Cancer Genome Atlas Pan-Cancer analysis project. Nature Genetics, http:///wwwwtuur..oo.oo. http://ww.aaueee..o/a/rrr..aaae/
October 2013. ISSN 1546-
1718. doi: 10. N0.1038/ng..774 URL hsttps://www.t n Publisher: Nature Publishing Group.
L.C. Wiest, D. Ferber. J. Zhu, and et al. medical information retrieval. npj Digit. Med., 7:257, 2024. doi: 10. 10.10388/417446224-000
large language models for structured
13

---

Published as a conference paper at ICLR 2025
Appendix
Below is an outline of the appendix for easy reference:
Statement Limitations
Statement Generalization
Statement on Future Directions
Dataset
Information Retrieval
Survival Prediction
Boltzmann Semantic Score
Software and Packages
LLM Representations
LVM Representations
Survival Prediction Experiments
A discussion TCGA-GBM Survival Prediction
LVM's Survival Prediction
Ablation Study on Patches
Boltzmann Semantic Score Algorithm
Commuttational Complexity
Boltzmann Semantic Score Experiments
Correlation with Downstrcam Tasks
Interpretation of Hypothesis Testing
Qualitative Examples
LLM Representation Plots
Consensus: Cancer by Cancer
14

---

Published as a conference paper at ICLR 2025
Table 32 TCGA cancer datasets with their full names and number of cases used in our study
Abbreviation
Full Name
Number Patients
ACC
Adrenocurtical Careinoma
90
BLCA
Bladder Urothelial Carcinoma
379
BRCA
Breast Invasive Careinoma
1034
CESC
Corvical Squamous Cell Carcinoma and Endocervical Adenocarcinoma
289
CHOL
Cholangiocarcinmmu
43
COADREAD
Colon Adenocarcinomu// Recttum Adenocarcinoma
580
DLBC
Lymphoid Neoplasm Difluse Large B-cell Lymphoma
47
ESCA
Esophageal Carcinoma
105
GBM
Gllioblastoma Multiforme
293
HNSC
Head and Neck Squamous Cell Carcinoma
520
KICH
Kidney Chromophobe
112
KIRC
Kidney Renal Clear Cell Carcinoma
KIRP
Kidney Renal Papillary Cell Carcinoma
280
LGG
Bruin Lower Grude Glioma
575
LIHC
Liver Hepatocellular Carcinoma
341
LUAD
Lung Adenocarcinomal
LUSC
Lung Squamous Cell Carcinoma
318
MESO
Mesothelioma
79
OV
Ovarian Serous Cystadenocarvinomaa
371
PAAD
Pancreatic Adenocarcinoma
176
PCPG
Pheochromocytona. and Paraganglioma
174
PRAD
Prostate Adenocarcinoma
446
SARC
Sarcoma
249
SKCM
Skin Cutancous Melanoma
102
STAD
Stomach Adenocarcinoma
319
STES
Stomach and Esophageal Carcinoma
83
TGCT
Testicular Gemn Cell Tumors
87
THCA
Thyroid Curcinoma
487
THYM
Thymoma
114
UCEC
Uterine Corpus Endometrial Carcinoma
546
UCS
Uterine Carcinosarcoma
56
UVM
Uveal Melanoma
65
Sum
9,238
A STATEMENT ON LIMITATIONS
Bias can be carried over both from text and image in the dataset, this can be variations within text reports originating from different hospitals, doctors, and cohorts on WSIs from using different staining protocols, hospitals, and cohorts. As BSS relies on LLMs for representing text reports, based their capability to extract information from textual data, we expect some robustness to -oriiattiiiginating biases. However, this needs to be further evaluated and also with newer LLMs, can be improved if it is proved to be biased. Therefore, other bias metries should complement BSS for a more comprehensive assessment of dataset-originating biases.
Our experiments have been limited two specific tasks: information retrieval and survival prediction. Therefore, further studies are necessary to evaluate the generalizability of our approach to other medically relevant tasks.
B STATEMENT ON GENERALIZATION
BSS generalization in the medical domain depends the nature of the task. For tasks directly linked to textual information, such as disease classification or subtype detection, we expect strong generalization as TCGA is made up of data from multiple clinical sites, which may have different reporting standards and procedures for what information is included in the pathology reports. For complex tasks like survival or treatment prediction, where insights are not explicitly available in reports, generalization may vary depending on the data as well as the patient cohorts, necessitating further experimentation. Additionally, tasks like segmentation, with limited text-image linkage, are less likely to generalize and depend on specific data context, which can vary by cancer type. In addition, while our study focuses on text and image modalities, the BSS framework can extend to other modality pairs (e.g., genomics and imaging), using genomics reference for biological semantics. We anticipate that BSS may generalize to other fields beyond medicine, though further validation is needed.
15

---

Published as a conference paper at ICLR 2025
C STATEMENT ON FUTURE DIRECTIONS
We believe it is essential to develop approaches that incorporate semantic representation at higher level. While current vision-language models focus primarily on patch-level representations, there is need for slide-level semantic representation approaches to advance the field. In terms of optimization, BSS offers a promising avenue it is differentiable and can be leveraged transfer semantic knowledge from text to vision models. By integrating BSS into training pipelines, vision models can better align with the semantic structures inherent in textual data. Additionally, this concept be combined with contrastive learning, where positive and negative samples defined based on BSS. Such combination could refine model training by encouraging the alignment of semantically similar pairs while distinguishing dissimilar ones. This approach could help bridge the gap between text and vision modalities, leading to more robust and semantically-aver models for tasks requiring holistic understanding the slide level.
loss function to
D DATASET
We used preprocessed reports from (Kefeli & Tatonetti 2024). Briefly, Kefeli al. utilized OCR to convert PDF reports to text. Additionally, Identifiable information (c.g., patient IDs, hospital names, pathologists) was removed by them, and the data was validated through proof-of-principle cancer-type classification task across 32 tissues, achieving
average AU-ROC of 0.992. Further details can be found in (Kefeli Tatonetti (024). Here, we list the cancer types we used for different experiments in our work. The number of patients is the same mentioned in (Kefeli & Tatonetti 2024
D.1 INFORMATION RETRIEVAL
For the information retrieval task, we used two settings. For the angan-independent search, we used all the cancer types present in Table 4 In this setting, query one of the cancer types among all the 32 cancer types.
For the organ-specific search, we distributed the cancer types as demonstrated in Table into different body organs. Then, when searching, we only queried among the cancer type associated with the same body organ.
Table 5: Organs and their associated cancers for organ- specifcc search in TCGA
Organ
Associated Cancers
Brain
GBM, LGG
Endocrine
ACC, PCPG, THCA
Gastro
COADREAD, ESCA, STAD
Gynacco
CESC, OV, UCS, UCEC
Liver-Panc
CHOL, LIHC, PAAD
Mclanocytic
SKCM, UVM
Prostate- Prostste-Testis
PRAD, TGCT
Pulmonary
LUAD, LUSC, MESO, THYM
Urinary
BLCA, KICH, KIRC, KIRP
D.2
SURVIVAL PREDICTION
For the survival prediction task, we used BRCA, GBM, KIRC, KIRP, LGG, LUAD, LUSC, and UCEC datasets from TCGA.
D.3 BOLTZMANN SEMANTIC SCORE
For calculating the Boltzmann Semantic Score used BRCA, GBM, KIRC, KIRP, LGG, LUAD, LUSC, and UCEC from TCGA. However, for the correlation test with the information retrieval task, divided these eight cancers into groups associated with their organs for conducting the retrieval. This is because, for a retrieval task, we at least need two classes to calculate accuracy. Fl score, and mean average precision score. Therefore, the groups
follows: GBM and LGG; KTRC and
16

---

Published as a conference paper at ICLR 2025
KIRP; and LUSC and LUAD. As BRCA and UCEC cannot be associated with another cancer in the selected TCGA datasets, we put both aside for this task. For the correlation test with the survival prediction task, we used the eight cancer sets individually and reported the results.
Along with the paper, we have uploaded the cancer by cancer Boltzmann Semantic Score for each dataset for all the ks in (1,3, 20, 50) in the supplementary materials. Please note that the error bars in the plots are the standard deviation of each LVM over all the patients present in that cancer type. Thus, error bars huge for small ks which shows the intense variability of LVMs. However, as k increases, the average Boltzmann Semantic Score increases and the variability reduces, which is due to the larger state spaces making easier for the LVMs to find matches. We believe k = 5 is good choice that provides a larger state space without losing sensitivity.
Please note that the supplementary materials organized in way such that the directory names are the cancer types, and the name of each 'png' file shows the name of the file with full details. For example, '/BRCA/topI_CComt Coral-Gemmal Beeemaa-b-Jamba-- Maamba-Llama3-ee Bee--8h---o-e- Rio-Llama3-8b-pe it belongs to BRCA cancer, and lop-1 shows that it is the average BSS calculated based on k 1. Please see 'Boltmaan Semantic Score.zip' to download related files.
shows that
E SOFTWARE AND PACKAGES
We utilized an array of different Python packages to conduct data processing and visualization tasks. The data manipulation and analysis were facilitated using Pandas (version 1.5.3) and NumPy (version 1 1.23.5), while statistical computations were performed using SciPy (version 1.10.1). For machine learning and deep learning models, we employed Scikit-Learn (version 1.2.1), OpenCV (cv2) (version 4.7.0), Torch Vision (version 0.14.1), and PyTorch (version 2.2. Icu.ll, respectively. Visualization of data and results was achieved through Matplotlib (version 3.7.0) and Seaborn (version 0. 0.12.2). Additionally, we leveraged the Transformers library (version 4.31.0) for natural language processing tasks and the Timm library (version 0.9.16) for working with some of the vision models.
F LLM REPRESENTATIONS
We use the Huggingface implementations of the LLMs and their default tokenizers to extract the representations. For each LLM,, the representations from the last hidden state extracted and then mean-pooled over all the tokens to obtain the report-level representation tensor for each report.
Table 6: Representation dimensions for different LLMs.
LLM
Jamba
Llama3-8b
Bio-Llama3-8b
Gemma-7b
Command-R
d1
1024
4096
4096
3072
8192
G LVM REPRESENTATIONS
First, we tile the Whole Slide Images (WSIs) at 20x microscopic magnification into patches of size 224 x 224. These patches are then passed through the LVM obtain patch-level representations. Subsequently, all the patch-level representations are pooled by both supervised and unsupervised pooling methods to obtain the WSL-level representation tensor. If patient has more than one WSI, we average the WSL-level representations to derive the patient-level representation.
Table 7: Representation dimensions for different LVMs.
LVM
ViT
SwinT
PLIP
CTransPath
Lunit-Dino
Phikon
UNI
d2
768
1024
512
768
384
768
1024
H SURVIVAL PREDICTION EXPERIMENTS
H.1 A DISCUSSION ON TCGA-GBM SURVIVAL PREDICTION
In Table 2 the TCGA-GBM dataset showed the lowest average performance in the survival prediction task L.M features, with C-index of 0.522. Biologically, this aligns with the aggressive and
17

---

Published as a conference paper at ICLR 2025
heterogeneous nature of glioblastoma multiforme, class of highly malignant brain tumors. The TCGA-GBM cohort has a median survival of only 12.23 months, with approximately half of the patients either deceased or lost to follow-up within the first year. Glioblastomas are usually detected in later stages and there is a lack of robust prognostic biomarkers in tissue images for patient risk stratification. Moreover, two of the most informative prognostic factors mutations in IDH and MGMT genes -are genomic variables that may be inaccessible to LLMs through pathology reports, potentially explaining the gap in performance.
H.2 LVM'S SURVIVAL PREDICTION
We used the pathology reports encodings of LLMs to train RSFs and report the C-index in Table 2 This is a clinically relevant task to compare the LLMs and LVMs. As we are using zero-shot LLMs here, Vit and SwinT are probably the best comparisons for zero-shot image encoder, they were trained on natural, non-medical images. Overall, the RSF models trained on LLM representations outperformed those trained on LVM features shown in Table 2 and 8 It is worth noting that CTransPath, Lunit-Dino, and Phikon were trained on TCGA data, and thus may be better able to extract representative imaging features. However, for some cancers like GBM and LGG, pathology- pretrained LVMs are performing better than LLMs. With that being said, LLMs' performance on average is better than LVMs even though they have not been trained on pathology data.
Table 8: The C-Index of RSF on Eight Cancer Types using LVMs
LVM
BRCA
GBM
KIRC
KIRP
LGG
LUAD
LUSC
UCEC
VII
0.579.acc
0.587.000
0.655_045
0.624...12
0.6988
0.567
0.572.o.6
0.628.am1
SwinT
0.577 .ea3
0.669 .oos
0.621.
01687
0.543
0.559
0.601
CTrunsPath
0.622-4oor
1.56
0.703
0.665_.m
0.706..m
0.582.ao:
0.6784oo3 0.568.a.e
Lunit-Dino
0.589 mo
0.563 ueas
0.682 Joo3
0.712
0.708
0.587
0.661 0.574.nm:
PLIP
0.575-004
0.541
0.665
0.706...1
0.6034.0o
0.5694am
0.545...:
0.6431nce
Phikon
0.626.n0s
0.582 le.os
0.711 .oos
0.736 oa1
(1688100:
0.593
0.696 0.515
UNI
0.6344.ac
0.590..0.0
0.715.2033
0.77
0.729400:
0.5644am:
0.5822a.6
0.697
Averuge
0.572....3 0.398
0.686
0.6
0.70140m:
0.572
0.5633.004
0.658
I ABLATION STUDY ON PATCHES
As an ablation study, we extracted a varying number of patches, ranging from 150 to 1500, from WSIs and mean-pooled them to obtain the slide-level rep- resentations. Afterward,
Arerage USS for UVMMe using ditderest LLMe
0.33
cunuiee Soare (855)
PLP
calculated the average BSS over each cancer dataset using different k's in (1,3,3 5, 20, 50). Figure&depin the eight cancer datasets of TCGA and the five LLMs for each vision model for = 5. As can be seen, the score is stable over different numbers of patches, which means that more patches do not guarantee a better semantic representation.
0.2
Chamsshch
the average BSS on
0.22
0.1S
200
100
2200
1200
1400
Number of pwichen
Figure 8: The average BSS with le = 5 over different LLMs and different eancer types TCGA.
J BOLTZMANN SEMANTIC SCORE ALGORITHM
We discussed the mathematical way of obtaining BSS, here we provide Algorithm I that demonstrates the mechanism to calculate By for the query q. Please note that the BSS is the average of all B, over one cancer type or class.
J.1 COMPUTATIONAL COMPLEXITY
If we consider a database size n, the order of complexity for KNN is O(nd + n log n), where d is the feature size. In Algorithm 1, we rely graphs with k neighbors obtained from KNN. To calculate the second-order Boltzmann factor, the order of complexity is O(k?). Therefore, the total order of complexity, including the KNN algorithm in calculations, is O(nd + n log k?). However, & is a constant that we chose to be k 5. Hence, k² is negligible compared to nd or log from the KNN algorithm. Consequently, the order of complexity remains that of the KNN
18

---

Published as a conference paper at ICLR 2025
Algorithm Boltzmann Semantic Score Algorithm
Require: re weffeenncegrrpppe observer_graph, di
Ensure: score B,
1: Initialize total _ostt_oottt_tt non, pon__matchinng.n sii_joods__cooe to 0
2: Initialize node index position counter
3: for each node in text text_graph excluding the query node do
4: Calculate Boltzmann Factor from current node to reference node in reference_grapp
5: if current node is not estimated by observer_grapl then
6:
Estimate corresponding node in bbserver_grappe Calculate the Second-order Boltzmann Factor from the current and estimated nodes 8: Update past_nodes___ __osst with computed Second-order Boltzmann Factor
7:
9: end if
10: Update total_cost _oo_ with computed Boltzmann Factor or Second-order Boltzmann Factor
end for
12:
Compute Bg by combining pust_nodes_ore and cxtro_nodes_____ _csst return Bg
13:
algorithm, which is Ond n n). We have explored this with k values ranging from 80, without any computational difficulty on our low RAM CPUs.
J.2 BOLTZMANN SEMANTIC SCORE EXPERIMENTS
For the unsupervised pooling, since no training was required, we tested the model on the entire dataset and reported the results. However, for the supervised pooling, we trained the AbMIL model on the features extracted by each LVM. To ensure robust evaluation, we employed 3-fold cross-validation scheme to train, validate, and test the model, leaving samples from one patient all on one set to prevent data leakage. We repeated the experiments with 10 different random seeds and selected the top-performing seed for the finalized model. Afterward, we calculated the BSS using the output of the AbMIL model as the WSI representation. For training, we used an eight-class classifier over the eight cancer datasets, with a learning rate of 0.001, weight decay of 0.01, and trained for 25 epochs. Patches were extracted from 20x magnifications.
K. CORRELATION WITH DOWNSTREAM TASKS
We the two-sided Pearson correlation test, where the null hypothesis (Ho) states that there is no significant correlation between the variables, and the alternative hypothesis (H1) 1) states that there is a significant correlation. Here, we have two tasks: information retrieval and survival prediction.
separate Random Survival Forest was trained on each the visual embedding sets. Each forest was built with 1000 estimators, and followed the algorithm (implemented using scikit-survival library) as described in Ishwaran et al (Ishwaran et al., 2008). A 5-fold cross-validation strategy was used such that for each fold fifth of the data into 10 random 80/20 splits of training and validation data. Thus for each dataset, we had total of 10 5 50 test C-index values. As such, we then had total of 50 350 C-index samples for the 7 pathology datasets considered in this paper.
held out the test set. The remaining data was organized
For each dataset, the Boltzmann Semantic Score (BSS) was calculated. We then used the two-sided Pearson correlation test to find the correlation between the BSS values and the C-index samples for each dataset. The p-value and correlation for each were reported in the body of this paper, with p-value < 0.05 considered significant. Algorithm describes the correlation test with survival prediction experiment accurately.
For each dataset, the Boltzmann Semantic Score (BSS) was calculated. We then used the Pearson correlation test to find the correlation between the BSS values and the top-k accuracy samples for each dataset. The p-value and correlation for each were reported in the body of this paper, with p-value- < 0.05 considered significant.
We adopt the above sampling approach to gather enough random samples for the Pearson correlation test so that the results of the test are reliable. Algorithm 3 describes the correlation test with information retrieval experiment accurately.
19

---

Published as a conference paper at ICLR 2025
Algorithm 2 Correlation Test with Survival Prediction
Require: Visual embedding sets {LVM1, L2,,.......... LVM7 of given dataset
1: Initialize number of estimators N Nestimators f- 1000
2: Initialize number of folds T
3: Initialize number of random seeds N₂ +- 10
4: for each visual embedding set LV M, do
5: for each random seed E {1,, SN. do
6: for each fold {1,.... . ] do
7:
Split LVM, into test set T, (20% of data) and remaining data R3 (80% of data) Train Random Survival Forest
8:
9:
Evaluate model on T, and obtain test C-index value Ci,t,s
Compute average Boltzmann Semantic Score (BSS) B
11: end for
12: end for
13: end for
14: Collect all the C-index samples in a list Collect all the BSS samples list
15: Calculate Pearson correlation between BSS list and C-index list
16: Perform two-sided Pearson correlation test and obtain p-value
if p-value 0.05 then
18: Report correlation and p-value as significant
else
20:
Report correlation and p-value as not significant
end if
22: Report correlations and p-values for the dataset
K.1
INTERPRETATION OF HYPOTHESIS TESTING
The hypotheses defined this study provide statistical proof and evidence that the Boltzmann Semantic Score (BSS) is predictive for downstream tasks. Ideally, if an LVM captures high semantic score, it is likely to perform well on downstream tasks, though the relationship is not completely linear. Therefore, we assert that when BSS is significantly correlated with Accuracy and/or the C-index in information retrieval and/or survival prediction, it indicates two key points: first, the correlation is not random event, and second, for the specific cancer data studied, LVMs with higher BSS are generally expected to perform better on these tasks, yet proportional to the correlation.
The variations in correlation observed in Table 3a for BSS with information retrieval top-1 accuracy across different datasets and LLMs can be attributed to several factors. Firstly, the dataset-specific differences, such the type and quality of information present in pathology reports, play a significant role. In some datasets, like TCGA-LUAD, we observe that LVMs with higher BSS values tend to align better with LLM semantic spaces, resulting in stronger correlations with information retrieval performance. This indicates a stronger semantic alignment with the LLM's semantic space in this dataset.
However, in datasets like TCGA-LGG, the correlation is weaker, potentially due to LVMs retrieving cases that are less semantically aligned yet still achieving high top-1 accuracy. This can occur because accuracy, when measured solely by subtype, may overlook nuances in semantic similarity that BSS captures. As result, some LVMs might succeed in retrieving correct subtypes but fail to retrieve semantically similar cases, leading to a drop in correlation.
Secondly, the latent spaces of different LLMSs have subtle distinctions, influencing the correlation values observed within each cancer dataset. These variations in LLM latent spaces likely contribute to the differing degrees of alignment and correlation across datasets, which matches our expectations given the inherent complexity and context-specifer nuances in each dataset.
Similarly, for the information retrieval task, we conducted the search and calculated the top-1, top-3, and top-5 accuracy with slide-level representation derived from varying patch numbers ranging from 150 to 1500 with a step of 100. This allowed us to gather 10 samples per each dataset. As such, we then had total of x 10 70 samples for each of the top-k accuracy metrics.
20

---

Published as a conference paper at ICLR 2025
Algorithm 3 Correlation Test with Information Retrieval
Require: Patch numbers P <- {150, 250,3 1500} for given dataset
1: Initialize top-k metrics < [1.3.5]
2: for each patch number / P do
3: Compute slide-level representation for D; using p patches
4: for each top-k metric & K do
5:
Conduct the search and Calculate top-k accuracy A 6: Compute average Boltzmann Semantic Score (BSS) Brpp
7: end for
8: end for
9: Collect all the top-k accuracy samples in a list Collect all the BSS samples in a list Calculate Pearson correlation between BS$ list and top-k accuracy list
11: Perform two-sided Pearson correlation test and obtain p-value
12: if p-value < 0.05 then
13: Report correlation and p-value significant
14: else
15:
Report correlation and p-value not significant
16: end if
Report correlations and p-values for the dataset
Query: TCGA-06-0876
Command-R Top-1 TCGA-06-0875
Top-1: TCGA-06-0879
Malignani allomo, possibe small oell g globiastoms
Nucicar
small-oal glod reoplesiic anapiasii
Vascular Recrosa and Shromoosi
Merovasoo. oel aroiforaton
Prommeet keryerfheods
Mizrossscu ber Popartopply and banthii
Frequin abmormal mitute Sqgunii
to nudel
GFAP
glafbrillart
DEAP wwwrenttieoooeoooooo foce Haaeyy glai fivilloemmer by neoplastic cols
Eaity Bambr pellaaing Withorr
micrasso proceeeeeent
p53 covroprraaionn whout 10% mecp
203 Connooprrsstoonn about 20 Beoplante
calls
MIB-1 prolierrtion indox between 0% and 30%
Query: TCGA-08-1800
Command-R Top-1: TCGA-06-6697
UNI Top-1: TCGA-06-0877
Elieamona (WHO Grade (V] Staming samporal
PaUSnt with rebropenesuslyy onha naing moss in the medial left temboral lobe. sotending fhelemic su dritheemmis and sentricalar subepernlynal regians Neoplasti cells glotteillogen & Avery smat minenty nemplestir call overproress p53 provein.
Gloearooma staining
Spindle sstrocytes
nogative cylskeratin.
Indicating prolfarsiion
GRAP $100. but
CD84
negative
NeuNannd synaptophysin demonstrane macpresli
Vimendn difusell pos BVg)
EMA showes scretered week posetitty
manskariin mecattin
Neuroftament
KEET Index Sppmmmattyyy3 30%
Two random queries from the TCGA-GBM dataset are shown. In this experiment, reported the top-1 I retrieved case for Command-R -R as a candidate from LLMs and for UNI as candidate from LVMs. By querying the case, we summarized the key mieroscopic details in the pathology reports here. The texts highlighted in green are semantically similar to the details in the reports; yellow indicates extra information mentioned in the retrieved case that is not fundamentally different; and red indicates semantically different information, making the case dissimilar to that of the query. In both cases, Command-R retrieved a more semantically similar case than UNI, although the cases retrieved by UNI had some similarities.
Figure 9:
L QUALITATIVE EXAMPLES
Two random queries from the TCGA-GBM dataset are shown in Figure
21

---

Published as a conference paper at ICLR 2025
Comrandd Top Matrics
Score
60
toza Plice
Preak ltees
roak tar
Yran Lane.
core
yoe coott
(a) Neerage
Command-R
Scar
10000
zest
ress
nack
(b) Gemma-7h
Jumba: Mettios for Ditasets TCGA
Riza Blace
Foska rike
sciak eas
aae
Lakc
Lace
eacs
(c) Jamba
Llama Bb: Accrage Top Metrics foe 22 Dutisets TE
tocck
Sclae stse
rape
Aca
(d) Llama3-8b
Biod bamac-thb Awwpgg The 3 Matrioc for 32 Dabscete of TORA
tockecck
Tosha toae
recuurra
ctae
(e) Bio-Llama3-Sb
Figure 10: Organ-Independent search on the original text reports
M LLM REPRESENTATION PLOTS
The LLM performance on each cancer dataset through organ-specific and organ-independent for two scenarios of original and perturbed text reports for different LLMs are shown in Figure 10 and
N CONSENSUS: CANCER BY CANCER
Cancer by cancer Cohen's 's Kappa Score is illustrated in Figure 14 As discussed in the body of paper, know that the agreement between different LLMss varies cancer by cancer. Here, LGG, KIRC,
22

---

Published as a conference paper at ICLR 2025
Command &
Top Metrios Datavets TODA
2000
eddaac
Maca caaace
machaca
Packew
rose
rals
micce
palaset
(a) Command-R
Gemun
Metrics 32 of TOGA
60
Esbacc
acckaack
teckio
iccunas
rasc
Dade
Aape
(b) Gemma-70 7b
Junba: Acorape Top Martic for datusets TeGA
Robbarc
Tecassc reda
teckanta
teca ttsss
tecua use
teckce
aaso
raao
Tea
Coss
eass
(c) Jamba
Hemms th: Arerags Top Metrica 32 Dutasets TE
14.
rasc
Mauch
caoe
(d) Llama3-8b
Awwage The Meeriox for Tatasees TOGA
20005
Micke
Lane
vase
masama
mase
rocce
tete
(c) Bio-Llama3-86
Figure 11: Organ-specific search the original text reports
23

---

Published as a conference paper at ICLR 2025
Command & Trp Metrios Datasets TODA
Rizz seace
Priak rike
sctak cce
Ricano pase
lunt
scce
eacy
(a) Command-R
Gemunt
Metrics 32 of TOGA
120000
60
Rick
sctaaie eas
Late
eacy
(b) Gemma-78 7b
Junba: Acurape Top Metnox for datasets TCGA
tecke..c
Tock sence
eccusao
tesuran
tecumna
Reaance
teskinc
raat
tose
bac
(c) Jamba
Llmms th: Average Top Metrica 32 Dutaseis OETOGA
14.
mazic
Stae
Mane
(d) Llama3-8b
Awwmge The Meerios Tataseea TOGA
20005
tacce
came
Lani
Vect
ca
rack
(c) Bio-Llama3-86
Figure 12: Organ-indeppndeet search on the perturbed text reports
24

---

Published as a conference paper at ICLR 2025
Command & Top Metrios Datavets TODA
2000
eddaac
rachaca
Packew
rose
rals
micce
(a) Command-R
Gemun
Metrics 32 of TOGA
F3000
60
Esbaac
acckaack
tockio
icounas
rasc
Dade
Sape
(b) Gemma-70 7b
Junbs: Acorape Top Mattic for datusets of TeGA
Robbarc
Coss
vsca
teckanta
teca ttsss
tecu use
eass
teckce
naso
raao
Tea
(c) Jamba
Hemms th: Arerags Top Metrica 32 Dutaseis TETAG
14.
rasc
(d) Llama3-8b
Awwage The Meeriox for Tatasees TOGA
20005
Rasa Micke
Lane
vase
masima
mas
rocce
tete
(c) Bio-Llama3-88
Figure 13: Organ-specific search the perturbed text reports
25

---

Published as a conference paper at ICLR 2025
LUAD, LUSC, and BRCA there are significant agreement between LLMS. Yet, on UCEC, KIRP, and GBM, LLMs do not similar agreement in ranking different LVMs.
2:44
1-30
L-30
2:44
0,40
0.30
Largrt Lagges Maxt
C,3
0.97
ESE
0.47
2,04
G07
1.00
100
10
GL7
1,00
1,0
9.00
LGG
GBM
2.0
6.77
1.00
1:20
1x
2.40
6.50
6.07
144
023
6.2
1//8
0.11
Unrel
Jeda Unael to
KIRC
KIRP
3D5
3.79
6:07
1:00
3.53
3.0
6:35
1.43
0.7
6:27
1.00
1.09
1:09
5:27
3:09
E.44
027
Gae seneeos derel Ap
Unail
LUAD
LUSC
Cakeets Oupoa
Kogertscoqoa UEFC
3:18
0000
1.31
E.00
0.60
sagaaa Mai
CDc
0.30
0.50
Lomgle sagaggg Make
6.33
0:51
0.57
0.5
3/00
6:00
0.50
1.3
1,00
1.90
Commies terriars
Largsangagge Noda
Oreiox terver
Largsangagge Noda
gusiee suchere
BRCA
UCEC
Figure 14: Cancer by Cancer Cohen's Kappa Score. As seen, on LGG, KIRC, LUAD, LUSC, and BRCA, there are significant agreements between LLMS. Yet, on UCEC, KIRP, and GBM. LLMs tend to disagree. According to this metric, a score ranging from 0.21 to 0.40 is considered fair, 0.41 0.60 is moderate, 0.61 to 0.80 substantial, and 0.81 1.00 represents almost perfect agreement.
26


Reviews:
[{"review_id": "PKLo1Udn7C", "reviewer": "Reviewer_1Uuo", "summary": "The paper proposes the Boltzmann Semantic Score (BSS) as a novel metric to evaluate the semantic performance of latent visual models (LVMs) by leveraging large language models (LLMs). The idea behind using BSS is to quantify how well the visual representations align with text expert annotations. The authors show that BSS could be used as a measure of semantic similarity for LVMs. This paper include applications to pathology reports and whole slide images from  The Cancer Genome Atlas (TCGA), a large publicly available cancer genome dataset. Evaluation on various tasks such as information retrieval and survival prediction is included. This paper suggests high correlations for certain cancer between BSS and performance in both survival prediction and information retrieval.", "questions": "•\tDid you visualize the semantic similarity and qualitatively assess the use of BSS as evaluation metric?\n•\tHow reliable is Boltzmann Semantic Score ? \n•\tWhat preprocessing was applied to the medical reports?\n•\tCould you explain the differences observed in Table 3 a) for the two-sided Pearon's Correlation Test? \n•\tWhat is the effect of bias originating from the datasets ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "•\tBoltzmann Semantic Score is a novel approach to evaluate the semantic perspective of LVMs.\n•\tThe work leverages a large dataset (TCGA), and experiments are performed on several benchmarks.\n•\tThe work provides interesting insights on the model performance using BSS based on observed results.\n•\tHigh correlation between BSS and two downstream tasks i.e information retrieval and survival prediction, highlighting the significance of the results.\n•\tInteresting experiments on clinical tasks showing correspondence between LLMs and patient survival.", "weaknesses": "•\tThe mathematics for the explanation of the Boltzmann Score and its application is rather heavy. A more concise and clearer explanation would enable to understand better the intuition behind the usefulness of BSS as an evaluation metric for LVMs.\n•\tThe authors could develop more on the clinical implication and real-word use of BSS in decision-making.\n•\tSome insights are provided to explain the differences between LVMs and LLMs performance, but the paper could investigate more thoroughly those differences and inherent variations.\n•\tA discussion of the limitations of this work in terms of generalization under different contexts is lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "akGwTyIh1q", "reviewer": "Reviewer_E1cX", "summary": "This paper introduces a novel semantic metric called Boltzmann Semantic Score (BSS), which is inspired by state space modeling, to evaluate the semantic capability of large vision models (LVMs) in medical image processing. The authors demonstrate the effectiveness of this metric through experiments, revealing that LVMs exhibit low semantic capabilities. Additionally, BSS shows a strong correlation with the performance of LVMs on two clinical tasks: information retrieval and survival prediction.", "questions": "1.Could the authors suggest ways to optimize BSS for large-scale datasets, or clarify if any tests on smaller subsets were conducted for comparative analysis?\n2. Since BSS performs better on information retrieval than survival prediction, could the authors elaborate on the reasons for this difference? Is there evidence BSS might generalize to other medical tasks?\n3. The paper notes limitations in LVMs' semantic capabilities. Do the authors have ideas on potential architectural or training adjustments that might address these limitations?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper is well-structured and clearly presented, which significantly improves its readability.\n2.The introduction of the Boltzmann Semantic Score (BSS) is an innovative approach inspired by state space modeling, providing a fresh perspective on evaluating the semantic capabilities of LVMs in medical image processing.\n3.The experiments demonstrate significant correlations between BSS and performance on the clinical tasks of information retrieval and survival prediction. Additionally, the experiments show LLMs' capabilities in these two key tasks and provide a quantitative comparison of LLM consistency. This consistency further supports BSS as an effective metric for evaluating the semantic capabilities of LVMs.", "weaknesses": "1.The computational complexity of BSS may be high in practical applications, particularly when applied to large-scale datasets.\n2.While the experiments show strong performance of BSS in the information retrieval task, its correlation with survival prediction is weaker. This may indicate that BSS lacks robustness across different types of tasks, especially in more complex medical applications. Therefore, its effectiveness as a general semantic metric remains to be further validated.\n3.The experiments focus on the tasks of information retrieval and survival prediction, but these tasks may differ in nature from other potential tasks. The consistency of LLMs and the effectiveness of BSS in other semantic tasks require further experimental validation across a broader range of tasks.\n4.The paper focuses on evaluating the semantic capabilities of existing LVMs, but it lacks concrete suggestions on how to improve their semantic performance. Although the limitations of LVMs are highlighted, there is little discussion on how to optimize or modify their architectures to overcome these shortcomings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "77Mi5t21ne", "reviewer": "Reviewer_F61b", "summary": "This paper proposes a semantic metric, BBS, to evaluate LVMs from a medically semantic perspective.\n\nThe paper also leverages LLMs and a large and collective database of medical reports across more than 30 cancer types that represent more than 9,500 patients and it also establishes a baseline of LLMs' performance in two large-scale digital pathology tasks.", "questions": "Would it be possible to make the BSS a training loss to guide and supervise vision encoder embeddings to align with the strong LLM embeddings? Will BSS have additional advantages over contrastive learning loss, such as smaller batch size requirements?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* Originality: Noval evaluation metric which evaluates the encoding space of LVMs from medical images using the encoding space of Large Language Models (LLMs) from medical reports. \n\n* Quality: Extensive experiments including experimentation on 32 datasets from The Cancer Genome Atlas collection using five state-of-the-art LLMs, comparison of seven LVMs with BSS, and two correlation analyses between BSS and performance in two downstream tasks.\n\n* Clarity: Well-painted figures and clear formulas.\n\n* Significance: Well-designed metric is important for the community, especially for the evaluation of latent embedding space.", "weaknesses": "I cannot find significant Weaknesses in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)