AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
Authors: Yilin Ye
Junchao Huang
Xingchen ZENG
Jiazhi Xia
Wei Zeng
Venue: ICML 2025 poster (2025)
Keywords: Visualization
Cross-modal Embeddings
Text-to-image generation
Primary area: general_machine_learning->evaluation

Abstract: Cross-modal embeddings form the foundation for multi-modal models. However, visualization methods for interpreting cross-modal embeddings have been primarily confined to traditional dimensionality reduction (DR) techniques like PCA and t-SNE. These DR methods primarily focus on feature distributions within a single modality, whilst failing to incorporate metrics (e.g., CLIPScore) across multiple modalities. This paper introduces AKRMap, a new DR technique designed to visualize cross-modal embeddings metric with enhanced accuracy by learning kernel regression of the metric landscape in the projection space. Specifically, AKRMap constructs a supervised projection network guided by a post-projection kernel regression loss, and employs adaptive generalized kernels that can be jointly optimized with the projection. This approach enables AKRMap to efficiently generate visualizations that capture complex metric distributions, while also supporting interactive features such as zoom and overlay for deeper exploration. Quantitative experiments demonstrate that AKRMap outperforms existing DR methods in generating more accurate and trustworthy visualizations. We further showcase the effectiveness of AKRMap in visualizing and comparing cross-modal embeddings for text-to-image models. Code and demo are available at https://github.com/yilinye/AKRMap.

AKRMap:
Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
Yilin Ye 12
Junchao Huang?
Xingehen Zeng'
Jiazhi Xia
Wei Zeng'
Abstract
1. Introduction
Cross-modal embeddings play a fundamental role for multi- modal models, functioning mrosl-modal encoders (Rom- bach et al., 2022), objective functions (Ramesh et al., 2022), or evaluation metrics (Hessel et al., 2021; Wu et al., 2023a) for tasks like text-to-image (T2I) generation (Peebles & Xie, 2023; Hu et al., 2025). To assess the alignment of multi- modal models, various evaluation metries based on these embeddings have been introduced, such as CLIPScore (Hes- sel et al., 2021) and Human Preference Score (HPS) (Wu et al., 2023a;b; Zhang et al., 2024). Despite their utility, cmbedding--ased evaluations are often difficult to interpret, as the metrics are typically reported as aggregated values, without providing insights into instance- performance. This deficiency undermines the trustworthiness and trans- parency of the evaluation.
Cross-modal embeddings form the foundation for multi-modal models. However, visualiza- tion methods for interpreting cross-modal em- beddings have been primarily confined to tradi- tional dimensionality reduction (DR) techniques like PCA and t-SNE. These DR methods primar- ily focus on feature distributions within a sin- gle modality, whilst failing to incorporate met- rics (e.g.. CLIPScore) across multiple modalities, This paper introduces AKRMap, new DR tech- nique designed to visualize cross-modal embed- dings metric with enhanced accuracy by learn- ing kernel regression of the metric landscape in the projection space. Specifically, AKRMap con- structs a supervised projection network guided by a post-projection kernel regression loss, and employs adaptive generalized kernels that can be jointly optimized with the projection. This ap- proach enables AKRMap to efficiently generate visualizations that capture complex metric dis- tributions, while also supporting interactive fea- tures such as zoom and overlay for deeper ex- ploration. Quantitative experiments demonstrate that AKRMap outperforms existing DR methods in generating more accurate and trustworthy vi- sualizations. We further showcase the effective- ness of AKRMap in visualizing and comparing cross-modal embeddings for text-to-image mod- els. Code and demo are available at https: // //qitt //githubb.. com/yo////////o y pilliiyee AKRMap.
To address this limitation, dimensionality reduction (DR) visualization serves a crucial tool, offering way to reveal the landscape of cross- metrics and enabling more comprehensive understanding of model performance. Re- cent studies (Liang et al., 2022; Wang et al., 2023b;c) have explored the use of DR methods for cross-modal embed- dings. These efforts often leverage established techniques such PCA (Li al., 2018), UMAP (Mclnnes et al., 2018) and t-SNE (Van der Maaten & Hinton, 2008), well as autoencoder based approaches (Le et al., 2018; Elhamod & Karpatne, 2024). However, existing DR methods are primarily designed to depict feature distributions within a single modality. When applied to cross-modal metrics, these methods often produce dense neighborhoods where points with significantly different metric values posi- tioned close, leading to overlap and local occlusion (Figure 1(b)). Moreover, multi-modal models are typically eval- uated on large-scale datasets containing millions of data points. This creates a need for rendering contour maps that can reveal continuous metric distributions. However, lo- cal neighboring points with a mix of high and low metric values may be misrepresented, as the contour map depicts only single aggregated value, causing inaccurate contour mapping as illustrated in Figure 1(c).
'The Hong Kong University of Science and Technology (Guangzhou) "The Hong Kong University of Science and Technol- ogy 'The Chinese University of Hong Kong (Shenzhen) "Central South University. Correspondence to: Wei Zeng <weizeng @ hkust- gz.edu.cn>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To construct trustworthy visualizations for cross-modal em- beddings, a key consideration to enhance the accuracy

---

AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
We propose a novel DR method for cross-modal met- ric visualization, which jointly learns the projection and metric contour mapping through kernel regression supervised DR with adaptive generalized kemel.
We develop a tool for trustworthy visualization of cross- modal metrics, incorporating visualization features such as a scatterplot view and a contour map, along with interactive features like zooming and overlaying.
(a) I-SNE projection
(b) local occlusion
(c) inacourate contour map
Figure 1. CLIPScore distribution on the COCO dataset by t-SNE (a). The visualization shows dense neighboring points with signifi- cantly different metric values, causing overlapping and occlusion (b) and highly inaccurate contour mapping (c).
We conduct quantitative experiments to demonstrate the superior performance of AKRMap in generating more accurate visualizations of cross-modal metric, and highlight its applications across three scenarios to enhance the trustworthiness of T2I model evaluation.
of metric contour mapping in the projected 2D space. As contour estimation typically relies on radial basis function (RBF) kernel (e.g., Gaussian kernel), we seek to enhance the kernel-based mapping by coordinating it with the DR pro- cess. Drawing inspiration from supervised t-SNE (Hajderanj et al, 2019), we introduce Adaptive Kernel Regression Map (AKRMap), supervised DR method that leverages adaptive kernel regression to effectively visualize the distribution of cross-modal embedding metrics. Specifically, AKRMap first constructs a supervised DR network explicitly guided by post-projection kernel regression loss with neighborhood constraint (Sect. 1.1. Next, to account for the gap be- tween figh-dimensional and low dimensional kernels, we improve the flexibility of the post-projection kernel through adaptive generalized kernel jointly learned with the projec- tion (Sect. 4.2. This approach enables the generation of scatterplots for discrete data points and contour maps for continuous metric landscapes, while also supporting advanced features such overlay views and zooming for in- teractive interpretation (Sect. 4.2). Both the DR method and the visualization tool have been implemented as a Python package and made publicly available.
2. Related Work
2.1. Cross-Modal Embedding-based Evaluation
Cross-modal embeddings like CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) form the foundation of multi- modal learning. Specifically, many evaluation methods for multi-modal models, such T2I models, rely on these em- beddings. Conventional metrics like Inception Score (Sali- mans et al., 2016) and Fréchet Inception Distance (Heusel et al., 2017) compute the average distance or distributional difference between generated and reference images in the embedding space, yet they are insufficient to measure cross- modal alignment and instance-level performance. With the advancement of multi-modal AI, modal embedding metrics like CLIPScore (Hessel et al., 2021) have emerged to measure the alignment between prompts and generated images. While being effective in capturing the alignment, CLIPScore fails to model human preferences. To address this limitation, recent studies have proposed training spe- cialized human preference models such as HPS (Wu et al., 2023a:b; Zhang et al., 2024), PickScore (Kirstain et al., 2023), and ImageReward (Xu et al., 2024), to better align with human judgments. For example. ImageReward (Xu et al., 2024) introduces a systematic pipeline for expert anno- tations to train preference models, while PickScore (Kirstain et al., 2023) leverages crowd-sourced human comparisons.
We conduct quantitative experiments to evaluate A AKRMap against traditional and autoencoder-based DR visualizations (Sect. 5.1). The results highlight the superior performance of AKRMap in accurately mapping cross-modal metrics for both samse and out-of-sample data points. We further demonstrate the practical applications of AKRMap across three distinct scenarios (Sect. 5.2): 1) visual exploration of human preference dataset (HPD) (Wu et al., 2023a), 2) vi- sual comparison of diffusion- based and auto-regressive T2I models, and 3) visual examination of the global impact for fine-tuning. These applications illustrate that AKRMap ef- fectively enables human-in- moman-in-thhe-ooe loop interpretation of model performance on large-scale datasets. Our method can also be potentially generalized to other modalities such as text-to- video task, with additional quantitative experiments shown in Appendix F.
However, these metrics generally provide only an average score, offering broad overview of model's performance across the entire embedding space. This highlights the need for neural embedding visualizations that can expose the de- tailed distribution of metric values within the cross-modal embeddings, facilitating instance-level inspection and sup- porting human-in mentaan---the--oee -o-ppelooop evaluation.
2.2. Visualization for Neural Embeddings
Visualization is an important tool to support data analysis in conjunction with Al (Wang et al., 2023a; Ye et al., 2024).
In summary, our contributions are as follows:
2

---

AKRMap:
Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
Embeddings
Adaptive Kernel Regression Map
Cross-modal Metric Visualization
High-dim Space
Regression supervision
Adaptive kernel
Myyiing beoeki wnt todiing de
Projection
Scatterplot
Contour map
Figure 2. AKRMap is a neural network based DR method designed to learn adaptive kernel regression for visualizing cross-modal embeddings. The network integrates two key components to jointly Icarn data point projection and cross-modal metric estimation: Kernel regression supervision, and 2) Adaptive generalized kernel. The resulting visualizations, including scatterplots and contour maps. provide clearer and more accurate representation of the moos- metric distribution.
Particularly, it has proven to be effective for enhancing the understanding of various types of neural embeddings, in- cluding word embeddings (Mikolov et al., 2013; Chiang et al., 2020), vision-language embeddings (Liang et al., 2022; Wang et al., 2023b; Ye et al., 2025), and parame- ter spaces within loss landscape (Li et al., 2018; Elhamod & Karpatne, 2024). Commonly, dimensionality reduction (DR) techniques are employed for visualization. These in- clude linear DR methods such as PCA (Abdi & Williams, 2010), as well as nonlinear DR methods like t-SNE (Van der Maaten & Hinton, 2008) and UMAP (McInnes et al., 2018). Parametric versions of traditional DR methods, which pro- vide explicit mapping for projections,
based supervised projection method combined with an adap- tive generalized kernl. Different from traditional kernel DR methods like kernel PCA (Schölkopf et al., 1997) that mainly use kernel to transform the tigh-dimensional fea- tures, we develop an adaptive kernel in the projection space to guide the DR. Notably, to construct the kernel guidance, we propose novel cross-validation supervision technique that bridges between traditionally nonparametric kernel re- gression and parametric DR. This design allows the propa- gation of contour estimation errors back into the projection process. this way, our method can dynamically adjust both the DR mapping and the kernel shape, enabling it to accurately fit the complex landscape of cross-modal metrics.
be achieved by training neural networks, such as parametric t-SNE (Gis- brecht et al., 2015; Damrich et al., 2023), parametric UMAP (Sainburg et al., 2021) and other parametric meth- ods for tasks like interactive clustering (Xia et al., 2023) or streaming data (Xia et al., 2024). Similarly, autoencoder- based DR methods (I.e et al., 2018; Elhamod Karpatne, 2024) have also been developed, incorporating an additional objective of reconstructing nigh-dimensional embeddings from their projections. Recently, there has been a growing interest in utilizing DR techniques to visualize cross-modal embeddings. For instance, UMAP has been employed to explore and visualize the modality gap between text and image embeddings (Liang et al., 2022; Wang et al., 2023b). To address this modality gap and enhance the cross-modal alignment in visualization space, some studies have intro- duced fusion-based DR methods (Ye et al., 2025), enabling the visualization of image embeddings in relation to text embedding anchor points.
3. Problem Definition
Taking text-to-image generation models as an example, we formally define embedding- based metric for multi-modal models: A pretrained neural network e(-) encodes the prompt and the generated image v into a high-dimensional embedding representations e(t, v). which are subsequently used to predict a metric score sents the final operation applied to the embeddings, such as cosine distance in HPS (Wu et al., 2023a).
[e). Here, f(-) repre-
The problem of generating trustworthy visualizations for cross-modal embeddings can be defined as follows: given dataset of prompt-image pairs D {(4,, x:) and its embedding representations e(D) {e(ti, e,)
J== we first seek to learn a manifold M (Goldberg et al., 2008) where el (D) resides in the high- -igeeniienii embedding space Rd This manifold can be "spread out" (reparametrized) in the visualization space R²2 to show the distribution of metric score across the dataset, where the process can be modeled as a projection mapping P(-) s.t. P(e(D)) E R². This pro- aims to learn explicit mapping function that projects high-dimensional embeddings to points in 2D space while accurately preserving the underlying metric distribution.
However, previous efforts treat multi-modal embeddings separately, failing to effectively depict the landscape of cross-modal metrics (e.g., CLIPScore and HPS). In addi- tion, existing metric-aware DR techniques focus lim- ited range of particular metrics like distance (Sainburg et al., 2021) and density (Narayan et al., 2021), without a gen- eralizable mechanism to preserve other different metrics.
Next, to visualize a continuous distribution from the dis- crete projected sample points in P(e(D)), a contour map
To address this limitation, we propose a kernel regression-
3

---

AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
in the 2D space needs to be estimated. Here, we adopt the approach used by contouring algorithms in various Python libraries, such Plotly, by dividing the projected 2D space into a grid and calculating the metric distribution values at each grid point, Specifically, suppose the projected coor- dinates lie within a normalized 2D space of [o, 1] x [0, 1]. For each grid x, = (v.., Ny). ). we compute a value 8(x,) based on the local metric distribution, which is then col- ored using a continuous colormap. Nw and Nr denote the number of grids along the x-axis and y-axis, respectively. In this manner, we can generate contour map depicting continuous landscape of the metric distribution from the discrete projected sample points.
the dataset D into training set Drr and validation set Dut of ratio 9 : 1 in each epoch. Then, in equation (1), we esti- mate the metric distribution only with points in training set ((tx, mx) E Dtr). Next, we seek to minimize the weighted mean square error loss:
MS DDpl 2,ED,
Σ (f(1)) si),, p E (ul,
(2)
MSF, = w/ MSF. + W2 Mtrr
(3)
where the overall regression loss MSE, is a weighted sum of the loss on validation set (M)E,) and that on training set (MSF))) We seek to balance these loss terms to ensure mapping accuracy at both in-sample and out-of-sample posi- tions on the contour, and the weights wi = and w2 = 0.3 are set empirically. The construction of this weighted train- val loss may differ from common practice but is motivated by our deeper thinking the problem of connecting pro- jection with kernel regression, which is further explained as follows. Unlike neural network methods, kernel regres- sion is nonparametric, where the parameter like bandwidth is traditionally either precomputed or optimized via cross- validation. Thus, we are facing an interesting problem of how bridge between parametric projection and nonpara- metric kernel regression, motivating us to leave validation set to learn kernel parameters jointly with the neural net- work. The validation loss improves the generalization of kernel regression for unseen positions. However, relying solely on the validation loss will decrease the local detail of the map. In fact, as shown by additional experiments in Figure 14 in Appendix E, this train-val scheme proves be essential to ensure robust mapping.
4. AKRMap
We propose Adaptive Kernel Regression Map (AKRMap), with the workflow illustrated in Figure 2. The input is a set of cross-modal modal embeddings with each high- high-dimensional vector representing a data point in the embedding space. First, to synchronize the contour mapping with the projec- tion for accurate metric landscape estimation, we propose cross-modal metric-supervised projection method com- prising two key components: 1) Kernel Regression Supervi- sion, which guides the projection to achieve more precise metric mapping (Sect. 4.1.1), and 2) an Adaptive Gener- alized Kernel, which accounts for the gap between high- dimensional and low-dimensional kernels and allows for more flexible contour mapping to capture complex metric landscapes (Sect. 4.1. Then, we design an interactive vi- sualization tool to facilitate multi-scale exploration of metric distribution and individual data points (Sect. 4.2).
4.1. Adaptive Kernel Regression
Neighborhood Preservation Constraint. To encourage the projection to maintain some neighborhood information of the high-dimensional embeddings, we combine our re- gression loss with a constraint term from traditional di- mensionality reduction. Specifically, we incorporate the KL--divergence loss from t-SNE:
1. KERNEL REGRESSION SUPERVISED PROJECTION
According to the problem definition, to achieve an accurate cross- cross-modal metric visualization, cross-modal metric super- vision is necessary. To address the challenge of constructing appropriate supervision for continuous cross-modal met- ric, we develop a kernel regression supervised projection method. Specifically, we propose learning a projection net- work P: R - R² with a Nadaraya-Watte Watson kernel regres- sion (Ali et al., 2023) in the projected space:
KL = Σ Σ
Pij
In
(4)
T
J.
where Psy and qú are the neighborhood distribution prob- abilities in high-dimensional and low-dimensional space respectively, as detailed in (Van der Maaten & Hinton, 2008). Specifically, we adopt a perplexity-free implemen- tation (De Bodt et al., 2018; Crecchi et al., 2020) of the KL loss for our parametric projection network. Overall, the objective function of our method is:
8(x) EN K(x- - P(e((I, vk))) 8k
EN K((-x((((1 P(e(la, vw)))
where P(e(tk, ux))) is the projected sample point of metric embeddings and 84 is the corresponding ground-truth metric value in the training set. K(-) is an RBF kernel in R².
Subsequently, to construct a supervised learning objective for the projection, inspired by the cross-vas cross-validation method for kernel learning (Silverman, 2018), we randomly split
L AMSE, + KKL,
(5)
where 0.125 is empirically set to balance the loss.
4

---

AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
4.1.2. ADAPTIVE GENERALIZED KERNEL
In this section we illustrate the design of kernel K-) in the regression supervision in equation (1). The key challenge is that we need to consider the discrepancy between high- dimensional and now-dimensional kernels. First, to avoid un- stable optimization caused by exponentials and make the re- gression more compatible with our -ow-dimeniional t-SNE neighborhood constraint, we adopt a t-distribution-l nel instead of a Gaussian kernel. However, for complex met- ric distribution landscapes, especially those in embedding- based generative model evaluation, a standard kernel may lack enough flexibility. Particularly, it is difficult to deter- mine proper decay rate of the kermel value in relation to the distance. To address the problem, we take inspiration from the observation in a previous study (Narayan et al., 2021) of an approximate power law relationship between the local radius of the low-dimensional space (re) and that of orig- inal high-dimensional space ro: Te (a:) R a [ro (z²))² We hypothesize that this transformation can also improve the adaptability of low-dimensional kernel. Therefore, we propose using an adaptive generalized t-distribution kernel:
(a) I-SNE (p=100)
(b) Pt-SNE
(c) AKRMap
ker-
Figure 3. Comparison of scatterplots generated by I-SNE and AKRMap for the HPSv2 metric. Despite the visual clutter in- troduced by the large-scale dataset, AKRMap provides a clearer and more accurate representation of the HPSv2 metric distribution.
2.9
2011
Prompt: draac sa das
HPS: 21.20
419
(a) Overview
(c) Instance
Figure 4. Contour map combined with zoom and overlay with point sampling for multiscalc exploration of the HPD datasct.
K(x, az B) = (1 a |x| 1x]
(6)
The visualization tool includes various interactive features designed to facilitate in-depth exploration, including:
where and 3 are learnable parameters that are jointly optimized with the projection. In this way, our method can dynamically change the shape of the kernel to accurately fit the cross-modal metric landscape and reduce the risk of overfitting or underfitting due to suboptimal decay rate.
Zoom. AKRMap enables efficient multiscale zooming by dynamically computing the contour map using vary- ing grid resolutions, illustrated in Figure 4. Specif- ically, it adjusts the level of detail by increasing the grid resolution proportionally to the zoom level, re- vealing finer details the map as users zoom in. This capability unique to AKRMap and not achievable with traditional DR methods, which lack the ability to preserve and display local details.
4.2. Cross-modal Metric Visualization
On the basis of AKRMap, we further develop an interac- tive visualization tool that provides two distinct views for exploring mross-modal embedding metrics:
Scatterplot view. The scatterplot view displays all in- dividual data points within the embedding space, with each point color-coded according to the cross-modal embedding metric. This view allows for direct inter- action with the data points. Unlike scatterplots gener- ated by baseline DR methods, which often suffer from significant occlusion that obscures the true metric dis- tribution, AKRMap effectively reveals extreme values with greater clarity, as demonstrated in Figure 3.
Overlay. The scatterplot view and contour map allow data points to be overlaid onto the contour map, en- abling users to simultaneously observe the overall dis- tribution while interacting with individual data points, as shown in Figure 4. To mitigate occlusion, we have implemented sampling techniques, such as random sampling and Poisson disk sampling.
The visualization tool has been built into an easy-to-use python package with minimal dependencies, requiring only PyTorch and Plotly, which can be seamlessly integrated into interactive computational notebooks.
Contour map. For large datasets like HPD and COCO, scatterplot can become overcrowded with cluttered points. To address this limitation, we introduce con- tour map that effectively represents cross-modal metric distribution in a continuous manner, by dividing the 2D space into grids and computing grid values based on the local metric distribution. This representation makes regions with extreme values more prominent and reveals distribution patterns with greater clarity.
5. Experiment
5.1. Quantitative Experiments
Baseline methods. We compare AKRMap against three commonly used DR methods: PCA, t-SNE and UMAP, as
5

---

AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
Table 1. Quantitative comparison of cross-modal embedding metrie visualizations for test (out-of-sample) points on the HPD dataset. Our
AKRMap outperforms baseline methods and ablations, achieving the best performance across all four cross-modal embedding metrics.
CLIPScure mape
HPPSy2
PickSeore mape
Aesthetic Score
Method
mae
rmsse
mape
mec
17755
mac
mape
rmec
PCA
4,0042
16,5506
5.1142
1.4444
5,7870
1.8749
1.1497
5,9206
1,4686
0,5322
11.5645
0.6756
L-SNE
4,0241
16.6643
5.1523
1.4361
5.7568
186299
1.1579
5,9429
1,4640
0,4725
10.3128
0.6018
UMAP
4,0819
16.9066
5.2179
1.4432
5.7817
1.8725
1.2214
6.2721
1.5536
0,4463
9.6323
0,5721
4.1843
16.9186
1.4483
5.8117
1.8696
1.2357
6,3952
1,5650
0,5809
12.7041
0.7251
Souoo-iiualizzr
12,4201
43,8219
13.7318
1.6871
6.4526
2.0649
1,9655
9,5295
2.3427
0,4900
10.6306
0.6252
MRR (w/ KR)
4.0164
16.4830
5.1274
1,3978
5.6039
1.8102
1.1284
5.8004
1.4353
0,4752
10.3943
0.6003
AKRMap (wfo GK)
2.2812
9.1189
2.9458
1,1430
4.5498
1,4695
0.9064
4.6194
1,1686
0,4555
9.9267
0,5718
AKRMap
1.8707
7.3649
2.4253
0,8108
3.1935
1.1200
0.7712
3.9225
1.0225
0.4305
9.3340
0.5488
well as two encoder-basee approaches: SAE (Le et al.. 2018) and Neuro- Neuro-Visualizer Visualizer (Elhamod & Karpatne, 2024). For the besoder-bbsee methods, the decoder is used to estimate grid colors, while the other baseline methods rely on the com- mon Gaussian RBF kernel estimate the 2D distribution map. Additional details about RBF parameter selection are provided in Appendix C. We use distance threshold to cut off empty areas for traditional DR techniques but keep the full landscape of autoencoder methods because far away positions in traditional DR results are ambiguous and can hardly be mapped back to original high-dingg high-dimensional space without an inverse decoder. On the other hand, as autoen- coder can easily lead value explosion, we set an upper bound for the map to align with the maximum metric value in the dataset.
also report the errors for in-sample points (maein, mapein. rmse:.) from the HPD training set in Appendix B. This dual evaluation is important because a good visualization method must accurately represent the training data distribu- tion while avoiding misleading maps caused by overfitting and ensuring reliable accuracy for test points.
Training implementations. The architecture of our pro- jection network is a 4-layer MLP, with shape (d, d) in the first three layers and (d, 2) in the last layer. where d is the dimensions of the input embeddings. Batch normalization and ReLU activation are applied to each layer. Our projec- tion model trained on one Nvidia L4 GPU with batch size of 1000 and 20 epochs. We use Adam optimizer with learning rate of 0.002. For the t-SNE and PCA implementa- tions, we use the python sklearn package, where the t-SNE method adopts the Barnes-Hut t-SNE (Van Der Maaten, 2014). For the UMAP implementation, we adopt the python umap-learn package. The Neuro- Visualizer implementation is based on the open-sourced code of the original paper (EI- hamod & Karpatne, 2024), while the SAE is based on github reimplementation'
Ablation Study. We evaluate AKRMap under two alter- native settings: 1) training the projection network using only the neighborhood constraint, without the supervision provided by Kernel Regression (AKRMap w/o KR), and 2) removing the adaptive Generalized Kernel component (AKRMap w/o GR).
Dataset. To evaluate performance on the moosl em- bedding metric, we select the widely used large-scale T21 dataset, the Human Preference Dataset (HPD) (Wu et al., 2023a). The HPD dataset contains over 430,000 unique images generated by various T2I models, along with their corresponding prompts, in the official training set, and 3,700 images in the test set.
The experiment results are presented in Table 1.
Comparison with baselines. Our method AKRMap consis- tently outperforms baseline methods in mapping accuracy across all the embedding metrics. Notably, AKRMap effec- tively reduces mapping errors for both training (in-sample) points (see Appendix B) and test (out-of-sample) points, demonstrating its ability to produce more trustworthy map- pings. For instance, when applied to the HPSv2 metric, AKRMap reduces the MAE by nearly 50% for in-sample points and approximately 43% for out-of-sample points com- pared to the performinf baseline, t-SNE. In addition, while autoencoder methods are effective for loss landscape, it has shown weakness in cross-modal metric mapping with relatively higher error and unstable performance. These find- ings highlight AKRMap's 's superior reliability and robustness across diverse embedding scenarios.
Cross-modal Embedding Metrics We compare AKRMap against baseline methods for visualizing several commonly used cross-modal embedding metrics in T21 generation, in- cluding CLIPScore (Hessel al., 2021), HPSv2 (Wu et al., 2023a), PickScore (Kirstain et al., 2023), and a unimodal embedding metric commonly used in this cross-modal sce- nario, the Aesthetic Score (Schuhmann et al., 2022).
Performance Evaluation. We evaluate the accuracy and trustworthiness of the visualization methods using map- ping errors, including mae, rmse, and mape. Specifically, we calculate these errors on the test set of HPD for out- of-sampee points that were not used during training. We
Ablation results. The ablation results highlight the impor-
http:://jiibbboooooooorcemm////s auttonncodrr
6

---

AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
tance of the two key components of our method: kernel regression supervision (KR) and the adaptive generalized kernel (GK), both of which are critical for enhancing map- ping accuracy. Among these, kernel regression supervi- sion proves to be the most impactful, as the results for the AKRMap w/o KR setting are nearly indistinguishable from those of the baseline methods in Table 1. This outcome is ex- pected, as removing kernel regression supervision leaves the projection network relying solely on the neighborhood con- straint, making it functionally similar to the t-SNE method. In addition, the adaptive generalized kernel demonstrates its value by further reducing errors across various metrics, un- derscoring its effectiveness in capturing and fitting complex metric landscapes. These findings validate the necessity of both components in achieving superior performance.
5.2.2. COMPARING DIFFUSION-BASED MODEL AND ACTO-REGRESSIVE MODEL
We conduct visual comparison of two representative T21 models from different architectural families: Stable Diffusion-v2.1 (Rombach et al., 2022) (SD-2.1), a diffusion- based model, and Infinity-2B (Han et al., 2024), an auto- regressive model. Using approximately 590,000 image cap- tions from the MS-COCO (Lin et al., 2014) training dataset as prompts, we generate corresponding images using both models. To illustrate their performance differences, we vi- sualize the HPSv2 score differences between Infinity and SD (calculated as Infinity's 's score minus SD-2.1's 's score), as shown in Figure 6. Here, red regions highlight areas where Infinity outperforms SD-22.1 significantly, while blue regions indicate smaller differences.
5.2. Applications
The visualization reveals interesting patterns, particularly in Region A, where a significant performance gap exists between the two models. A deeper analysis reveals that this region is primarily associated with prompts related to sports players and athletes. By examining instance-level gener- ated images with overlay feature, we find that SD-2.1 often struggles to generate human figures or produces black-and- white photographs, whereas Infinity consistently generates high-quality, colored images of athletes; see the right side of Figure 6. This indicates that Infinity demonstrates stronger capabilities in generating human figures, especially in sports- related contexts. Overall, AKRMap effectively highlights performance distribution patterns across large-scale datasets, facilitating a detailed comparative analysis of the strengths and limitations of different models.
5.2.1. VISUALIZING LARGE T21 DATASET
AKRMap can provide an accurate overview of large T2I dataset by effectively capturing the cross- modal metric dis- tribution. Figure 5 shows the contour maps of ClipScore, HPSv2, PickScore, and Aesthetic Score distributions in the HPD dataset, generated by our AKRMap and four baseline methods: PCA, UMAP, t-SNE, and Neuro- Visualize..
In alignment with the results from the quantitative experi- ments, our method demonstrates superior performance by accurately depicting score distributions with rich local de- tails. In contrast, the baselines suffer from various limi- tations, such over-smoothing due to averaging effects, which obscure local structures. For example, PCA intro- duces pronounced block effects with sharp edges at region boundaries, significantly disrupting the overall smoothness of the visualizations. Similarly, UMAP and t-SNE often struggle with low-value regions being either overshadowed by high values or averaged out, making it challenging to identify clusters with suboptimal performance. Surprisingly, Deuro-Visualizer performs the worst among the baselines for visualizing cross-modal embedding metrics. Except the Aesthetic Score, its results are highly unstable, with excessively large estimations in out-ll- areas, and the contour maps are riddled with jagged terrains, exhibit- ing poor smoothness. This highlights the increased chal- lenge of cross modal embeddings compared to unimodal embeddings. It also further underscores the strength of our AKRMap in producing more accurate and visually coherent representations. Overall, our AKRMap achieves a superior balance between local accuracy and smoothness. Further- more, the mapping performance of traditional methods can deteriorate significantly if a smaller bandwidth is manually set, as demonstrated by additional results in Appendix C.
5.2.3. VISUALIZING FINE-TUNED MODEL
We showcase AKRMap's 's ability to analyze the impac of model fine-tuning by comparing the SD model (Rombach et al., 2022) with Dreamlike Photoreal 2.0 (DP-2.0) (dream- like.art, 2024), a fine-tuned variant of SD-1.5 optimized for photorealistic image generation. Our evaluation utilizes two distinct datasets: the MS-COCO (Lin et al., 2014) val- idation set, from which we extract 25,000 image captions as prompts, and the PartiPrompts (Yu et al., 2022) dataset, which contains 1,600 diverse prompts. Consistent with the previous analysis, we visualize the HPSv2 score differences between 2P-2.0 and SD-1.5 to illustrate their comparative performance, with the results presented in Figure 7.
Overall, the effects of fine-tuning are more pronounced on the PartiPrompts benchmark, as evidenced by a larger pres- ence of red regions in the right visualization. This is likely because MS-COCO dataset contains common captions on the web that are similar to the pretraining data of SD model, while PartiPrompts contains manually created sophisticated prompts that are challenging in different aspects. Moreover, the visualizations highlight several regions of interest (A,
7

---

AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
CLIPScore
HPSv2
PlokScore
Aesthelic Score
UMAP
1-SNE
Neuro-Visualizer
AKRMap (Ours)
Figure 5. Qualitative comparison of contour map visualizations of ClipScore, HPSv2, PickScore, and Aesthetic Score distributions in the HPD dataset generated by our AKRMap four baselines: PCA, UMAP, t-SNE, and Neuro-Visualie Visualizer.
plaeer sirligg
captures the intended styles but also excels in rendering intricate details, such as patterns on ties and textiles. Region C encompasses a variety of automotive scenes, where both the environmental contexts and the vehicles themselves ex- hibit greater realism and richer detail compared to SD-1.5. Importantly, visualization demonstrates that DP-2.0 con- sistently outperforms SD-1.5 across the entire distribution space. This indicates that the fine-tuning process success- fully enhanced the model's photorealism while maintaining its general capabilities across other domains.
Infinity
yosing Nannhell payytall boidhig
Indalta
Contour map
Figure AKRMap can
compare generative perfor- mance of auto-regressive model and diffusion model.
6. Discussion
B. and C) where DP-2.0 demonstrates significant improve- ments in photorealism and overall image quality. In Region A, further analysis of individual instances reveals that DP- 2.0 achieves enhanced realism in both static and dynamic objects (such as oranges and flying birds) and shows notable improvements in scenes, such as beach waves and sand textures. Region B primarily includes various stylistic rendi- tions of raccoon images, where DP-2.0 only accurately
Limitations. As AKRMap prioritizes metric mapping, the neighborhood preservation performance may not be com- parable to traditional DR methods. Nevertheless, due to the incorporation neighborhood constraint, AKRMap is able to maintain a desirable level of neighborhood preserva- tion, with detailed comparison results shown in Appendix A. Furthermore, for complex embedding metrics used
8

---

AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
A boul chat somer orunges IO
"Jaellratoaan and suppeet weceeeetaahhheee Hiparting
B
A whitee hird Plying brash."
Tha 1977 cnuastar
Juer locking preas elley as cloudy day."
DP-24
SQ-14
DP-28
SD-13
(8) COCO evalustion
ParliPrompts evaluation
Figure 7. AKRMap can used to show the global impact fine-tuning by comparison to base model.
in other tasks, such as those computed over sequences of embeddings (e.g., CodeBertScore (Zhou et al., 2023)), it remains uncertain how AKRMap can effectively determine a single vector representation for each instance prior to projec- tion. This presents a potential area for further investigation.
itate trustworthy visualization of cross-modal metric for transparent evaluation.
Acknowledgments
We would like extend our gratitude the anonymous re- viewers for their valuable comments. This work is partially supported by the National Natural Science Foundation of China (62172398, U23A20313, 62372471), The Science Foundation for Distinguished Young Scholars of Hunan Province (NO. 2023J110080), and Guangzhou Basic and Applied Basic Research Foundation (2024A04J6462).
Future Work. A recent study introduces Multi-dimensional Human Preference (MPS) (Zhang et al., 2024), which eval- uates embedding scores across four dimensions: Overall, Aesthetics, Alignment, and Detail. Leveraging AKRMap to visualize multi-dimensional comparisons of different T21 models would be an interesting avenue for future explo- ration. AKRMap offers the potential enhance existing cross-modal metric models through interactive human feed- back. For example, aim to fine-tune models to address domain-specific needs, such as evaluating and filtering game assets generated by T2I models.
Impact Statement
AKRMap advances transparent multi-modal evaluation by trustworthy visualization of cross cross-modal modal embedding metrics, which is pivotal for enhancing uuman-in-the-lowe evaluation in tasks like T2I generation. The strategy of jointly learning discrete point DR with continuous metrie landscape esti- mation proves highly effective in scenarios like visualizing large generated dataset, comparing different models' per- formance, and revealing the global impact of fine-tuning. Our method also provides a general framework that can be adapted to embedding- tasks such as 3D generation and pretraining data filtering.
Moreover, our method provides a versatile framework that can be adapted to other value landscape mapping challenges for sigh-dimensional data. For instance, AKRMap could be employed to visualize the distribution of predicted values in classical multivariate regression tasks. Beyond evaluation, AKRMap could also play a role in supporting trustworthy filtering of pretraining data by visualizing filtering scores across large-scale datasets. One pertinent example involves use of Aesthetic Scores to filter the massive LAION- 5b dataset (Schuhmann al., 2022). We plan to seale our approach datasets containing billions of data points, thereby increasing transparency in pretraining data selection and promoting better understanding of these processes.
metric visualization for other
Our method highlights the synergy between automatically computed metries and human's interactive interpretation. With the rapid development of AI models trained and evalu- ated on datasets of tremendous scale, it is critical to strike balance between the efficiency and trustworthiness of evalu- ation, where the transparency and visibility to humans are an essential factor.
7. Conclusion
In this paper, we introduce AKRMap, a dimensionality re- duction method to visualize cross-modal embedding metrics through kernel regression supervised projection with adap- tive generalized kernel. Based on AKRMap. we develop visualization tool to support metric-aware visualization of cross-modal embeddings for the evaluation of text-to- image generative models. Quantitative experiments and three application scenarios show that AKRMap can facil-
References
Abdi, H. and Williams, L. J. Principal component anal- ysis. Wiley Interdisciplinary Reviews: Computational Statisties, 2(4):433 459, 2010.
Ali, H., Hayawi, H. A.-.., and Botani, D. S. I. Estima-

---

AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
tion of the bandwidth parameter in nadaraya-watson ker- nel non-parametric regression based universal thresh- old level. Communications in Computation, 52(4):1476-1444 2023.
Harwath, D. and Glass, J. Deep multimodal semantic em- beddings for speech and images. In IEEE Workshop on Automatic Speech Recognition and Understanding, pp. 237-244, IEEE, 2015.
and
Cheng, S. and Mueller, K. The data context map: Fusing data and attributes into a unified display. IEEE Trans- actions on Visualization and Computer Graphics, 22(1): 121-130, 2015.
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., and Choi, Y. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 7514-7528, 2021.
Chiang, H.-Y., Camacho-Collados, J., and Pardos, 7. Un- derstanding the source of semantic regularities in word embeddings. In Proceedings of the Conference on Compu- lational Natural Language Learning, pp. 119-131, 2020.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S, Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
Crecchi, F., De Bodt, C., Verleysen, M., Lee, J., Bacciu, D., et al. Perplexity-free parametric t-sne. In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, pp. 387-392, 2020.
Hu, X., Wang, H., Lenssen, J. E., and Schicle, B. Per- sonahoi: Effortlessly improving personalized face with human-object interaction generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition ( (CVPR), 2025.
Damrich, S., Böhm, N., Hamprecht, F. A., and Kobak, D. From t-sne to umap with contrastive learning. In The International Conference on Learning Representations, 2023.
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H.,, Li, Z., and Duerig, T. Scaling up visual and vision- noisy text supervision. In International Conference on Machine Learning, pp. 4916. PMLR, 2021.
De Bodt, C., Mulders, D., Verleysen, M., and Lee, J. A. Perplexity-free t-sne and twice student tt-sne. In Eu- ropean Symposium on Artificial Neural Networks, pp. 123-128, 2018
representation learning with
Kaski, S., Nikkilä, J., Oja, M., Venna, J., Törönen, P., and Castrén, E. Trustworthiness and metrics in visu- alizing similarity of gene expression. BMC Bioinfor- matics, 4(1):1-13, 2003. doi: https://doi.rr//1..11 1166/ 1471-2105-4-48.
dreamlike.art. Dreamlike photoreal 2.0 photorealistic model based stable diffusion 1.5, 2024. URL https: http://hggggggg..ee.e/ cttp:/af.aee.c/aaaaaeeeeeeeeeeee www/reemmlkeeeaeeeeeeeeeeeeeeeee dreaml ke weeeeeeepottaaaaa-2 0.
Elhamod, M. and Karpatne. A. Neuro-visualizer: A novel auto-encoder-based loss landscape visualization method with an application in knowledge-guided machine learn- ing. In International Conference on Machine Learning, 2024.
Kirstain, Y., Polyak, A., Singer, U., Matiana, S., Penna, J., and Levy, O. Pick-a-pie: An open dataset of user pref- erences for text-to-image generation. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
Gisbrecht, A., Schulz, A., and Hammer, B. Parametric nonlinear dimensionality reduction using kernel t-sne. Neurocomputing, 147:71-82, 2015.
Le, I.,, Patterson, A., and White, M. Supervised autoen- coders: Improving generalization performance with un- supervised regularizers. Advances in Neural Information Processing Systems, 31, 2018.
Goldberg, Y., Zakai, A., Kushnir, D., and Ritov, Y. Manifold learning: The price of normalization. Journal of Machine Learning Research, 98), 2008.
Li, H., Xu, Z.,, Taylor, G., Studer, C., and Goldstein, T. Visualizing the loss landscape of neural nets. Advances in Neural Information Processing Systems, 31, 2018.
Hajderanj, L.. Weheliye, I., and Chen, D. A new super- vised t-SNE with dissimilarity measure for effective data visualization and classification. In Proceedings of the International Conference on Software and Information Engineering, pp. 232-236, 2019.
Liang, V. W., Zhang, Y., Kwon, Y., Yeung, S., and Zou, J. Y. Mind the gap: Understanding the modality gap in multi-moda contrastive representation learning. Ad- vances in Neural Information Processing Systems, 35: 17612-17625, 2022.
Han, J., Liu, J., Jiang, Y., Yan, B., Zhang, Y., Yuan, Z., Peng, B., and Liu, X. Infinity: Scaling bitwise autoregres- sive modeling for high resouutioo image synthesis. arXiv preprint arXiv:2412.0 044431, 2024.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft
10

---

AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
COCO: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer. 2014.
Silverman, B. W. Density estimation for statistics and data
analysis. Routledge, 2018.
Van Der Maaten, L. Accelerating t-sne using tree-based algorithms. The Journal of Machine Learning Research, 15(1):3221-3245, 2014.
McInnes, L., Healy, J., Saul, N., and Großberger, L. UMAP: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861, 2018.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of Machine Learning Research, 9(11,, 2008.
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. Distributed representations of words and phrases and their compositionality. Advances in Neural Informa- tion Processing Systems, 26, 2013.
Wang, X., Wu, Z., Huang, W., Wei, Y., Huang, Z., Xu, M., and Chen, W. VIS+AI: integrating visualization with artificial intelligennce for efficient data analysis. Frontiers of Computer Science, 17(6):176709 2023a.
Narayan, A., Berger, B., and Cho, H. Assessing single- cell transcriptomic variability through density preserving data visualization. Nature Biotechnology. 39(6)7765-774, 2021.
Wang. Z. J., Hohman, F., and Chau, D. H. WizMap: Scal- able interactive visualization for exploring large machine learning embeddings. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 516-523, 2023b.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 2023.
Wang, Z. J., Montoya, E.,, Munechika, D., Yang, H.,, Hoover, B., and Chau, D. H. DiffusionDB: A large-scale prompt gallery dataset for text-to-image generative models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 993-911, 2023c.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Weglarezyk, S. Kernel density estimation and its application. In ITM Web of Conferences, volume 23, pp. 00037. EDP Sciences, 2018
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint (iXiii2224.06122 1(2):3, 2022.
Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., and Li, H. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. 2023a.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resse resolutioo image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
arXiv preprint arXiv:2306.092:
Wu, X., Sun, K., Zhu, F., Zhao, R., and Li, H. Human preference score: Better aligning text-to-image models with human preference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2096- 2105, 2023b.
Sainburg, T., McInnes, L., and Gentner, T. Q. Parametric umap embeddings for representation and semisupervised learning. Neural Computation, 33(1 28:1 19077 2021.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. Advances in Neural Information Processing Systems, 29, 2016.
Xia, J., Huang, L., Lin, W., Zhao, X., Wu, J., Chen, Y., Zhao, Y., and Chen, W. Interactive visual cluster analysis by contrastive dimensionality reduction. IEEE Transactions Visualization and Computer Graphics, 29(1): 744, 2023.
Schölkopf, B., Smola, A., and Müller, K.-R. Kernel princi- pal component analysis. In International Conference Artificial Neural Networks, pp. 583-588. Springer, 1997.
Xia, J., Huang, L., Sun, Y., Deng, Z., Zhang, X. L., and Zhu, M. A parallel framework for streaming dimension- ality reduction. IEEE Transactions on Visualization and Computer Graphics, 30(1):142-152, 2024.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 35278-252944 2022.
Xu, J., Mei, T., Yao, T., and Rui, Y. MSR-VTT: A large video description dataset for bridging video and language. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5288-5296, 2016.

---

AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
Xu, J., Liu, X., Wu, Y., Tong. Y., Li, Q. Ding, M., Tang, J., and Dong, Y. Imagereward: Learning and evaluating hu- man preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Ye, Y., Hao, J., Hou, Y., Wang, Z., Xiao, S., Luo, Y., and Zeng, W. Generative Al for visualization: State of the art and future directions. Visual Informatics, 8(2):43-66, 2024.
Ye, Y., Xiao, S., Zeng, X., and Zeng, W. ModalChorus: Vi- sual probing and alignment of multi-modal embeddings via modal fusion map. IEEE Transactions Visualiza- tion and Computer Graphics, 31(1):294330) 2025.
Yu, J. Xu, Y., Koh, J. Luong, T., Baid, G., Wang. Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., al. Scal- ing autoregressive models for content-rich text-to-image generation. Transactions Machine Learning Research, 2022.
Zhang, S., Wang, B., Wu, J., Li, Y., Gao, T., Zhang, D., and Wang. Z. Learning multi-dimensional human prefer- ence for text-to-image generation. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8018-8027, 2024.
Zhou, S., Alon, U., Agarwal, S., and Neubig, G. Code- BERTScore: Evaluating code generation with pretrained models of code. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp- 13921-13937, 2023.
12

---

AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
Appendix
A. Performance in Traditional Neighborhood Preservation Objective
Traditional dimensionality reduction methods often seek to preserve the neighborhood in high-dimensional space in the visualization scatterplot. Since the target of our method is not to optimize the neighborhood preservation, we do not expect method to be better than traditional dimensionality reduction method in this regard. In Table 2, we provide statistics of traditional neighborhood trustworthiness metric (Kaski et al., 2003) to show that our method does not significantly harm the neighborhood property. Furthermore, we note that in our application where we focus on continuous metric, cluster separation is not meaningful target since we are not looking at discrete categories as in classification tasks. As shown in Figure 8, for the HPSv2 metric on HPD dataset, even in scatterplot view AKRMap can reveal the distribution of metric significantly better than other traditional DR methods. Here we also note that we deliberately select larger perplexity for the t-SNE method because of the large scale of the HPD dataset, and we show in Figure 9 that smaller perplexity will lead to worse results.
Table 2. Neighborhood Trustworrhiness metrics for visualization of HPSv2 embeddings on HPD.
n=20
n=30
n=
n=50
PCA
0.7243
0.7246
0.7245
0.7242
t-SNE
0.8945
0.8786
0.8667
0,8575
UMAP
0,8604
0.8467
0.8380
0,8309
AKRMap
0.8284
0.8287
0.8283
0.8277
(a) I-SNE (0erplexit==100)
(b) UMAP (n_neighbors=100)
(c) PCA
(d) AKRMap
Figure 8. Comparison of scatterplots produced by different DR methods for HPSv2.
440
-20
Perplexity
Perplexiy
Perplexity =30
Figure 9. Results of smaller perplexity for t-SNE on HPSv2.
B. Mapping Accuracy Evaluation for in-sample Points
In this section, we record the quantitative results for in-sample mapping accuracy of different methods, as shown in Table 3. The results combined with Table indicate that AKRMap outperforms other methods consistently for both in-sample and out-of-sample positions.
13

---

AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
Table Quantitative comparison of mapping accuracy in-sample points of HPD dataset.
CLIPScure
IIPSy2
PickScure maape
Aesthetic
Method
mac
mape
P123c
P77.00
7106
mac
rmac
PCA
4,0494
22,8348
5.3954
1.3744
5.3870
1.7690
1,1341
5.7472
1.4437
0.5209
10.6224
0.6749
1-SNE
4,0573
22.8162
5.3798
1.3567
5.3259
1.7606
1.1200
5.6774
1.4340
0.5192
10.5890
0.6728
UMAP
4.0906
22.8612
5.4106
1.4555
5.7803
1,9073
1.8044
5,9824
1,5145
0.5241
10.6851
0,6782
SAF
4,2591
23,2463
5.5196
1,5323
5,9966
1,9738
1.2731
6,4886
1.6770
0.5207
10.6322
0.6746
Neuro-Visualiinn
10.4468
39.8135
12,1020
1.8716
7,0312
2.2237
2,0031
9.6188
2.3844
0,3102
0,4009
Map (w/o KR)
4.0506
22,6324
5.3415
1,3696
5.3706
1,7761
1.1237
5.6914
1.4343
0.3619
7.2980
0.4654
AKRMap (w/ GK)
1.6359
8,3539
2.1271
L.0499
4.0655
1,3358
0.9062
4,5448
1.1537
0,3390
6.7634
0,4320
AKRMap
1,1652
5,4238
1.5029
0,6834
2.6159
0.8980
0.6815
3,4181
0.8862
0,3142
6.1747
0.4018
C. RBF Kernel Settings for Traditional DR Methods
Here provide details and more experiment results concerning RBF kernel settings for traditional projection method. We show in Table 4 and Table 5 that traditional methods cannot achieve satisfactory performance regardless of the kernel parameter settings. Specifically, we test different commonly used bandwidth selection methods:
Plug-in method (Silverman) (Silverman, 2018). This is the default method we used in the paper due to its efficiency. For d-dimensional data, the Silverman's rule is given by:
4
444
h =
6,
(d + 2)m
where ē is the estimated standard deviation.
Adaptive local bandwidth (ALB) (Cheng & Mueller, 2015). This method computes bandwidth for each point by adapting pre-computed bandwidth based on local estimated density f (x;).
1/n
h; A h, X; = (G/S(,))) G=
II
Cross-validation (LOOCV) (Weglarczyk, 2018). The cross validaain is performed in a Leave-one-ou manner:
n CV(h) n-1
Σ Iy 8 (xz))) (xz), 1=1
(9)
where 8, (Iz) is the leave-one-out estimator for yj that computed on all data points except Zy, and w(x,) is nonnegative weight function (all set to by default). Then h is selected to minimize this validation error.
Table Quantitative comparison of sat-ol-sample mapping accuracy of different bandwidth selection methods for traditional DR.
CLIPScore
HPSv2
PickScore mope
Method
mae
mape
rmss
mae
mope
rm7se
moe
rmss
PCA+Silverman
4.0042
16.5506
5.1142
1.4444
5.7870
1.8749
1.1497
5.9209
1.4686
PCA+ALB
4.0163
16.5445
5.1148
1.4390
5.7662
1.8651
1.1469
5.9140
1.4658
PCA+LOOCV
4.0000
16.5140
5.1034
1.4347
5.7492
1.8598
1.1415
5.8769
1.4557
Siveran
4.0241
16.6643
5.1523
1,4361
5.7568
1.8629
1.1579
5.9429
1.4640
I-SNE+ALB
4.0215
16.6381
5.1497
1.5147
6.1203
1.9546
1.1463
5.8838
1.4524
t-SNE+LOCVV
4.0205
16.6276
5.1366
1.5250
6.1516
1,9573
1.1520
5,9210
1.4639
UMAP+Silverman
4.0819
16.9066
5.2179
1.4432
5.7817
1.8725
1,2214
6.2721
1.5536
UMAP+ALB
4.0142
16.7284
5.1332
1,4171
5.6262
1.8285
1.2019
6.1807
1.5276
UMAP+LOOCV 4.0417
16.7337
5.1617
1.4830
5,9180
1.9233
1.2715
6.5419
1.6274
In addition, we also show that it is not possible to achieve accurate mapping by manually setting smaller bandwidth than the automatically selected one. fact, this will evidently cause severe overfitting issue and result in fragmented blocks in the map, as shown Figure 10 t-SNE method. This also causes the quantitative accuracy of the mapping to drop. For example, for the case in Figure 10, the error has increased as shown Table 6.
14

---

AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
Table Quantitative comparison of in-sample mapping accuracy of different bandwidth selection methods for traditional DR.
CLIPScore
HPSv2
PickScore
Method
Traee
mnpe
TIBSc
made
madp
TYnse
Trape
TWnse
4.0494
22.8348
5.3954
1.3744
5.3870
1.7690
1.1341
5.7472
1.4437
PCA+ALB
4.0415
22.8072
5.3806
1.3702
5.3707
1.7621
1.1284
5.7254
1,4394
PCA+LOOCV
4.0505
22,9057
5.3927
1.3747
5.3874
1.7674
1.1351
5.7502
1.4447
L-SNE+Silverman
4.0573
22.8162
5.3798
1.3567
5.3259
1.7606
1.1200
5.6774
1.4340
L-SNE+ALB
4.0476
22.7261
5.2578
1.3549
5.3168
1.7564
1.1164
5.6626
1,4292
I-SNE-LOOCV
4.0447
22.6818
5.3511
1.3545
5.3158
1.7569
1.1131
5.6390
1,4270
UMAP4Silverman
4.0906
22.8612
5,4106
1,4755
5.7803
1.9073
1.1804
5.9824
1,5145
UMAP+ALB
4.0675
22.9090
5.3831
1.4621
5.7202
1.8871
1.1729
5,9458
1.5051
UMAP+LOOCV
4.0684
22.7450
5,3680
1,4699
5.7576
1.9013
1.1730
5.9430
1.5083
28.0
26.4
23.6
23.0
(a) auto-selected bandwidth
(b) manually set 0. bandwidth
Figure 10. Attempts manually selecting smaller bandwidth for traditional methods may lead to worse mapping performance with severe overfitting, as exemplified by the t-SNE visualization of HPSv2 score distribution.
D. Zoom in by Different Grid Numbers
This section shows the zoom-in effect of AKRMap when increasing the grid number. As shown in Figure 11, AKRMap can accurately estimate the local distribution of cross-modal metric and adds detail on demand. Specifically, with a small grid number of 30 by 30, AKRMap can already achieve the level of detail close to most baseline methods with grid number of 500 by 500. When increasing the grid number, contour of metric distribution.
in Figure 11 that method accurately show the detail
E. Hy Tlyperparameter
In this section we discuss some hyperparameter trade-off our method with additional experimental results. First, we show the trade-off effect by different À value to weight the regression loss and KL loss, with (out-of-sample) quantitative results shown in Table 7, and visualization effects shown in Figure 12. Next, show the scatterplot view of our ablation studies in Figure 13 demonstrate the effect of our proposed component in large-scale DR point plot. Finally, in Figure 14, we illustrate some qualitative and quantitative effect of setting the train-val weights W2 to be zero to demonstrate the necessity of the trlin-val balance scheme in our kernel regression guidance loss.
Table 6. Impact of manually setting smaller bandwidth for Figure
maca.
mape:
rmSS
MaCout
mapCout
rMSCout
auto-selected bandwidth
1.3567
5.3259
1.7606
1.4361
5.7568
1.8629
0.1x bandwidth
1.5403
6.0284
2.0470
1.7050
6.7994
2.1951
15

---

AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
grid=30
grid=50
grid=500
Figure 11. Zoom-in effect by different grid numbers in our method.
Table 7. Experimental results on trade-off effect by À settings.
CLIPScure
HPS+2
PickScore
Aesthelic Score
Method
trust
trust.
trust
trust.
1.7624
1.4027
0.125
1.8707
2,4253
0.7672
0.8108
1.1200
0.8284
0.7712
1.0225
0.8143
0.4305
0,5488
0.8704
0.25
4.4237
0.6865
0.5621
08111
0.5511
0,7413
0.7990
0.4399
0.5149
0.8626
À 0.5
3.4811
4,4620
0.7083
0.4453
0.6554
0,7847
0.4597
0.6346
0.7791
0.3875
0.4925
0.8545
F. Test on Other Modality Embeddings
We further perform quantitative experiments CLIP text-video embeddings MSR- VTT dataset (Xu et al., 2016) which consists of 200,000 text-video pairs (Table 10), well as text-audio embeddings and image-audio embeddings Flickr 8k Audio Caption Corpus dataset (Harwath & Glass, 2015) consisting of 40,000 pairs (Table 9 and Table 8). Overall, the results indicate that our method can improve upon traditional DR in test set mapping accuracy.
Table ImageBind image-audio embeddings mapping accuracy on Flickr Audio Caption Corpus.
PCA
1-SNE
UMAP
AKRMap
mae
4.1239
3.9576
3.8811
1.9425
mape
19.1321
18.3808
18.1024
8.9521
rmse
5.1737
5.0254
4.8633
2.6425
G. Weight Balancing Mechanism
We also test automatic weight balancing method, using a sigmoid function to adjust the weight KL loss.
w(e, p)) = akk(2 - H)) = e k(z-n)*
(10)
where y is a threshold of acceptaale KL loss which we set to 2 by default and & is a parameter to control the decay rate which we default 1. Then the total loss becomes:
L AMSE, +w(K, KI,
(11)
where À is fixed to the default value 0.125.
Alternatively, for users who care more about the neighborhood preservation, we can this method to weight the MSE term with user-specified threshold μi:
L = (MSS,,, AMSE, + KKL.
(12)
The experimental results for different y and μn settings are presented Table
16

---

AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
CLIPScore
HPSv2
Sick
Aesthetic Score
0.05
0.125
0.25
0.5
Figure 12.
AKRMap mapping results different settings.
Table 9, ImageBind text-audio embeddings mapping accuracy on Flickr 8k Audio Caption Corpus.
PCA
t-SNE
UMAP
AKRMap
3.7194
3.5992
3.6801
1.8629
mape
21,1113
20.3858
20.8498
10.1821
rmse
4.7175
4.5607
4.6541
2.4841
H. Interactive Features Evaluation
We conduct study among 12 users interactive features including and overlay. They complete 7 Likert-scale survey after using interactive visualization of HPSv2 score HPD dataset. As shown in Figure 15, users generally appreciate our interactive features in several usability aspects including effectiveness, ease of use, interpretability. and future use.
I. Convergence Properties of the Adaptive Generalized Kernel Regression
In this appendix, we summarize the theoretical conditions and practical considerations related the convergence of our adaptive generalized kernel regression method, which extends the classical Nadaraya-Wand Watson kernel regression framework.
To guarantee convergence in terms of mean squared enror (MSE), the following conditions are essential:
1) Integrability of the generalized kernel on R.2: The kernel defined as:
K (z) = (1 + 20)1
(13)
For K ((x)) to be integrable and finite over ?R? parameters must satisfy > 0 and B > ensuring sufficient decay of the kernel function at infinity.
2) Growth behavior of the parameter with sample size N: The parameter a must effectively increase as the
17

---

AKRMup: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
AKRMap w/o KR
AKRMap wio GK
AKRMap
CLIPScore
HPSv2
PickScore
Aesthetic Score
Figure 13.
Scatterplots of ablation.
Table CLIP text-video embeddings mapping accuracy on MSR- VTT datasets.
PCA
t-SNE
UMAP
AKRMap
1.4908
1.4201
1,4562
0.6763
mape
8.3080
7.9589
8.1636
3.4832
rmse
2.1231
2.0216
2.0723
0.9578
sample size N grows, analogous to the classical kernel bandwidth decreasing with larger N. This condition is critical for establishing consistency and uniform convergence of the estimator.
3) Smoothness of the regression target function s<)): The target function s(x) should satisfy smoothness assumptions such as Lipschitz continuity. This is standard requirement in kernel regression theory to control the bias term and facilitate uniform convergence bandwidth parameters tuned.
In our training procedure, we enforce the non-negativity of parameters and / through reparameterization (by squaring them) and include them within set of learnable parameters updated via backpropagation. Intuitively, this process pushes the network automatically identify suitable decay rates for the kernel in each epoch mini-batch, adapting progressively to match the magnitude scale of the Nadaraya-Watte Watson kernel. Empirical experiments demonstrate that the learned parameter values (a, 99) converge successfully. For example, the values for HPSv2 are (68.57, 1.61), for PickScore are (59.13, 1.35), for CLIPScore are 7104.70,3 3. 38) and for Aesthetic Score are (74.95, 1.11). These results indicate that our data-driven approach effectively satisfies conditions (1) and (2) mentioned above.
In practical multimodal scenarios. although the target function might not strictly satisfy Lipschitz continuity, numerical experiments indicate that kernel regression methods remain robust long as the target distribution does not exhibit extreme inegularitiis, such highly discontinuous severely jagged patterns. In other words, piecewise smooth locally continuous target functions can yield stable and accurate kernel regression estimates in practice. Specifically, in our scenario involving high-dimensional to oow-dimensional mappings, there exists an continuous relationship between evaluation metrics and the projected cross-modal embeddings. For example, CLIPScore, which is computed using cosine similarity between image and text embeddings, is an obvious continuous function between the original high- high-dimensional embeddings and the metric. Furthermore, our parametric projection network is an explicit continuous and differentiable function. The Implicit
18

---

AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings
HPS
maee0.81 81,
1.41, mapees mase11.22
1.37
omsee0.00
moer0.85
PickScore
118
OUE n.2e-..77 3.p moc-1 .22 miwend 6A 12 masen..e 54
DUE macet, 12, ni00c-6.77, misec1.43
QUE maeeO 76, mupens 02, mmseet 04 in mwwett 75, mapped 65, mosem..0
In mowee 100, mipees 52, mosel
AKRMp
(b) W1=0
(c) W2=0
Figure 14. Comparison with setting wi and w2 our MSE loss on HPSv2.
Table 11. Weight balancing results of I11 and L2 HPSv2.
mape
rmse
trust
w/o weight balance
0.8108
3.1935
1.1200
0.8284
2
0.5118
2.0318
0.7464
0.8053
= 1.8
0.4963
1.9632
0.7051
0.7979
= 1.6
0.5336
2.1115
0.7772
0.8059
=
1.0427
4.1393
1.3787
0.8399
P1 0.8
0,9610
3.8069
1.2686
0,8316
141 0.6
1.0118
1.9999
1.3478
0.8426
Zoom
Overlay
Elaali
Inderprnablle
Fusuro uap
Efecttive
Interpretalio
Future usd
Eary
Easy
Figure 15. User study results on interactive features.
Function Theorem can then ensure the continuity of the metric w.r.f the projected 2D embeddings. Thus, our practical setup implicitly fulfills the smoothness assumption (condition 3) required for convergence.
Furthermore, the mapping function s(x) continuously updated during dimensionality reduction (projection). Our adaptively updated generalized kernel not only naturally accommodates complex distributions encountered practice but also ensures consistency in metric spaces through supervised kernel regression. Regarding projection stability, kernel regression loss be regarded as regularization term that enforces the global structure of the projection, it explicitly pushes the projected sample points to regress a global metric distribution.
19


Reviews:
[{"review_id": "bhCsphbR5c", "reviewer": "Reviewer_m4Lr", "summary": "The paper introduces a new framework for dimensionality reduction and visualization of cross-modal embeddings. The method incorporates adaptive kernel regression to more accurately capture metric distributions in the projection space. It jointly optimizes a projection network and kernel function, enabling more reliable scatterplots and contour maps with interactive features like zooming and overlaying. Experimental results demonstrate that the proposed method outperforms existing approaches, enhancing the trustworthiness of multi-modal model evaluations, particularly in text-to-image comparisons and human preference assessments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive feedback and constructive comments that help us improve our work. Below, we answer your specific questions, with two additional figures provided at the link:https://docs.google.com/document/d/e/2PACX-1vR8toOEJ8KrADdAzAXJbIMazphKXlKq4K4oqwDxAvvpLzqPu0jufg7ns67Argf36yCDXGt34dL85eZP/pub\n\n## Q1: Do the results in Table 1 refer to only out-of-sample points, or do they include both in-sample and out-of-sample points?\nA1: The results in Table 1 refer to only out-of-sample points, while the in-sample points results are presented in Table 3 in appendix B.\n\n## Q2: Could the authors provide quantitative experiments on additional datasets beyond the HPD dataset?\nA2: We have added experiments on three different types of embeddings on two additional datasets: ImageBind audio-text embeddings and audio-image embeddings on Flickr 8k Audio Caption dataset, as well as CLIP text-video embeddings on MSR-VTT datasets, showing that our method can improve upon traditional DR in test set mapping accuracy (Please refer to Table 1-3 below). The results are as follows:\n### Table 1. ImageBind text-audio embeddings mapping accuracy on Flickr 8k Audio Caption Corpus.\n||PCA|t-SNE|UMAP|AKRMap|\n|-|-|-|-|-|\n|mae|3.7194|3.5992|3.6801|1.8629|\n|mape|21.1113|20.3858|20.8498|10.1821|\n|rmse|4.7175|4.5607|4.6541|2.4841|\n### Table 2. ImageBind image-audio embeddings mapping accuracy on Flickr 8k Audio Caption Corpus.\n||PCA|t-SNE|UMAP|AKRMap|\n|-|-|-|-|-|\n|mae|4.1239|3.9576|3.8811|1.9425|\n|mape|19.1321|18.3808|18.1024|8.9521|\n|rmse|5.1737|5.0254|4.8633|2.6425|\n### Table 3. CLIP text-video embeddings mapping accuracy on MSR-VTT.\n||PCA|t-SNE|UMAP|AKRMap|\n|-|-|-|-|-|\n|mae|1.4908|1.4201|1.4562|0.6763|\n|mape|8.3080|7.9589|8.1636|3.4832|\n|rmse|2.1231|2.0216|2.0723|0.9578|\n\n## Q3: Do the authors think the method would work on other modalities beyond text-to-image? Could the authors elaborate on this?\nA3: Our method works effectively across various modalities and shows potential for further extension depending on the complexity of embeddings. As mentioned in A2, we've performed additional experiments on text-audio, image-audio, and text-video embeddings with positive results (please refer to Tables 1, 2, and 3 in Q2 for quantitative results), and we also provide visualization of dimensionality reduction results for text-to-video in Figure 1 in our provided link. In addition, we've also successfully applied our approach to image-to-3D tasks, where we visualized and compared two video-diffusion-based 3D generation models on 360-degree videos of generated 3D assets from the GSO dataset (Figure 2 in our provided link). Some complex cases such as 3D and video generation involve multimodal embeddings (e.g., CLIP) for evaluation. Even though these tasks require embeddings from multiple views of a single instance, we can average these embeddings before projection, as the linear operation in averaging is associative with the dot product operation used in computing CLIPScore.\n\n## Q4: The results in Table 1 lack standard deviations.\nA4: Thank you for the suggestion. We have rerun most of regression tests 10 times in Table 1 (excluding the auto-encoder method that takes too long to run and they are clearly worse than other methods in previous experiments) to obtain the standard deviation: \n### Table 4. Standard deviation for HPSv2\n||PCA|t-SNE|UMAP|w/o KR|w/o GK|AKRMap|\n|-|-|-|-|-|-|-|\n|mae| 0.0073 | 0.0063 | 0.0079 | 0.0054 | 0.0033 | 0.0058|\n|mape| 0.030| 0.026 | 0.035 | 0.023 | 0.015 | 0.023|\n|rmse| 0.010 | 0.0089 | 0.012 | 0.0060 | 0.0047| 0.0085|\n### Table 5. Standard deviation for CLIPScore\n||PCA|t-SNE|UMAP|w/o KR|w/o GK|AKRMap|\n|-|-|-|-|-|-|-|\n|mae| 0.016 | 0.016  | 0.019 | 0.011 | 0.0051 | 0.0042 |\n|mape| 0.036 | 0.041 | 0.080 | 0.026 | 0.028 | 0.018 |\n|rmse| 0.013 | 0.012 | 0.037 | 0.010 | 0.011 | 0.0046 |\n\n### Table 6. Standard deviation for PickScore\n||PCA|t-SNE|UMAP|w/o KR|w/o GK|AKRMap|\n|-|-|-|-|-|-|-|\n|mae| 0.0061 | 0.0053 | 0.0080 | 0.0024 | 0.0029 | 0.0032 |\n|mape| 0.021 | 0.024 | 0.031 | 0.015 | 0.016 | 0.0086 |\n|rmse| 0.0055 | 0.0057 | 0.0069 | 0.0031 | 0.0037 | 0.0042 |\n\n### Table 7. Standard deviation for Aesthetic Score\n||PCA|t-SNE|UMAP|w/o KR|w/o GK|AKRMap|\n|-|-|-|-|-|-|-|\n|mae| 0.0028 | 0.0047 | 0.0027 | 0.0019 | 0.0033 | 0.0026|\n|mape| 0.077 | 0.12 |0.070 | 0.048 | 0.064 | 0.071|\n|rmse| 0.0037 | 0.0059 | 0.0033 | 0.0021 | 0.0030 | 0.0032 |\n\n## Q5: It would be beneficial to provide the code for AKRMap, either by uploading the files or via https://anonymous.4open.science.\nA5: Thank you. To facilitate convenient test of the code which requires large precomputed embedding npy files exceeding github upload limit, we have uploaded the code files to an anonymous osf: https://osf.io/7mwv3/?view_only=18869f3a1cf542b4bf2f39db4e19abd7. The link may take a while to load and it is better to paste it on another tab. You can find the files under the Files tab. We will open source the code on github after the review period.", "claims_and_evidence": "The claims made in the paper are supported by evidence from both quantitative results and visualizations, demonstrating that the proposed method outperforms other dimensionality reduction techniques. However, the results in Table 1 lack standard deviations and rely on a single dataset (HPD). Additionally, while the visual experiments provide evidence of AKRMap’s ability to handle large datasets, effectively compare model performance, and visualize the impact of fine-tuning, it would be useful to understand how AKRMap generalizes to other cross-modal tasks beyond T2I.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are both theoretically sound and practically relevant for the cross-modal embedding problem in text-to-image generation. However, while the HPD dataset provides a comprehensive evaluation, including experiments on additional datasets would further demonstrate the method's reliability and robustness across diverse embedding scenarios.", "theoretical_claims": "There are no formal proofs in this paper, but the approach is grounded in empirical design choices with inspiration from prior studies.", "experimental_designs_or_analyses": "The method is compared against different existing baselines, evaluated using different metrics. However, the quantitative results lack standard deviations and rely solely on the HPD dataset. Overall, the experimental design is generally sound, but it would benefit from including statistical measures of error and evaluating the method on a broader range of datasets.", "relation_to_broader_scientific_literature": "The proposed method builds upon dimensionality reduction, kernel regression, and cross-modal evaluation research, addressing gaps in metric-preserving visualizations. It proposes a novel method for cross-modal embeddings, specifically optimizing dimensionality reduction for cross-modal metrics. Unlike traditional methods, it is designed to enhance metric preservation and interactivity, offering a more trustworthy and interactive visualization approach for large-scale multi-modal datasets.", "essential_references_not_discussed": "I am unaware of any essential references missing from the paper.", "other_strengths_and_weaknesses": "The paper is well-written and easy to follow. The images are clear, and the captions are self-explanatory. The idea is novel, with clear contributions to the field. However, a broader experimental evaluation could strengthen the paper.", "other_comments_or_suggestions": "* It would be beneficial to provide the code for AKRMap, either by uploading the files or via [https://anonymous.4open.science/](https://anonymous.4open.science/).\n* As mentioned previously, incorporating standard deviations or other statistical error measures in the experimental results would strengthen the paper's claims by providing a more comprehensive view of the performance and reliability of the proposed method.", "questions_for_authors": "* Q1: Do the results in Table 1 refer to only out-of-sample points, or do they include both in-sample and out-of-sample points?\n* Q2: Could the authors provide quantitative experiments on additional datasets beyond the HPD dataset? \n* Q3: Do the authors think the method would work on other modalities beyond text-to-image? Could the authors elaborate on this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j2RWtHkNOf", "reviewer": "Reviewer_eXC7", "summary": "The paper proposes AKRMap, an adaptive kernel regression-based dimensionality reduction (DR) method for trustworthy visualization of cross-modal embeddings. Unlike conventional DR techniques (e.g., PCA, t-SNE, UMAP), AKRMap jointly learns a supervised projection network and an adaptive kernel regression loss, improving visualization accuracy of cross-modal metric landscapes. It integrates a generalized adaptive kernel, enabling enhanced contour mapping of evaluation metrics like CLIPScore and Human Preference Score (HPSv2). Empirical results demonstrate AKRMap's superior visualization fidelity over existing methods, facilitating human-in-the-loop analysis of text-to-image (T2I) models, comparative model assessment, and fine-tuning impact evaluation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your great review helping us improve the work. We respond to your questions below and provide additional figures and tables in the link: https://docs.google.com/document/d/e/2PACX-1vQDqGw_hxxTv_CQbFS_zdI5LtgIE6HTfAYl7uBRgpczNScFBuja6pwbuYrV-76D5CleVrujYDH0O1yP/pub\n\n## Q1 Have you tested AKRMap on other cross-modal tasks (e.g., video-text, speech-text embeddings)? \n\nA1: Thank you for the suggestion. We have added quantitative experiments on text-audio, image-audio, and text-video embeddings, as well as a qualitative example on image-3D embeddings, demonstrating that our method can generalize to different modalities and tasks. The results are presented in Fig. 1, 2 and Tab. 1, 2, 3 in the link. Please see our response to R1 (m4Lr) Q2 and Q3 for the detailed setup.\n\n\n## Q2 Given Table 2 shows lower neighborhood trustworthiness than t-SNE/UMAP, have you considered a weighted trade-off mechanism between metric mapping and local structure preservation?\n\nA2: We show the manual adjustment results in response to R3 (63Bw) Q4. Following your suggestion, we test a simple weight balancing technique on the HPSv2 score, where we use a sigmoid function centered at a threshold $\\mu$ to adjust the weight of the KL loss:\n $w(x, \\mu) = \\sigma(k (x - \\mu)) =\\frac{1}{1 + e^{-k (x - \\mu)}}$,\n$L_1=\\lambda MSE_r+ w(KL, \\mu) \\cdot KL$.\n Under this setting we find that we can further decrease the mapping error from 0.8 to 0.5. We also test a similar thresholding by $\\mu_1$ on the MSE, which can increase the trustworthiness: \n\n### Table 1: weight balancing result:\n||mae|mape|rmse|trust|\n|-|-|-|-|-|\n|original|0.811|3.194|1.120|0.828|\n|$\\mu=2$|0.512|2.032|0.746|0.805|\n|$\\mu_1=0.6$|1.012|1.999|1.348|0.843|\n\n## Q3 Have you conducted any human evaluation to measure the usability of AKRMap’s interactive zoom/overlay features or response time?\n\nA3: Thank you. We have added a user evaluation with 12 participants for the interactive features, as detailed in Fig. 3 and Tab. 4 in the link. We also record response time on colab notebook. For each zoom interaction on a map, the extra time cost by our dynamic update of contour and sampled points is between 0.0063 and 0.0170 seconds, while the total response time is roughly under 1 second, but it is due to plotly’s innate time cost. \n\n## Q4 Does the adaptive kernel regression method have convergence guarantees? Could certain assumptions ensure stability in the projection process?\n\nA4: We have attempted to analyze theoretical convergence guarantees for our adaptive generalized kernel, which hold under the following assumptions:\n1. Parameters must satisfy $\\alpha > 0$ and $\\beta \\geq 1$, ensuring the generalized kernel function $K(x)=(1 + \\alpha|x|^{2\\beta})^{-1}$ is integrable over $R^2$.\n2. Certain smoothness conditions (e.g., Lipschitz continuity) on the 2D regression target function $s(x)$ help control the bias term.\n3. Parameter $\\alpha$ should increase appropriately with sample size $N$, analogous to bandwidth reduction in classical kernel regression.\n\nIn practice, we enforce non-negative parameters $\\alpha, \\beta$ by squaring reparameterization, and optimize them via backpropagation. Empirical experiments show successful convergence of learned parameters (e.g., HPSv2: (68.57, 1.61); PickScore: (59.13, 1.35); CLIPScore: (104.70, 3.18); Aesthetic Score: (74.95, 1.11)), validating conditions 1 and 3. The continuous mapping between the embeddings and the metrics like CLIPScore can guarantee condition 2, although specific bounds need more complex derivation.\nRegarding projection stability, our kernel regression loss can be regarded as a regularization term that enforces the global structure of the projection, as it explicitly pushes the projected sample points to regress a global metric distribution.\nWe will provide more detailed discussion of the theoretical aspects in updated appendix.\n\n## Q5 Minor grammatical issues in some sections (e.g., \"contour estimation errors back into the projection process\" could be clearer).\n\nA5: Thank you. We will fix the issue in the updated version.\n\n## Q6 Essential references not discussed.\n\nA6: Thank you. We will add discussion of your suggested references in the updated version.\n\n## Q7 However, the claim that AKRMap optimally balances metric mapping accuracy and neighborhood preservation lacks formal theoretical justification:\nA7: Thank you. We did not intend to claim that AKRMap achieves optimal balance, as we mainly focus on mapping accuracy while the neighborhood only serves as a secondary constraint. In addition, we believe that users can make findings under different trade-off parameters (like tuning perplexity for t-SNE). We will revise our writing to tone down our previous misleading claim. But we also think that your question is very inspiring as how we can define what is the optimal balance between trade-off objectives in all supervised dimensionality reduction methods. We hope to dig deeper into this question in future research.", "claims_and_evidence": "The paper's claims are mostly well-supported by quantitative results and qualitative case studies. AKRMap's superior visualization accuracy is validated through lower mapping errors (MAE, RMSE, MAPE) across multiple metrics. The ablation study confirms the necessity of kernel regression supervision and adaptive generalized kernels. However, the claim that AKRMap generalizes to all cross-modal metric landscapes lacks broad empirical support beyond text-to-image embeddings. Further validation on diverse tasks (e.g., speech-text, video-text embeddings) would strengthen this claim. Additionally, while AKRMap improves metric contour accuracy, its neighborhood preservation is weaker than traditional DR methods, which may impact interpretability.", "methods_and_evaluation_criteria": "The proposed AKRMap method aligns well with the goal of trustworthy cross-modal metric visualization. The use of kernel regression supervision and an adaptive generalized kernel is well-motivated. The evaluation criteria—mapping accuracy (MAE, RMSE, MAPE) on cross-modal metrics (CLIPScore, HPSv2, PickScore, Aesthetic Score)—are appropriate. The HPD dataset and real-world applications (T2I evaluation, model comparison, fine-tuning analysis) provide strong validation. However, broader benchmarks (e.g., video-text, speech-text datasets) would strengthen generalization claims. Neighborhood preservation is evaluated but could be further analyzed in trade-offs with metric accuracy.", "theoretical_claims": "The paper does not present formal theoretical proofs but introduces kernel regression-based projection and an adaptive generalized kernel as key theoretical contributions. The formulation of Nadaraya-Watson kernel regression and the adaptive kernel function is mathematically sound. However, the claim that AKRMap optimally balances metric mapping accuracy and neighborhood preservation lacks formal theoretical justification. Additionally, while the adaptive kernel function is empirically validated, a deeper theoretical analysis of its convergence properties or generalization bounds would strengthen its foundation. No significant errors were found, but more rigorous theoretical guarantees would enhance credibility.", "experimental_designs_or_analyses": "The experimental design is generally sound, with quantitative evaluations (MAE, RMSE, MAPE) and qualitative case studies across multiple datasets. The ablation study effectively isolates the contributions of kernel regression supervision and adaptive kernels. However, generalization beyond text-to-image tasks is not tested. The neighborhood preservation trade-off is acknowledged but not deeply analyzed. While interactive features (zoom, overlay) are highlighted, no user studies or response-time benchmarks are provided. A broader set of cross-modal datasets and human-in-the-loop evaluations would further validate AKRMap’s applicability. Overall, the analysis is strong but could be more comprehensive.", "relation_to_broader_scientific_literature": "The paper builds on prior dimensionality reduction (DR) methods like t-SNE, UMAP, and autoencoders, but innovates by integrating adaptive kernel regression for cross-modal metric visualization. It extends supervised t-SNE (Gisbrecht et al., 2015) and parametric UMAP (Sainburg et al., 2021) by incorporating a learnable adaptive kernel. The work is relevant to cross-modal embedding evaluation (CLIPScore, HPSv2) and aligns with efforts in trustworthy AI visualization. Compared to Neuro-Visualizer (Elhamod & Karpatne, 2024), AKRMap achieves better metric fidelity. However, broader validation on other multi-modal tasks (e.g., speech-text embeddings) would strengthen its impact.", "essential_references_not_discussed": "The paper lacks discussion on kernel-based DR methods like Kernel PCA (Schölkopf et al., 1998) and Density-Preserving Visualization (Narayan et al., 2021), which relate to its adaptive kernel regression approach. Additionally, multi-modal embedding alignment methods (e.g., ALIGN (Jia et al., 2021)) could provide more context for evaluating cross-modal metrics. While it cites supervised t-SNE (Gisbrecht et al., 2015), newer metric-aware DR techniques (e.g., Triplet UMAP (Sainburg et al., 2021)) are missing. Including these references would strengthen the theoretical foundation and contextualize AKRMap’s contributions.", "other_strengths_and_weaknesses": "Strengths:\n1. Introduces adaptive kernel regression for metric-aware cross-modal visualization, a novel extension to DR methods.\n2. Well-structured, with strong visual explanations and quantitative comparisons.\n\nWeaknesses:\n1. Only evaluated on text-to-image embeddings, limiting applicability to other cross-modal tasks.\n2. Lower than t-SNE/UMAP, with limited discussion on trade-offs.\n3. No empirical validation of interactive visualization usability (e.g., response time, interpretability).", "other_comments_or_suggestions": "Minor grammatical issues in some sections (e.g., \"contour estimation errors back into the projection process\" could be clearer).", "questions_for_authors": "1. Have you tested AKRMap on other cross-modal tasks (e.g., video-text, speech-text embeddings)? If not, how do you anticipate it would perform in those settings?\n2. Given Table 2 shows lower neighborhood trustworthiness than t-SNE/UMAP, have you considered a weighted trade-off mechanism between metric mapping and local structure preservation?\n3. Have you conducted any human evaluation to measure the usability of AKRMap’s interactive zoom/overlay features?\n4. Does the adaptive kernel regression method have convergence guarantees? Could certain assumptions ensure stability in the projection process?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HM7KuZkIM8", "reviewer": "Reviewer_63Bw", "summary": "In their paper \"AKRMap: adaptive kernel regression for trustworthy visualization of cross-modal embeddings\", the authors suggest what I would call *supervised parametric t-SNE* where supervision is by a continuous variable (as opposed to discrete classes). The algorithm combines the t-SNE loss function with a kernel regression loss function, to ensure that the resulting 2D embedding allows to predict the continuous variable of interest. The authors then apply this to visualize concatenated high-dimensional embeddings of (text prompt, generated image) pairs, colored by the prompt-image alignment metric (which provides the supervised signal). They authors argue that the resulting visualizations are helpful to get insight into the performance of image-generating models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the great review helping us improve the work. We respond to your questions below and provide the figures you ask for in the link: https://docs.google.com/document/d/e/2PACX-1vROjYSQAj8XMbrghXX4Ba_JrAxdFNagPOB84DtTK2PN5B_Ir-CAWew_c6zfhod7h3bU0VgCj_K8zuz8/pub\n\n## Q1 Why do contour maps not exclude empty embedding regions?\n\nA1: We have fixed the contour maps with a distance threshold to leave white space as you suggested and you can see the effect such as from Fig. 1 to Fig.5 in the link. We will make sure we fix all the figures in the updated version.\n\n## Q2 Why do you combine training and validation losses?\n\nA2: Thank you. We want to explain that unlike pure neural network methods, kernel regression is nonparametric, where the parameter like bandwidth is traditionally either precomputed or optimized via cross-validation. Thus, we are facing an interesting problem of how to bridge between parametric projection and nonparametric kernel regression, motivating us to set a validation set to learn kernel parameters jointly with the neural network. The validation loss improves the generalization of kernel regression for unseen positions. $w_1$=0 will shrink the kernel to training points with a sharp map and small peaks, while $w_2$=0 will make the map smoother but reduce local detail. Figure 4 in our link displays these two extreme settings and shows that quantitative results may deteriorate when either $w_1$ or $w_2$ is set to zero.\n\n## Q3 What is the t-SNE perplexity that you use in the KL loss in Eq 5?\n\nA3: Thank you for pointing out the confusion. We actually use a perplexity-free method proposed in this paper: Perplexity-free t-SNE and twice Student tt-SNE, which can compute $\\sigma$ using multi-scale Gaussian similarity without manual perplexity. This method can also compute the affinities on batches. We will clarify this in the updated version with reference to recent methods including your suggested paper.\n\n## Q4 It would be nice to see AKRMap visualizations for different values of lambda.\n\nA4: Thank you for the suggestion. We have added experiments varying lambda between 0.05 and 0.5 to show the trade-off, with both quantitative results and visualizations. The visualization is shown in Fig. 5 in our link. The quantitative results are presented as follows, showing mae against neighbor trustworthiness (n=20). We can see clear trade-off effect ($\\lambda$ reduces mae but also decreases neighborhood trust) except in CLIPScore where both low and high $\\lambda$ are problematic.\n### Table 1. trade-off\n|Method| Metric|λ=0.05|λ=0.125|λ=0.25|λ=0.5|\n|-|-|-|-|-|-|\n|**HPSv2**|MAE|1.36|0.81|0.56|0.45|\n|| Trust  |0.86|0.83|0.81|0.78|\n|**PickScore**| MAE    | 1.10   | 0.77   | 0.55   | 0.46   |\n||Trust|0.85|0.81|0.80|0.78|\n|**Aesthetic**|MAE|0.46|0.43|0.40|0.39|\n||Trust|0.87|0.87|0.86|0.85|\n|**CLIPScore**|MAE|2.08|1.87|3.44|3.48|\n||Trust|0.78|0.77|0.69|0.71|\n\n## Q5 Ablations: it would be nice to see visualizations like in Figure 3 / Figure 8 for AKRMap w/0 KR and w/o GR. Also, what does \"without GR\" mean exactly? What did alpha and beta converge to in your experiments?\n\nA5: Yes, w/o GK means fixing those parameters to 1. We have provided a figure comparing the scatterplots of the ablation results in our link (Fig. 6). In addition, the learned (alpha, beta) values are - HPSv2: (68.57, 1.61), PickScore: (59.13, 1.35), CLIPScore: (104.70, 3.18), Aesthetic Score: (74.95, 1.11).\n\n## Q6 In Figure 6 and 7 which show score differences, it would be good to use a diverging colorscale centered at 0. Figure 6 should have a colorbar.\n\nA6: Thank you. We have added the colorbar in a revised Figure 6 here in our link (Fig. 2). However, we originally only intended to use color to indicate high and low values instead of sign and we want to explain why we did not center colorscale at 0: the differences between models can be uneven on positive and negative axes. For example, the original Fig. 6 shows an extreme case comparing a 2022 SD model with a quite recent VAR model, where the recent model almost always beats the old one (with sparse outliers). We provide here another two examples of more comparable models: 1) SD3 and SDXL (Fig. 7 in our link), and 2) a new image-to-3D model comparison (Fig.8 in our link). In these examples we can see both positive and negative regions.\n\n## Q7 Figure 7a: two example promts seem very different and have nothing in common. Why do they appear so close in the embedding?\n\nA7: Thank you. This is possibly because the embedding we project is a concatenation of text and image embeddings. Even though the prompts are different, the images do seem to have similar background features. \n\n## Q8 Section 2: parametric t-SNE should cite https://openreview.net/forum?id=B8a1FcY0vi\nA8: Thank you. We will add the reference in the updated version.\n\n## Q9 Table 1: everything could be easily rounded to 1 decimal digit\nA9: Thank you. We will reduce the digit in the updated version.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There are none.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The paper is related to recent literature on multimodal embeddings and text/image similarity scores.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths: The algorithm is novel and seems to perform well. The applications are interesting and do provide curious insights.\n\nWeaknesses: The algorithm is quite simple but appears under-studied. In particular, I am missing more systematic hyper-parameter studies and comparisons. All contour maps would be much less confusing if they excluded empty regions.\n\nMethodologically this is a borderline paper (and based only on methodology I would rather tend to rejection), but I do like the applications so am tending towards to acceptance.\n\n\nMAJOR COMMENTS\n\n* Why do contour maps not exclude empty embdding regions? E.g. in Figure 1, the panel (a) contains some white space, esp. in the corners but also in between t-SNE islands. I would expect the contour map in panel (c) to contain white space in the same places. Instead, the contour map is the full square. This is extremely confusing. Your contour maps are based on a square grid, so can't you make almost-empty squares white? This applies to all contour maps shown in the paper.\n\n* Section 4.1.1: why do you combine training and validation losses? This sounds like a weird approach. When training a model, the loss is usually computed on the training set. Why do you need the validation set here at all? What would you get if you set w_1 = 0 or alternatively w_2 = 0?\n\n* What is the t-SNE perplexity that you use in the KL loss in Eq 5? This is never specificed. Furthermore, you use batch size 1000, but your dataset size is 430,000. The t-SNE's KL loss includes a global normalization, so it's unclear how you can optimize it with mini-batches. Do you compute the p_ij and q_ij t-SNE's affinities (which enter the KL loss) based on each batch separately? If so, and if you use e.g. perplexity 30, then your lambda=0 setting should be similar to t-SNE with effective perplexity 30 * 430,000/1000 = 12,900. Is that right? This needs to be discussed. Note that there is a way to optimize proper t-SNE (with p_ij computed globally for the entire dataset) with minibatches, see e.g. https://openreview.net/forum?id=B8a1FcY0vi, but I assume that's not what you do.\n\n* It would be nice to see AKRMap visualizations for different values of lambda, interpolating from KL to KR, ideally with some metrics like MSE and trustworthiness computed for each lambda and showing a trade-off.\n\n* Ablations: it would be nice to see visualizations like in Figure 3 / Figure 8 for AKRMap w/0 KR and w/o GR. Also, what does \"without GR\" mean exactly? That you set alpha=beta=1 instead of optimizing these two parameters? By the way, what did alpha and beta converge to in your experiments?\n\n* In Figure 6 and 7 which show score differences, it would be good to use a diverging colorscale centered at 0. Figure 6 should have a colorbar.\n\n* Figure 7a: two example promts seem very different and have nothing in common. Why do they appear so close in the embedding?", "other_comments_or_suggestions": "MINOR COMMENTS\n\n* Section 2: parametric t-SNE should cite https://openreview.net/forum?id=B8a1FcY0vi\n\n* Table 1: everything could be easily rounded to 1 decimal digit", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a new approach to visualizing cross-modal data. While the method is theoretically underdeveloped, it outperforms other methods and the paper includes good experimental studies and interesting applications. The reviewers all lean toward acceptance. Thus I recommend that the paper be accepted.
Decision: Accept (poster)