It's All Relative: Relative Uncertainty in Latent Spaces using Relative Representations

Authors: fabian martin mager
valentino  maiorca
lars kai hansen
marco  fumero
clementine  domine
zorah  lähner
donato  crisostomi
luca  moschella
kimberly  stachenfeld
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-23 20:21:28

Abstract: Many explainable artificial intelligence (XAI) methods investigate the embedding space of a given neural network. Uncertainty quantification in these spaces can lead to a better understanding of the mechanisms learned by the given network. When concerned with the uncertainty of functions in latent spaces we can invoke ensembles of trained models. Such ensembles can be confounded by reparameterization, i.e., lack of identifiability. We consider two mechanisms for reducing reparametrization "noise", one based on relative representations and one based on interpolation in weight space. By sampling embedding spaces along a curve connecting two fully converged networks without an increase in loss, we show that the latent uncertainty becomes overestimated when comparing embedding spaces without considering the reparametrization issue. By changing the absolute embedding space to a space of relative proximity, we show that the spaces become aligned, and the measured uncertainty decreases. Using this method, we show that the most non-trivial changes to the latent space occur around the midpoint of the curve connecting the independently trained networks.

Main paper: 
 introduction: 
 A neural network (NN) trained to perform a certain downstream task, e.g., regression or classification, learns a mapping f : x i → y i given a set of observations D = {(x i , y i )} N i , where x i = [x 1 , x 2 , ..., x D ] and N is the number of available observations and D is the latent space dimension. The learned function f is hierarchical, i.e. f (x) = h L (h L-1 (...h 1 (x))) where the intermediate representation of the observation x i after layer l is denoted as z i . We note the mapping of the observation x i to its latent representation z i as h l : x i → z i . The latent space is assumed to be semantically meaningful for relevant downstream tasks. Therefore, many XAI methods investigate the properties of the latent space of a NN, for example, its geometry [9] or the semantic structure [5] . XAI has many dimensions [11] , one of which is referred to as local vs. global. Local explainability concerns predictions of individual samples, whereas global explainability considers the network as a whole. Most local methods give attributions to the input features [x 1 , x 2 , ..., x D ], which are important to the output ŷ. Recently, Wickstrøm et al. [10] proposed a method that maps attributions to input features based on the importance of its latent vector z. In contrast, global explainability methods such as Concept Activation Vectors (TCAV) [5] aim to explain the entire latent space based on semantics. One persistent challenge in assessing geometry and semantic structure of the latent space is the reparametrization issue, driven by the fact that there is no "set of optimal parameters and that we can always parametrize the manifold in a different, but equally good, way" ( [2] ). In other words, while the end-to-end function f remains the same, different parametrizations lead to different but equally performing embedding functions h. Moschella et al. [7] introduced Relative Representations, which define an alternative representation of the latent space. They empirically show that the latent spaces of two identical models trained on the same data but different initializations are identical up to angle-preserving transformations. Their proposed method uses a set of latent vectors A = {z i } A i , called anchors, and redefines the position of each latent vector to be relative to the set of anchors, according to a similarity function sim : R D × R D → R. Depending on the choice of the similarity function, the latent space becomes invariant to certain transformations. The reparametrization issue is also related to the loss landscape in NNs. Several works have exploited the reparametrization issue to efficiently train ensembles of models, which outperform their single-model counterparts [4] . Garipov et al. [1] investigated the geometrical properties of loss landscapes and showed that two identical NNs trained on different seeds can be connected by a simple curve in weight space, such that the loss under the curve is constant. This curve ϕ parameterized by θ connects two points ŵ1 and ŵ2 in parameter space and is found by minimizing the lossl(θ) = 1 0 L(ϕ(t)), dt = E t∼U (0,1) [L(ϕ θ (t))]],where L is the loss function used to find ŵ1 and ŵ2 . We refer to section B for details on the curve-finding procedure. Once θ is fitted, one can sample from ϕ θ (t)) for 0 ≤ t ≤ 1 and build an ensemble of models from living on the curve ϕ θ (t). Ensembling methods reduce both the bias and variance of ŷ. The uncertainty over K samples of the end-to-end function f in the prediction of ŷi is given byEQUATIONwhereȳi = E k∈K [f k (x i )].Following the above reasoning, the embedding function's uncertainty h could be obtained by replacing f k with h k and y i with z i . Due to reparametrization however, the latent uncertainty σ 2 (z i ) will be overestimated (see Appx. A). We argue that there are two factors contributing to the overall observed uncertaintyEQUATIONwhere σ 2 R refers to the uncertainty caused by reparametrization and σ 2 M to the uncertainty of the model. In equation 2 we assume independence of the model and reparametrization uncertainty. Such independence could arise as a consequence of initialization being independent of the data. In order to isolate the model uncertainty σ 2 M (z i ), the reparametrization uncertainty σ 2 R (z i ) → 0. In Appx. A we show that, for rotation and scaling transformations, the Relative Representation framework, using a cosine similarity function, can eliminate σ 2 R (z i ). In this work, we• Define an alignment score ρ, which estimates the signal-to-noise ratio for a given ensemble of latent observations. The alignment score ρ is a measure of separability between N latent points given K samples each, similar to Fisher's Linear Discriminant objective for N classes. • Show empirically that when sampling models along the curve ϕ, transforming the embeddings into a space of relative proximity increases alignment between latent observations. • Show that the latent observations for the curve ϕ(t) have high alignment around the endpoints and little alignment at the bending point, suggesting limited information gain when sampling around the endpoints and high information gain when sampling across the midpoint. 
methods: 
 The intuition behind our alignment metric is the following: Given K samples of a latent vector z i , where each sample is an embedding of a given observation x i obtained from a unique function h k , the alignment score ρ i should be high if all samples form a compact region in the latent space and are well separated from all other points (see Figure 1 ). The alignment score ρ of the entire embedding space can be estimated as the average of individual scores. This formulation is similar to Fisher's Linear Discriminant (FLD) objective, which measures the separability of two or more classes of objects. The FLD objective is given by the ratio of between-class vs. within-class covariance. Given K realizations of latent vector z i , the objective can be applied on the scale of a single observation, i.e. the ratio of between-observation vs. within-observation covariance. Due to the high dimensionality of z and the few number of available samples, it is not feasible to calculate the full covariance matrix. We therefore use the variance estimator and express the ratio as a signal-to-noise ratio, such that the alignment metric ρ is bounded 0 ≤ ρ ≤ 1. We define the within-observation variance σ 2 W and between-observation variance σ 2 B of k realizations of the latent space and their alignment ρ asEQUATIONEQUATIONρ = σ 2 B σ 2 B + σ 2 W (5)We conduct the following experiment independently using a VGG-16 [8] and a Preactivation-ResNet [3] on CIFAR-100 [6] . First, fully train on three different seeds until convergence, resulting in three unique models ŵ1 , ŵ2 , and ŵ3 . For each pairwise combination of seeds, we fit a Bezier curve with fixed endpoints and a single bend following the procedure described in [1] . Once a curve ϕ ij (t) between two modes ŵi , ŵj is found, we sample K models at steps t = k * ∆t, where k ∈ [0, ..., K -1] and ∆t = 1 K . For all experiments, we set K = 21. We proceed by evaluating the performance at each step t, as well as the ensemble performance over all models up to step t using a hold-out test set D test = (x i , y i ) N i . We investigate the alignment of the embedding spaces along the curve, i.e. the last layer before the classification layer. We define the absolute embedding space Z k as the ensemble of all latent vectors {z i } in D test and the space of relative proximity Q k as the ensemble of all transformed vectors {q i } given a set of anchors A and a similarity function sim : R D × R D → R. The number of anchors is chosen to match the number of dimensions D of the absolute embeddings. For the VGG16, D = 512, and for the ResNet110, D = 256. Anchors are sampled randomly from D test , following the procedure of [7] . Table 1 provides an overview of the proposed similarity functions. For the cosine and basis transformations, we center the embedding space Z k before calculating similarities using an estimate of the mean based on D train . For the Euclidean transformation, the space is additionally scaled to unit variance. Finally, using the proposed metric, we measure the cumulative alignment for increasing k of both the absolute and relative embedding spaces. We compare the observed alignment with the alignment of eleven independently trained networks. Code is available here 1 
results: 
 Figure 2 : Top: Curve fitting results (solid) for VGG (red) and PreResNet (blue), pairwise connecting three modes with individual (left) and cumulative ensemble error rates (right). The individual error rate is the classification error rate for CIFAR100 for a given set of parameters t on the curve. The cumulative error rate is determined by the arg max of the sum of probabilities across all sampled parameters on the curve in [0, t]. Bottom: Alignment of embedding spaces along the curve for the absolute (green) and relative (purple) space. Dashed lines represent reference values, given as the average error and alignment of eleven independently trained networks.In Figure 2 we show the individual and cumulative ensemble error rates for each model type and curve. For all models and curves, the test accuracy increases when approaching the midpoint. The cumulative ensemble error decreases with increasing ensemble size. Notably, there is little variation across the pairwise curves. The embeddings of relative representations show higher alignment than the embeddings of absolute representations. This shows that the embedding spaces are confounded by reparametrization. However, the results vary across projection methods. For the VGG, the cosine similarity performs best, whereas for the ResNet it is Euclidean distance. Alignment measured in absolute space decreases almost linearly. The alignment measured in relative space, however, seems to converge. Compared to the baseline experiment, where alignment is measured across eleven independently trained networks, the measured alignments in relative space are closer than those measured in absolute space. Furthermore, the figure shows that the highest negative slope occurs around t = 0.5, whereas the slope has limited variation around the endpoints. This finding suggests that ensembles used for uncertainty quantification should be samples far from the endpoints of a connecting curve. 
discussion: 
 In this work, we investigate the alignment of latent observations when sampling along a curve connecting two modes. Our results show that transforming the spaces into a space of relative proximity reduces uncertainty significantly. This suggests that the uncertainty of ensembles is indeed confounded by reparametrization. The measured uncertainty varies across different relative projections within each modality. This is a limitation, as no general recommendation for a projection method can be made. Each projection will make the latent space invariant to some transformation, and the optimal choice of projection is model-dependent. Certain projections might bias uncertainty estimates. For example, the unit-norm scaling of the cosine similarity measure will increase uncertainties close to the zero vector. In this work, we used a random set of anchors, which eases the workflow of sampling from the curve. However the choice of anchors, e.g. archetypes or centroids, will likely influence the projection quality. The influence of projection function and anchor choice should be further investigated.Table 1 shows transformations of a latent obersvation z n into a latent observation of relative proximity q n based on a set of anchors A = {z i } A i .Rel. Cosine Basis Transformation, such that the new basis isq n = znA T ||zn|| ||Ai||the cosine distance of each anchor to each point Rel. Euclidean Basis transformation, such that the new basis is q n = [q 1 , q 2 , ..., q D ] ,the euclidean distance of each anchor to each point where q i = ||z n -a i || 2for i ∈ [1, ..., D]Rel. Basis Change of basis based on anchor pointsq n = znA T ||Ai|| 2Table 1: Summary of transformations for a latent vector z i ∈ R D and a matrix of anchors A ∈ R D×D , where each row vector a i in A corresponds to an element in the set of anchors A = {z i } A i .Proceedings of the II edition of the Workshop on Unifying Representations in Neural Models (UniReps 2024).https://github.com/fmager/it-s-all-relative 
 
 Appendix:  


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Empirical analysis of uncertainty in latent spaces with the relative representations.
main_review: main_review: Strengths:

Originality: A new alignment score is defined based on previous work, which was cited quite well. 
Quality: It is a nice paper investigating the uncertainty in the latent space, and the alignment between the latent spaces of same models trained with different initial parametrization. It is a work in progress, and the claims are well-supported. Limitations of the work are addressed too.
Clarity: The paper is mostly well written and organized, with small typos. 
Significance: The authors indeed introduce a new approach to measure the alignment between latent spaces and demonstrates the impact of relative representation. It shows preliminary but positive results for further exploration. 

Weaknesses / Suggestions:

*  It would be interesting to explore the impact of using different numbers of models (K), on line 89. 
* Impact of the number of anchors could be investigated for the future work. 
* More comprehensive details on the eleven models used for benchmarking could improve transparency and contribute to reproducibility.
* The rationale behind Eqs. (3–5) is not fully explained. While the reasoning for Eq. (1) and (2) is clear, Eqs. (3–5) need further clarification.
* GitHub link is missing, but probably due to anonymity. Needs to be updated for the final submission.


Minor Edits:
* On line 49, loss is defined using L instead of $\mathcal{L}$
* In Eq. (1), $\mathbf{x}_n$ needs to be replaced by $\mathbf{x}_i$. 
* It appears that "N" represents the number of embeddings, though this is not explicitly stated in the paper.
title: title: Revision of UniReps '24 — #39
main_review: main_review: **Paper summary**

In this paper, the authors explore uncertainty in neural networks' latent spaces by focusing on reparameterization issues that distort uncertainty estimation, proposing to address these problems using relative representations, which compare latent space features relative to fixed anchor points. Specifically, they define an alignment score $\rho{}$, which estimates the signal-to-noise ratio for a given ensemble of latent observations. The alignment score $\rho{}$ is a measure of separability between N latent points given $K$ samples each, similar to Fisher’s Linear Discriminant objective for $N$ classes. Then, the authors show empirically that when sampling models along the curve $\phi{}$, transforming the embeddings into a space of relative proximity increases alignment between latent observations. Finally, they also show that the latent observations $\phi{}(t)$ for the curve have high alignment around the endpoints and little alignment at the bending point, suggesting limited information gain when sampling around the endpoints and high information gain when sampling across the midpoint.

**Paper strengths**

The intuition behind our alignment metric is the following is interesting and addresses a significant challenge in understanding latent spaces.
Furthermore, the authors clearly define the problem and provide detailed mathematical formulations to support their proposal (as far as possible given the 4 pages available), and by demonstrating that most latent space uncertainty arises in the middle of the interpolation curve between networks, the paper provides practical insights for building neural network ensembles.

**Paper weaknesses**

The results show that transforming the spaces into a space of relative proximity reduces uncertainty significantly. This finding suggests that the uncertainty of ensembles is indeed confounded by reparametrization. However, the measured uncertainty varies across different relative projections within each modality, and the anchor choice will likely influence the projection quality. As a result, further in-depth research is necessary to evaluate the proposal thoroughly.
title: title: Interesting work, but clarity needs improvement
main_review: main_review: This paper proposes using relative representations for an ensemble of embedding spaces that are interpolated along a curve connecting two independently trained NNs with different initializations. Using relative representations, as opposed to absolute representations, enables the model uncertainty of embeddings/representations to be captured more reliably by reducing the reparametrization uncertainty. 

**Pros:**
* This work provides interesting insights for how to sample ensembles along a connecting curve: sample far away from the endpoints. 
* The experimental result shows the benefits of using relative representations over absolute representations in reducing reparametrization uncertainty (Figure 1 Right).
* A good discussion of the limitations is provided in the discussion section.

**Cons:**
1. Lines `60-61`: The authors mention that "the relative representation framework can eliminate $\sigma_R^2(z_i)$". However, I couldn't find a discussion of this in the main paper and Appendix A. Why and how does the relative representation framework eliminate $\sigma_R^2(z_i)$, i.e., the uncertainty caused by reparametrization?
2. The clarity and presentation requires improvement. For example, the motivation of the paper could be discussed more clearly in the introduction. Clearly discussing important problems such as the item (1) mentioned above would also help readers gain a better understanding. 

**Questions:**
* Figure 2: How are the error rates measured? Are relative or absolute representations used for computing the error rates for the models?
title: title: Mathematically sound but lacking in clarity, while the model choices and the weak significance of the results limit its impact and generalizability
main_review: main_review: Most approaches are mathematically reasonable and valid, while lacking clarification details probably due to the page limit. However, further illustration could always fit into the appendix, such as the selection of anchors, the exact definition of relative embedding space, the transformation matrix and how it influences whether the reparameterization introduces additional variance, why scaling to unit variance is appropriate for Euclidean distances in the context of embedding space. Here are more details regarding this issue:

- The Appx. A contains the explanation of the reparametrization and the linear transformation matrix Q applied to the weights, latent vectors, and even latent space. mathematically reasonable, though some middle process should be illustrated, such as the definition of Q should be addressed when it first appears.
- However, Appx. A does not provide evidence to the statement in your last part of introduction “Due to the reparametrization however, the latent uncertainty σ2(zi) will be overestimate”.  which means that your transformation Q change the latent spread of the hidden representation, while orthogonal rotation and uniform scalar won’t introduce additional variance in most cases (except close to the zero vector). Under the same condition, the other statement hold true: “In Appx. A we show that, for certain transformations, the relative representation framework can eliminate σR2 (zi)”.
- If the author mean that the transform matrix Q can be anything, which probably would introduce additional uncertainty during reparameterization, while they could make Q=αU, where U^T U=I which won't lead to overestimation of uncertainty in most cases, then they need to be specifically illustrate these logics. If the author meant that Q=αU always holds, then the I question whether transformation Q would lead to more uncertainty in reparameterization.
- the equation 2:  σO2=σR2+σM2  assume uncertainty (variance) is additive, which might hold when 2 components are independent. But it might not always hold in practice. If reparametrization introduces transformations that interact with the model's uncertainty, the relationship could be more complex than direct addition. In many cases, uncertainties combine in a more nuanced way, such as through variance propagation (e.g., σO2=σR2+σM2+2Cov(σR,σM)).


The results and conclusions are relatively weak, so does the significance of the study. it’s a validation of the method for training ensemble of models developed in the previous literature. It’s great that transform of the hidden layers of the ensemble models increased their alignment, but it is not surprising as long as good transformation matrix are specified. The conclusion about similarity metric selection is model-specific, while only 2 relatively older models are used: VGG-16 (developed in 2014) and Preactivation-ResNet-110 (developed in 2016). More recent architectures such as Vision Transformers (ViTs) or newer variations of efficient convolutional networks could be used (e.g., EfficientNet, ConvNeXt), whereas the VGG and RestNet could be baselines. Limitations of the architecture of the models used here could be stressed or discussed but not. There is also no explanation of the continued relevance of older architectures to the current question. Additionally, I question if the current paper offer insights that can be generalized to more complex tasks, such as multi-task learning, domain adaptation, or few-shot learning. The uncertainty varies across different relative projections for different modality, and even model-specific.

Comments: title: Review Reply
Review Reply
Review Reply
Review Reply
comment: Dear reviewer,
Thank you for taking the time to read our paper. We would like to clarify our argument regarding $\sigma_R^2(\mathbf{z})$. In Appendix A we show that the Relative Representation framework with a cosine distance metric can eliminate rotation and scaling transformations. Thus, if scaling and rotations are part of transformations caused by reparametrization on the latent space, the framework can remove those transformations and thereby eliminate their impact on uncertainty estimation. We have added a few lines to clarify this argument.
Dear reviewer,
Thank you very much for taking the time to read our work. We appreciate your thorough and constructive comments.
- We agree that a more recent architecture would strengthen our findings, and we plan to add additional models in future work.
- More details have been added in Appendix A.
- Wrt. eq. 2, we agree that there could be an interaction of model and reparametrization uncertainty. However, we believe independence is a valid assumption as the initialization is independent of the data. 
- We agree that the work needs further research to understand the transformation posed by reparametrization and the inductive biases introduced by relative transformations with certain distance metrics.
Dear reviewer,
Thank you very much for taking the time to read our work. We agree that the work needs further research to understand the transformation posed by reparametrization. As we also mention in the discussion of our work, a limitation of the Relative Representation framework used in this context is the inductive bias introduced by a given distance metric, which will lead to different uncertainty estimates.
Dear reviewer,
Thank you very much for taking the time to read our work. We appreciate your suggestions and will incorporate them as we continue our work, e.g. investigating the number of anchors.
- All minor edits were corrected
- The link to the codebase was added
- We added additional explanations to eq. 3-5
- As for the number of sampled models K, we find that the change of alignment along the curve is very smooth, so adding more samples does not change the results.