Investigating the Effects of Fairness Interventions Using Pointwise Representational Similarity
Authors: Camila Kolling
Till Speicher
Vedant Nanda
Mariya Toneva
Krishna P. Gummadi
Venue: Accepted by TMLR ()

Abstract: Machine learning (ML) algorithms can often exhibit discriminatory behavior, negatively affecting certain populations across protected groups. To address this, numerous debiasing methods, and consequently evaluation measures, have been proposed. Current evaluation measures for debiasing methods suffer from two main limitations: (1) they primarily provide a global estimate of unfairness, failing to provide a more fine-grained analysis, and (2) they predominantly analyze the model output on a specific task, failing to generalize the findings to other tasks. In this work, we introduce Pointwise Normalized Kernel Alignment (PNKA), a pointwise representational similarity measure that addresses these limitations by measuring how debiasing measures affect the intermediate representations of individuals. On tabular data, the use of PNKA reveals previously unknown insights: while group fairness predominantly influences a small subset of the population, maintaining high representational similarity for the majority, individual fairness constraints uniformly impact representations across the entire population, altering nearly every data point. We show that by evaluating representations using PNKA, we can reliably predict the behavior of ML models trained on these representations. Moreover, applying PNKA to language embeddings shows that existing debiasing methods may not perform as intended, failing to remove biases from stereotypical words and sentences. Our findings suggest that current evaluation measures for debiasing methods are insufficient, highlighting the need for a deeper understanding of the effects of debiasing methods, and show how pointwise representational similarity metrics can help with fairness audits.

Published in Transactions on Machine Learning Research (05/2025)
Investigating the Effects of Fairness
Interventions Using
Pointwise Representational Similarity
Camila Kolling
MPI-SWS
Till Speicher
tapeicher&mppi-sus.oom
MPI-SWS
Vedant Nanda
vaanda@mpissus.or
MPI-SWS
Mariya Toneva MPI-SWS
mtoneea@mpieeus.og
Krishna P. Gummadi
gummadi@mpi-sus
MPI-SWS
Reviewed on OpenReview: https: // openreview. net/ forum? id= CkVl t20gdb
Abstract
Machine learning (ML) algorithuss can often exhibit discriminatory behavior, negatively fecting certain populations across protected groups. To address this, numerous debiasing methods, and consequently evaluation measures, have been proposed. Current evaluation measures for debiasing methods suffer from two main limitations: (1) they primarily provide global estimate of unfairness, failing to provide more fine-grained analysis, and (2) they predominantly analyze the model output on a specific task, failing to generalize the findings lo other tasks. In this work, we introduce Pointwise Normalized Kernel Aligmment (PNKA), a pointwise representational similarity measure that addresses these limitations by mea- suring how debiasing measures affect the intermediate representations of individuals. On tabular data, the use of PNKA reveals previously unknown insights: while group fairness predominantly influences small subset of the population, maintaining high representational similarity for the majority, individual fairness constraints uniformly impact representations across the entire population, altering nearly every data point. We show that by evaluating representations using PNKA, we can reliably predict the behavior of ML models trained on these representations. Moreover, applying PNKA to language embeddings shows that existing debiasing methods may not perform as intended, failing remove biases from stereotypical words and sentences. Our findings suggest that current evaluation measures for debiasing methods are insufficient, highlighting the need for deeper understanding of the effects of debiasing methods, and show how pointwise representational similarity metrics can help with fairness audits.
1 Introduction
Machine lcarning algorithms are now deeply integrated into several aspects of our daily lives. These algo- rithins not only recommend movies and products or suggest, potential dating partners (Sun et al. 2019 Yu et al. 2023 Wu et al.) 2022), but they are also increasingly employed critical decision decision-making processes, such as approving loans and making hiring and health choices (Alctras et al. 2016 Liang et al.] 2014 Sheikh et al. 2020). Despite their impressive performance, ML models face significant reliability challenges, par-
ticularly in decision-making tasks (Geirhos et. al.] 2018 Hendrycks & Dietterich 2019: Taori al. 2020

---

Published in Transactions on Machine Learning Research (05/2025)
Szegody et al. 2013 Papernot et al. 2016 A major concern is their discriminatory behavior against certain protected groups (Angwin et al.) 2016; O neil 2017 Buolamwini & Gebru 2018). manifesting as biases that adversely affect individuals based on race, gender, age, or other protected characteristics. These biases can result in unfair outcomes that negatively affect individuals based on their characteristics. To mitigate such discriminatory behavior, researchers have proposed a variely of approaches that intervene al various stages
of the ML pipeline, including data pre-processing (Feldman et al. 2015: Quadrianto et al. 2019 Ryu et al. learning fair intermediate representations (Dwork et al.
2018 Wang & Deng 2020),
2012
Zemel et al. 2013
Edwards & Storkey,
2016
Madras et al. 2018: Beutel et al.
2017
Wang et al. 2019
in in-processing by
adding constraints to the objective function Zafar et al. 201 7b
a 2019
Donini et al.
2018
Goel et al. 2018
Padala & Gujar 2020 Agarwal et al. 2018 Martinez et al.
2020
Diana et al.
2020
Lahoti et al.
2020
and post-processing by changing model outpuls (Hardt el al. 2016
Wang et al.
2020
Savani el al. 2020
Following the introduction of these debiasing methods, the research community has developed several evalu- ation measures to assess their effectiveness. These measures aim to quantify the fairness of ML models and include metrics such as equalized odds, equality of opportunity, and demographic parity Hardl et al. (2016);
Dwork et al. (2012); h: Kusner et al. (2017). However, current evaluation approaches face two main limitations. First, they primarily focus on providing global estimate of unfairness by analyzing the disparitics between
groups categorized by single protected attribute, such as race or gender (Hardt et al. 2016
Dwork et al.
2012
Kusner et al.] 2017]. However, as recent work reveals (Buolamwini & Gebru 2018 Kearns et al.
Kim et al. 2020), fairness is multifaceted and can manifest in several different ways, making a more fine- iroograined analysis necessary. Second, fairness evaluation approaches predominantly analyze model behav- ior specific target task (Hardt et al., 2016 Dwork et al., 2012 Kusner et al. 2017). In most cases they do not consider the representations that models to reason about the data. Conceptually, most algorithms first compute an intermediate representation = (xx) of an input x, before deriving outcome y = f()). Such intermediate representations appear, for example, as a result of feature engineering or at the intermodiate layers of deep neural networks, and are becoming increasingly important with the rise of large, pretrained foundation models (Bommasani et al. 2021; Radford et al. 2021 Brown et al. 2020), where the representations of one model serve as the basis for many different ream tasks. The insights from prior methods that only focus on tpsk-specific outcomes y cannot easily be generalized to other tasks that use the same represcntations
2018
In this work, rather than only evaluating task-specific outcomes on a global level, we address the limitations of previous evaluation measures by analyzing how debiasing methods modify the intermediate representations of individual data points. More specifically, we analyze how much the representation of a data point changes from "baseline" (non-debiased) to "fair" (debiased) models. To do so, we propose a new measure that is inspired by the existing rich body of rescarch on representational similarity measures in the machine learning community (Laakso & Cottrell 2000: Li al.) 2015 Wang et al.) 2018 Raghu et al.) 2017; Morcos et al., 2018 Kornblith et al. 2019). Representation similarity measures, such as the widely-used CKA (Kornblith] ct al. 2019), analyze how two different models represent a given dataset. They capture similarity a real number that reflects the degree of similarity between the representations, with 1 indicating identical representations. These measures thus allow us to analyze how model interventions impact all fownsstrean tasks that use these representations.
An important limitation of previously introduced representation similarity measures is that they only provide an aggregate similarity score across the entire dataset. However, such aggregate scores are not suitable for achieving our goal of analyzing at a local level how debiasing methods are affecting individuals. To enable the fino-grained study of representation similarity, we modify the widely-used CKA measure (Konbblihh ct al. 2019) into pointwise representational similarity measure, which we call Pointwise Normalized Kernel Alignment (PNKA). PNKA provides a similarity score for each individual data point, allowing us to study how model interventions affect individuals at the representation level.
Our key contributions are summarized as follows:
Prior work has included learning fair representation in both pre- and in- pracessing, e.g. see Zafar al. 20199 : here choose to list separately since it the main focus of our paper.
2

---

Published in Transactions on Machine Learning Research (05/2025)
We introduce PNKA, pointwise representation similarity measure. PNKA allows to analyze the effect of model interventions, such as debiasing, at a local level, by measuring how interventions change the representations of individuals. PNKA is broadly applicable to all debiasing approaches that modify the data or the models. Morcover, it offers a more generalized solution by climinating the need for manually developing domain-specific evaluation methods for each individual use case.
We demonstrate PNKA's utility in auditing representations of debiasing approaches. On the COM- PAS and Adult datasets, PNKA reveals previously unknown insights: while group fairness predom- inantly influences a small subset of the population, maintaining high representational similarity for the majority, individual fairness constraints uniformly impact representations across the entire pop- ulation, altering nearly every data point. Our observations on representation similarity allow us to predict the effect that training MT. models on debiased representations will have, and we demonstrate that the actual outcomes do indeed match the predictions.
By applying PNKA to contextual and non-contextual language embeddings, we show that debiasing methods in these domains may not perform as intended. Specifically, PNKA shows that debiasing approaches do not consistently remove gender properties from stereotypical words and sentences as anticipated. Our results also suggest that the fairness evaluation measures currently employed for evaluating (non-)contextual debiased embeddings are both limited and insufficient for comprehen- sively assessing these debiasing methods.
2 Pointwise Representational Similarity Measure for Assessing Debiasing Approaches
When an algorithm processes data x for an individual, it lypically operates in two stages: 1) computing an (intermediate) representation = = g(z) and 2) computing the outcome y = f2). To ensure that algorithms produce fair outcomes, a popular approach is to replace a potentially biased feature extractor g with an unbiased version g', such that the modified representation 2' = =x'(x) does not contain information about sensitive feature (Zemel et al. 2013) Lahoti et al., 2019). We use the notation here to refer to the reprosentation of single individual x, where is d-dimensional vector, i..., E Rd. We use uppercase Z to refer to the representations of set of N individuals, i.e., Z E Nxd
When debiasing feature extractor it is important to be able to assess the effects of the change on perfor- mance and fairness. Most prior work focuses on studying how changes to affect the outcomes y (Hardt) et al. 2016 Dwork el al.] 2012 Kusner el al. 2017), but the insights from such assessments only apply to specific downstream tasks. To gain a better understanding of how changes to the representations Z will im- pact different downstream tasks, we have to study Z directly. Such an asscssment can be performed using representation similarity measure s(Z, Z") that measures how similar Z' is to Z. Representational similarity measures typically provide a single overall score that estimates how similar two different representations of an entire set of input points are. However, overall representation similarity scores do not allow us to assess how much the representation for individual points change, and can thus overlook potential adverse effects that changes to the representation can have on small groups or individuals.
To address these issues, we adopt pointwise measure that assigns an individual representational similarity score to each data point. With this measure, we can effectively determine whether data model inter- ventions, such as debiasing, impact all instances uniformly (i.e., whether all individuals are affected by the debiasing method) or disproportionately impact certain data points (i.c., some individuals are more affected than others). By comparing representations of a baseline (non-debiased) model g with its debiased version g', we can identify and characterize which individuals are most affected and whether fairness interventions effectively target those they are aimed at. For instance, by focussing on the individuals with the lowest similarity scores, we can determine whose representation change the most from the baseline to the debiased version. Next, we describe our pointwise representation similarity measure.
3

---

Published in Transactions on Machine Learning Research (05/2025)
2.1 Intuition for Pointwise Similarity Across Representations
An initially appealing way to measure representational similarity of the i-th point in representations Z and Z' is to directly apply (dis)similarity metric, such as the Euclidean distance or cosine similarity, to its two representations Z; and Z c.g.; by defining s(Z, Z', = cos( Z2). One immodiate failure mode of such an approach is when Z; and Z; have a different number of dimensions d d d'. However, even when the number of dimensions matches, any such approach that directly compares the two representations suffers from a subtle but important shortcoming of not being invariant orthogonal transformations. Consider an example where Z R with R being an orthogonal matrix, such that Z Z = O Vi. Even though cos (Z. Z 0 Vi, i.c., representations appear very dissimilar when directly measuring their cosine similarity, they are, however, from an information-theoree similarity score obtained from directly comparing points is misleading. In fact, previous work [LCunn et al.] [1990p; Xie et al.] (2017): : Liu et al.] (2021) has shown that orthogonal transformations do not change the training dynamics of neural networks and thus invariance to them is a desirable property for any similarity metric operating on neural representations, as also discussed in Kornblith al. (2019). A similar argument could be made against other choices of direct comparisons such Euclidean distance.
standpoint, identical for any downstream application{] Therefore, the low
To overcome the issues involved in directly comparing two different representations, we propose an indirect comparison. We leverage the simple, but powerful insight from prior work Kornblith et al. (2019); Kriegesko- rie et. al. (2008) that while we cannot, directly compare similarity across representations, we can do so within the same representation. We argue that the representations Z. and 2 of a point i should be considered sim- ilar across representations Z and Z' if their positions relative to other points in the respective representation are similar. Therefore, to determine whether the representations Z4 and Z of point are similar, we can first compare how similarly is positioned relative to all the other points within each representation. We then compare the relative position of i across both representations. This approach builds well-established principles from both representation similarity literature Kriegeskorte et al. (2008); Gower (1975 | Kornblith et al. 2019) and neuroscience Kriegeskorte & Kievit (2013): Cichy et al. (2017); Mehrer et al. (2021), w where shifts in relative positioning are key to analyzing changes in (brain) representations. Prior work, such as CKA, discusses the ongoing debate on the desirable properties of robust. representation similarity measures, and we adopt the properties it formulates as the basis for our analysis.
2.2 Measuring Similarity in the Relative Position of Points
We can now formally describe our proposed measure, Pointwise Normalized Kernel Alignment (PNKA), which calculates representational similarity for individual points by first. comparing the relative similarity between a point and other points within the same representation, and then across the two different repre- sentations. Given a set of (column-centered) representations Z (and analogously for Z') and a kernel k(.,.), we can define a pairwise similarity matrix between all N points in Z as (Z) with K K(Z),.; = k(Z,, 2,) In our work, we use linear kernels, i.e., k(Z,, 22) = 2, -z,, but other kernels, e.g., RBF Korublith et al. (2019) kernels, could be used well We leave the exploration of other types of kernels for future work. Given two similarity matrices K(2) and K KZZ), we measure how similarly point i is represented in Z and Z' by comparing its position relative to all other points. To this end, we define
K(2) K(2),, [K(Z) [K(())+]
PNKA(Z, Z', oos(K K(Z'))
where K(Z); and k(Z';; denote how similar point i is to all other points in Z and Z', respectively. We use cosine similarity to compare the rithen-rrpresentattinn similarity of a point with the other points, across representations, for two reasons. First, cosine similarity provides us with normalized similarity scores for each point. Second, by normalizing by the length of the similarity vectors K(2), and K(Z'),, we compare the relative instead of the absolute similarity of points, i.c., how similar point i is represented relative to points and j'. We can further extend our measure into an aggregate version (PNKA) between sets of representations by computing the average of the similaritics across all the N points.
2The projection matrix R is invertible, so by multiplying the weight matrix of any linear downstream operation 2 with R-1, it can be directly applied to 2' and produce the same result. 3 Results using the RBF kernel are consistent with those from the linear kernel, leading similar conclusions. Detailed results for all evaluation sections are provided in the corresponding appendices.
4

---

Published in Transactions on Machine Learning Research (05/2025)
1200
330
Number of Pairts
1000
Number Points
300
Number Points
500
250
150
100
100
50
0.5
0.5
PNKKA
PNICA
PNKA
(a) Group Fairness.
(b) Individual Fairness.
(c) Group and Individual Fairness.
OKA-6.79 75
OKAwG 66
OKAwb
Number of Points
06
Number Points
2300
3000
7000
5009
6000
06
46
1000
500
1000
PNKKA
PNKA
(d) Group Fairness.
(e) Individual Fairness.
(f) Group and Individual Fairness.
Figure 1: Distribution of PNKA similarity scores. The first row (blue plots) shows results for the COMPAS dataset, debiased with respect to race, while the second row (green plots) displays results for the Adult dataset, debiased based on gender. The vertical dotted line shows the overall similarity scores provided by CKA Kornblith et al. (2019). We compare the baseline representations (trained only for utility) with those from models trained using three different loss functions: Utility + Group Fairness (a, d), Utility Individual Fairness (b, e), and Utility + Group and Individual Fairness (c, f).
2.3 Properties of PNKA
PNKA captures the similarity between neighborhoods. Previously argued that for data point to be similarly reprosented across two representations, it should be similarly positioned relative to the other points in each representation. In Appendix A.1 we empirically show, across several architectures, datasets, distances measures, and values of k, a clear relationship between high PNKA scores and high overlap of nearest neighbors across representations, indicating that PNKA captures how similar the neighborhoods of the points are. We further show in the same Appendix that the direct comparison of representations through cosine similarity does not exhibit the same trend.
Relationship of PNKA with aggregate measures of representational similarity. PNKA is inspired by CKA, and in Appendix A.2 we empirically show that the aggregate version of our measure (PNKA) produces results close to CKA.
Invariance properties. Previous work Kornblith et al. 2019 has identified invariance to orthogonal transformations and isotropic scaling as desirable properties for representational similarity measures. We provide mathematical proof that measure possesses both of these invariance properties in Appendix A.3
3 Investigating Debiased Tabular Data Representations
Prior work (Zemel et al. 2013 Creager al., 2019 Louizos al. 2015 Xu et al. 2018) has proposed lcarning fair data representations which retain minimal information about sensitive features. Here, we conduct case study to verify that PNKA works expected when auditing the effect of debiasing approaches. Our goal in this section is to establish that PNKA yields meaningful and reliable insights.
To achieve our goal, we compare the original (baseline) and debiased representations, focusing on well- established debiasing methods. One such techniques the approach proposed by Zemel et al. (2013), which learns representations of tabular data by optimizing a loss function that maintains as much utility as possible, while removing information about protected attributes. The learning algorithm modifies the data representation based on three distinct objectives: classification accuracy (denoted by utility), statistical

---

Published in Transactions on Machine Learning Research (05/2025)
0.22
0.4
0.5
proporton
Proportion
0.4
0.4
0.4
Caucestan
Female
Coucassin
Cauc.sson
Categories
Categories
Categories
(a) Group Fairness.
(b) Individual Fairness.
(c) Group and Individual Fairness.
0.2
0.8
0.5
proportion
Propertion
0.6
0.4
0.4
Categories
Categories
Categories
(d) Group Fairness.
(e) Individual Fairness.
(f) Group and Individual Fairness.
Distribution of the binary attributes for the 10% most alfected individuals (i.e., lowest PNKA score). The first row (blue plots) shows results for the COMPAS dataset, debiased with respect to race, while the second row (green plots) displays results for the Adult dataset, debiased based on gender. We compare the baseline representations (trained only for utility) with those from models trained using three different loss functions: Utility Group Fairness (a, d), Utility + Individual Fairness (b, e), and Utility + Group and Individual Fairness (c, f). The horizontal red dotted line shows the population average per attribute.
Figure 2:
parity (to achieve group fairness, and data loss (as a proxy for achieving individual fairness). By applying combinations of these objectives, we obtain four types of representations, one based on utility, serving as the baseline, and three debiased versions: utility and group fairness, utility and individual fairness, and a combination of all three objectives (utility, group and individual fairness).
We use the COMPAS [Larson et al. 2016) dataset, debiased for race, and the Adult (Becker & Kohavi 1996) datasct, debiased for gender [] to analyze with PNKA the effect of the different debiasing objectives. We first investigate the overall effect of debiasing in Section 3.1 followed by a more detailed analysis of the individuals whose representations change the most in Section 3.2 Finally, we show that the predictions made based on PNKA scores match the prodictions observed on downstream tasks in Scction 3.3
3.1 How Do Individual Representations Change?
The distribution of PNKA similarity scores, obtained by comparing the representations of the baseline with each of the debiasing methods, is visualized in Figure 1 with COMPAS Larson et al. (2016) results depicted in the first row (Figures Ta Tc and Adult Becker & Kohavi (196)) shown in the second row (Figures) 1d 11 As shown in Figures la and ld for both datasets, under group fairness, the majority of individuals maintain high representational similarity, suggosting that the majority of individuals' reprosentations remain similar to the baseline, with significant changes primarily occurring for a small subset. of the population. contrast, as depicted in Figures 1b and Tel individual fairness constraints lead to more uniform representational change across the entire population, slightly impacting nearly every data point's representation. Finally, when combining group fairness and individual fairness, observed in Figures Ic and If the impact on the data points' representations varies depending on the dataset. For the COMPAS dataset, the resulting pattern closely resembles that observed under the influence of only group fairness. In contrast, the pattern observed for the Adult dataset aligns more closely with the effects seen when applying only individual fairness.
We pre-process each dataset as done Zemel Trusted-AT/ATEE
2013 More information be obtained in https::/gSttubb.o

---

Published in Transactions on Machine Learning Research (05/2025)
3.2 Whose Representations Change the Most?
Next, we analyze the 10% of the population with the lowest PNKA similarity scores, representing those whose representations have undergone the most significant changes due to the applied fairness constraints. We focus on the distribution of the protected attributes in this group, specifically Figure 2 illustrates the distribution of the binary sensitive attributes for the subset of the most affected people in both COMPAS, in the first row, and Adult, in the second row. The horizontal red dotted line shows the population average per attribute [5]
race and gender.
In the COMPAS dataset (Figures 2a 2ch, where data representations are debiased according to race, we observe that the gender distribution remains relatively stable, aligning closely with the population bascline distribution, while the race distribution exhibits notable shifts. More specifically, for COMPAS, under the group fairness objective, the individuals whose representations are altered the most are primarily Caucasians. This happens regardless of whether the group fairness constraint has been used alone or in combination with individual fairness. Howover, when only individual fairness is applied, the distribution resembles that of the baseline one, suggesting a minimal impact of individual fairness on attribute distribution compared to group fairness.
In contrast, as shown in Figures 2d 2 for the Adult dataset, whose representations are debiased with respect to gender, under group fairness conditions, the female category is the most drastically affected group. This happens regardless of whether individual fairness is applied in combination with group fairness. Interostingly, however, when individual fairness constraints are applied, the male representations are more affected than female ones. The race attribute largely retains its baseline distribution.
3.3
Do PNKA's Predictions Match Downstream Outcomes?
The previous results suggest that under some debiasing objectives, the data representations of specific demo- graphics are lispply affected. However, do the changes in the data representations also translate to changes in the behavior of models that use them? In other words, can the analysis at the representation level provide insights at the output level?
To test the utility of PNKA in predicting downstream behavior, we use the insights obtained from PNKA in Sections 3.1 and 3.2 together with prior knowledge about the design of the fairness constraints, to predict what outcomes can expect from models that use the modified data representations. We then train models on the representations to test whether the predictions made using PNKA match the actual outcomes. In our analysis, we focus on the effect of the GF-constraints, because (i) we observe in Section 3.1 that they strongly affect a small part of the population that we can easily characterize in Section 3.2 (as opposed to the more uniform changes across the board for the other constraints, which make it harder to characterize the demographics of the affected individuals), and (ii) it easy to understand their effect, namely minimizing disparities between groups that differ in the protected attribute.
Hypothesis: For the COMPAS dataset, we observe in Figure 2a that the people whose representation are changed the most by the group fairness constraints are predominantly Caucasians. Given that the COMPAS datasot is known to exhibit a bias in favor of Caucasians, i.c., that they are less likely to recidivate than non-Caucasians, a change in the representations of Caucasians therefore suggests that a model using the group fairness debiased representations might assign Caucasians less favorable outcomes than one that uses the baseline representations. Conversely, for the Adult dataset we observe in Figure [d that females are the predominant demographic in the 10% of most affected representations. Given that in this dataset males tend to be more likely to have the positive label (income above 50K per year), we thus expect that model using group fairness debiased representations will assign more favorable outcomes to females compared to a model using baseline representations.
To test whether these hypotheses, based on PNKA's similarity scores, are correct, we train logistic regression models for each of the representation types. On the COMPAS dataset the goal is to predict recidivism, whereas on the Adult datasets the goal is to predict whether an individual's income is above 50K dollars
We show the distribution of the remaining attributes in Appendix R.2

---

Published in Transactions on Machine Learning Research (05/2025)
per year. The overall accuracy and debiasing results of these models presented in Table 1 Following the approach used by Zemel et al. (213), we measure group fairness through the statistical parity difference, which is the ratio of favorable outcomes received by unprivileged versus privileged classes. In the context of the COMPAS dataset, where the model predicts the risk of criminal recidivism, a positive (negative) score indicates a higher predicted risk for Can-Cauassians (Caucasians). For the Adult dataset, where the model trained to predict whether income exceeds 50K per year, a positive (negative) score mcans females (males) are less likely to receive a high annual income. Individual fairness (Zemel et al.) 2013) is assessed using the consistency score, which evaluates how similar the predicted labels are for neighboring instances. Table I shows that in both the COMPAS and Adult datasets, the inclusion of group fairness leads to models exhibiting a lower statistical parity difference compared to the baseline. This suggests a reduction in bias against non-Caucasians for COMPAS, and females for the Adult dataset.
To better understand whether the reduced statistical parity on both datasets is indeed due the hypothe- sized changes for each of the groups, we investigate how the benefits that cach of the groups receives on the two datasets changes. In the case of COMPAS, false positives are deemed to be far more costly the widely accepted Blackstone's is ratio posits that "it is better that ten guilty persons escape than that one innocent suffer" [Blackstone] 2016). On the other hand, in the case of Adult, false negatives are decmed to be more costly, as it suggests that the individual has a lower income, which can be associated with limited financial resources, reduced access to opportunities, and potentially lower socio-economic status. Thus, for COMPAS, we measure precision and for the Adult dataset, recall, since these measures capture the change in beneficial outcomes.
The results for each protected group are shown in Table 2 The results corroborate the earlier predictions made by PNKA. For the COMPAS dataset, the use of group fairness significantly impacts Caucasians, leading to a notable decrease in their precision score, In particular, the precision score changes more strongly for Caucasians (~~ 13%) than for non- non-Caucasians (~~ 1%). On the Adult dataset, the use of group fairness results in substantial increase of recall scores for females. Again, females experience a much larger change in recall (~~ 34%) than males (~ 6%). Both of these findings match the predictions made using PNKA, i.e., the groups achieving the largest change in their outcomes are those that are most prevalent among the individuals whose representations changed the most due to the group fairness constraints.
Takeaways. The results in this section show that PNKA can be valuable auditing tool, as it enables a comprehensive and detailed analysis of the changes that occur when modifying representations. Further, we demonstrate that the insights provided by PNKA are reliable, exhibiting predictive power for downstream applications and aligning with findings from prior outcome-basse studies. Importantly, unlike prior work, which requires studying each downstream application of the representations separately and depends on prior knowledge to identify relevant groups, PNKA directly quantifies the extent of change in representations the individual level, offering a more targeted and efficient approach.
Dataset
Constraint Type
Accuracy
Statistical Parity
Consistency
Utility Group Fairness
0.6780
0.2308
0.9811
COMPAS
Utility +
0.6585
0.0376
0.9220
Utility
0.8015
-0.1784
0.9385
Adult
Utility +
Group Fairness
0.7878
-0.0426
0.9968
Table 1: Overall accuracy, statistical parity, and consistency results for lincar regression models trained with baseline (Utility) data representation and with the group fairness debiased representations. For statistical parity, in the context of the COMPAS dataset, where the model predicts the risk of criminal recidivism, a positive score indicates a higher predicted risk for non-Caucasians. On the other hand, in the Adult datasot, where the model is trained predict whether income exceeds 50K per year, a negative score means fewer females are likely to receive high annual income. Tdeally, statistical parity should be 0, while score of 1 is optimal for both accuracy and consistency.
8

---

Published in Transactions on Machine Learning Research (05/2025)
Measure
Dataset
Protected Attribute
Data
Utility
Utility
Group Fairness
Caucasian
0.6797
0.5465
Precision
COMPAS
Non-Caucasian
0.6980
0.7090
Male
0.4340
0.3731
Recall
Adult
Female
0.0752
0.4146
Precision and recall scores for COMPAS and Adult datasets, respectively, of linear regression models trained with bascline (U) data representation and the group fairness debiased data representations. In the COMPAS dataset, where there is a known bias against non- non-Caucasians, we found that precision for Caucasians is reduced due to group fairness interventions, indicating an adverse effect. In the Adult dataset, characterized by bias against females, females are positively affected by group fairness.
Table 2:
4 Investigating Debiased Language Embeddings
In our next case study, analyze PNKA tool for investigating debiased word embeddings. Previous work [Bolukbasi et al. (2016); Gonen & Goldborg (2019) Lu et al. (2020) has identified the presence of stereotypical biases in word embeddings, L.., vector representations of words, and since has developed several debiasing methods Bolukbasi et al. (2016); Zhao et al. (2018): Kaneko & Bollegala| (201)). Among them
are: (1) Gender Neutral (GN-)GloVe Zhho et al. (2018). which focuses on discntangling and isolating all the gender information into certain specific dimension(s) of the word vectors; and (2) Gender Preserving (GP- Gl Kaneko & Bollegala (2019), which targets preserving non-discriminative gender-related information while removing stereotypical discriminative gender biases from pre-trained word embeddings. The latter method is also used to finetune GN-GloVe embeddings, creating a third, combined method called GP-GN- GloVe.
We start, in Section 4.1 by briefly explaining how these debiasing approaches are traditionally evaluated. In Section 4.2 we show how PNKA can be leveraged to investigate whether these approaches modify the group of words (i.e., stereotypical words) as originally intended. Using the insights from PNKA, we formulate hypothesis about the effects of the debiasing approaches, and test them in Section 4.3
4.1 Traditional Measures for Debiased Word Embeddings
The aforementioned debiasing models for word embeddings have been originally evaluated using Sem- Bias Zhao et al. (2018), dataset designed to assess whether the debiasing methods have successfully removed stercotypical gender information from the word embeddings. Each instance in SemBias consists of four word pairs: a gender-definition word pair (e.g. "waiter - waitress"), a gender-stereotype word pair (e.g. "doctor - nurse"), and two other word-pairs that have similar meanings but no gender relation, named gender-neutral (e.g. "dog -a---aa cat"). To assess the bias in word embeddings, the evaluation scheme measures the cosine similarity with the canonical gender vector, i.e. cos( 2 - 5. he she) for each of the four word pairs b) in SemBias instance. The word pair with the highest cosine similarity is selected as the "predicted" answer. II the word embeddings are correctly debiased, then the cosine similarity of the he she vector with the gendor-defnition words should be high, and the similarity with the gender-stercotype words should be low, i.e., the frequency of predictions for these categories should be high for the gender-defining word pairs and low for gender stereotypical word pairs.
Thus, GP- and GN-GloVe evaluate how bde)bi embeddings are based on whether they predict stereotyp- ical or definitional word pairs in each instance of the SemBias dataset. We show results for evaluating GP- and GN-GloVe on SemBias in Appendix C.2 The evaluation shows that GP-Golve embeddings offer only a marginal improvement over the bascline embeddings, while GN-GloVe and GP-GN-GHI embeddings show substantial reductions in bias in the prediction task. These findings suggest that the latter models are more effective in mitigating gender bias in word embeddings.
9

---

Published in Transactions on Machine Learning Research (05/2025)
Percariagne of Points
Gondar Definition
Percentage of Points
50
Gondar Neutral
210
of
890
Gender Stereotype
40
40
40
0.2
0.6
0.6
0.5
0.5
0.4
0.5
0.5
PNKA
PNICA
PNKKA
(a) GloVe x GP-GloVe.
(b) GloVe x GN-GloVe.
(c) GloVe x GP-GN-GloVe
Figure 3: Distribution of PNKA scores per group of words for SemBias dataset Zhao et al. 2018). We
compare the baseline (GloVe) model and its debiased versions. Words with the lowest similarity scores are the ones that change the most from the baseline to its debiased version. Across all debiased embeddings, the words whose embeddings change the most are the gender-defiitte words.
4.2 How Do Individual Representations Change?
We next employ PNKA to better understand which word embeddings have changed the most due to the debiasing procedure of GP- and Ve methods. As before, we use PNKA to measure similarity between the original GloVe (baseline) and the debiased versions of GloVe embeddings, i.e., (GN-)GloVe Zhao et al. (2018), Gender Preserving (Ge and GP-GN-GloVe Kaneko & Bollegala 2019 Figure 3 shows the distribution of PNKA similarity scores for words in the SomBias datasct grouped by their category (i.c., gender defining, gender neutral, and gender stereotype).
We first observe, in Figure that GP-GIo Ve representations exhibit high degree of similarity to the original GloVe embeddings for almost all words. This suggests that the GP-GloVe method may not significantly alter the word representations maintaining close resemblance to the original embeddings for most words. However, Figures 36 and 3e show that GN-GloVe and GP-GN-GloVe, respectively, considerably change the representations for subset of the words. This observation aligns well with the results of the prior evaluation, detailed in Table | of Appendix C.2 in which GP-GloVe was shown to yield results similar GloVe (suggesting similar representations), while GN-GloVe and GP-GN-GloVe achieve better debiasing results.
Moreover, illustrated in Figure 3 an intriguing pattern emerges across all three debiasing methods: the words with lower PNKA scores, i.c. the words whose representations change the most, belong predominantly to the gender-definitio category. This finding stands in stark contrast to the expected behavior: As discussed in Zhao et al. 2018) and Kaneko & Bollegala (2019), the primary goal of these debiasing techniques for word embeddings is to retain gender-specific information in feminine and masculine words (i.e., gender-definig words), maintain ncutrality in gender-neutral words, and climinate biases in stercotypical words. Thus, the expectation is that debiasing should primarily alter gender-stereotsypcal word embeddings, while mostly preserving gender-definitional ones.
4.3
Do PNKA's Predictions Match the Projection Analysis?
Our analysis using PNKA reveals that, contrary to the expected effect of debiasing, the most profound changes occur the gender-defining words, challenging the conventional understanding of how these debias- ing methods function and prompting a more careful analysis of their effects. We formulate a new hypothesis about the impact of debiasing word embeddings: instead of removing the gender information in gender- stereotypical words as initially intended, debiasing methods inadvertently amplify gender information in the gender-definition words. Such an effect could be missed by the conventional evaluation procedure discussed previously, which only assesses the relative cosine-similarity. Increasing the gender-related information ion in cender-defining words would make gender-defiing words more conder-aligned and thus increase their pre- diction frequency, relative to other word pairs in SemBias. This effect could be achieved, however, without removing gender-related information from the gender stereotypical words, as intended by the debiasing methods.
10

---

Published in Transactions on Machine Learning Research (05/2025)
Percentage Difference
Gender Definition
Percentagge Differencc
Percentage Difference
Gender Stereotype
Gender Neutral
2.0
1.5
10
0.2
0.4
0.6
0.2
0.6
0.8
0.4 0.6
0.8
PNKA
PNKA
PNKA
(a) GloVe x GP-GloVe.
(b) GloVe x GN-GloVe.
(c) GloVe x GP-GN-GlOVe
Figure 4: Relationship between PNKA A scores (x-axis) and percentage difference (y-axis) in magnitude of the projection on the gender direction he she. A positive or negative percentage difference value indicates a shift in magnitude along the gender direction. Word embeddings that change their gender information are the ones that obtain low PNKA scores.
To test this hypothesis, we measure for each word how much its embedding changed in terms of gender information, when compared to the original GloVe embedding, by projecting it onto the canonical gender vector he - she. More specifically, for each embedding approach e and word i, we project the corresponding (e)
word embeddings df()) =(i)) onto the gender vector direction g(a) the o magnitudes p = oi higher -f is, the more gender information contained in the word embedding vector of") To understand how much each of the debiased embedding methods change the amount of gender information, relative to the
(e) and compute the projection l -g(eeT, where - represents a dot product, and 9 is the normalized gender direction. The
original glove embeddings, we analyze the percentage difference in magnitude, defined as w/ p (glave)
lp. (
A w = 0 indicates that the gender information in the debiased embedding has not changed relative to
(baseline) GloVe, while w(() 0 (w(*) 0) indicates an increase (decrcase) in the male (female) gender information associated with word i.
Figure 4 depicts the relation between PNKA scores (x-axis) and the percentage difference in magnitude (y-axis) for each word in the SemBias dataset. We can see that the GN-GloVe and GP-GN-GloVe debiasing methods primarily amplify the gender information in gender-definition words (red dots), rather than reduce it for gender-stcrootype words (blue dots), i.e., w/(h) * 0 for gender-defnnng and w(() ~ for gender-stcrcotype words. The words exhibiting the most significant change in gender information, identified as the ones with lower PNKA similarity scores, predominantly fall within the gender-defining category. This observation supports our alternative explanation that these debiasing mothods might be enhancing gender information in gender-definnng words, rather than diminishing it in gender-stereolyyical words.
Finally, we applied a similar analysis using PNKA to investigate bias mitigation efforts in contextualized embeddings of transformer-basee language models. Unlike static word embeddings, these models generate word representations that depend on surroudding context, which makes identifying and mitigating biases challenging yet crucial, given that biases in the pre-trained models can propagate to numerous downstream tasks. We focused on a debiasing method that aims to remove gender bias from contextual representations of stereotypical words (c.g., associating "math" with a male bias or "poetry" with a female bias) while retaining gender-specific information in gender-defining words. Our analysis using PNK Areveass that, similar to findings with static embeddings, the debiased contextualized embeddings showed only minimal shifts along the gender direction, failing to reduce bias in the intended stereotypical contexts. Detailed results from this investigation are available in Appendix D
Takeaway. The findings in this section shows that PNKA is powerful tool for auditing word embedding debiasing techniques, as it allows for a detailed, represenlation-leve analysis of how gender information is altered by these methods. Through PNKA, we gain reliable insights that not only complement traditional outcome-basod evaluations but also reveal underlying representational shifts that might otherwise go un- noticed. Crucially, unlike prior approaches that rely on predictive task assessments or predefined gender categories, PNKA quantifies representational changes at the individual word level, enabling more precise
11

---

Published in Transactions on Machine Learning Research (05/2025)
and efficient evaluation of bias mitigation efforts. This ability to identify specific shifts in gender-related representations positions PNKA as an essential tool in refining and improving debiasing strategies.
5
Related Work
In this section, we first review prior work on developing and assessing debiasing methods in the fairness domain. We then review work on representational similarity measures.
5.1 Fairness in ML Systems
ML algorithms have become fundamental to several aspects of daily life, yet these models often exhibit discriminatory behavior that negatively impacts protected groups. To address these biases, several debias- ing methods have been developed, intervening at different stages of the ML pipeline (d'Alessandro et all, 2017 Mchrabi et al., 2021). Pre-processing debiasing methods, including rewcighting, resampling, data aug- mentalion, or even learning fair data represenlations, modify the training data to remove the underlying discrimination before it is used by the model (Brunet et al. 2019 Kamiran & Calders) 2012). In-processing methods modify the learning algorithms, by, for example, adding constraints to the optimization objective, in order to remove discrimination during the model training process (Zafar et al. 2017c Zhang et al.) 2018). Finally, post-processing methods change the decision thresholds or apply recalibration methods to adjust model outputs to achieve fairness after the model has been trained (Hardt al.) 2016 Bolukbasi et al.
2016). PNKA can be used to analyze any debiasing method that operates at the pre- or in-processing stages of the ML pipeline.
With the increased adoption of debiasing methods, auditing these approaches has reccived considerable at- lention. Most fairness measures fall under two main categories (Pessach & Shmueeij 2022): : (1) group fairness,
which requires parity of some statistical measure across protected groups (Dwork et al. 2012 Hardt et al. 2016). (2) individual fairness, which requires that similar individuals be treated similarly (Joseph et al. 2016
Heidari & Krause 2018). Both categories are output-basele i.c., they solely look at the predictions/dccisinn made by the model.
Some work Gonen & Goldberg 2019 Caliskan et al. 2017 Bolukbasi et al. 2016 Ethayarajh et al. 2019
Garg et al. 2018) has analyzed representations, mostly in the realm of word embeddings. For instance, Bolukbasi et al. (2166) evaluates gender bias in word embeddings by identifying a gender subspace using explicitly gendered word pairs and analyze the proximity of gender-neutral words to gender-specific terms, uncovering socictal stercotypes. Caliskan et al. (2017) uses the WEAT test to systematically measure biases in word embeddings by comparing the association strength between pairs of target words (e.g., "male" and "female" names) and attribute words (e.g., "carcer" and "family").
(e.g.,"male"
The closest work to ours is the one by Gonen & Goldberg (2019), which investigates word embeddings and shows that debiasing methods for word embeddings mostly hide the bias, instead of removing it. More specifically, using a clustering algorithm, they show words that receive implicit gender from social stercotypes (e.g. receptionist, captain) still lend to group with other implicit-gender words of the same gender, similar as for non-debiased word embeddings. Using PNKA, we reach similar conclusion: these debiasing models primarily mask, rather than mitigate, the bias. However, as we showed in Section 4 our analysis offers a fresh perspective and instead reveals that these methods are augmenting the gender-related information of the gender-defining words, instead of removing this information from the gender-stereotyyiic words. None of the previous work provide a general measure that can be applied to representations in different contexts. To the best of our knowledge, our measure is the first one that can be broadly applied to any debiasing method that alters the representations used by models.
5.2 Representational Similarity Measures
Several measures have been proposed and used as tools to better understand the internal representations of machine learning (ML) models. Recently, approaches that compare the representational spaces of two models by measuring representational similarity have gained popularity [aakso & Cottrell (2000 : Li et al.
12

---

Published in Transactions on Machine Learning Research (05/2025)
2015); Wang et al. (2018) Raghu al. (0177): Morcos et al. (0118): Kornblith et al (2019). At their core, representational similarity measures (RSMs) quantify how set of points are positioned relative to each other within the representation spaces of two different models. Among the RSMs proposed in the literature, CKA Kornblith et al. (2019) has gained popularity and has now been extensively used to study representations ions Nguyen et al. (2021): ; Ramasesh et al. (2020); Raghu et al. (2019 2021 CKA based on the idea of first choosing kernel and then measuring similarity as the alignment between these two kernel matrices. We take inspiration from this insight propose PNKA.
Most widely used RSMs, however, yield only aggregate estimate (i.e., single score) of similarity across entire set of points. Being limited to aggregate measurements makes these measures unsuitable to study nuances of representational similarity at a more granular, local level. Therefore, approaches of represen- tational similarity for individual points have been proposed, such as for words Hamilton et al. (2016) and nodes in graphs Chen al. (2021). However, the applicability of these measures is constrained due to their task-spocific nature. Work by [hahh ct al. (2023) proposes a method estimate the contribution of individual points to learning algorithms, but mainly focuses on understanding what, fealures of the inputs are encoded in the representations, and do not evaluate the similarity of representations directly. Finally, a pointwise RSM proposed by Moschella et al. [2022) resembles our proposed mcasure PNKA. However, the goal of Moschella et al. 2022) differs significantly from ours, as they use their measure to enable model stitching, whereas we employ PNKA to audit representations. To the best our knowledge, this is the first time representational similarity has been employed to study the effects of debiasing methods at a ime-grained level.
6 Discussion
We introduce a framework that uses ropresentational similarity to assoss how debiasing methods impact individual data representations. Crucial this framework is PNKA, pointwise representational similarity measure that quantifies changes in an individual's representation due to fairness interventions. Our findings datasets like COMPAS and Adult confirm established insights while uncovering new effects, highlight- ing differences between group- and individual-level fairness. PNKA also enables anticipation of biases and fairness constraints' impacts before model training and deployment, and reveals inconsistencies in how de- biasing methods alter targeted groups. Our results suggest that existing fairness metrics are limited, and pointwise representational similarity mcasures such as PNKA provide a comprehensive tool for auditing fairness interventions at both group and individual levels.
References
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. A reductions approach to fair classification. In ICML, volume 80, pp. 60 69, 2018. URL http://prooeedingss mlr.
preas//880/agaraa1188 hhtml
Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preotiuc-Pietro, and Vasileios Lampos. Predicting judi- cial decisions of the european court of human rights: A natural language processing perspective. PeerJ computer science, 2:e93, 2016.
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. In Ethics of data and analytics, pp. 254-204 264. Auerbach Publications, 2016.
Barry Becker and Ronny Kohavi.
Adult. UCI Machine
Learning Repository, 1996.
DOL:
Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H Chi. Data decisions and theoretical implications when adver- sarially learning fair representations. arXiv preprint
2017.
William Blackstone. Commentaries on the Laws of England. Oxford University Press, 2016.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam Kalai. Man computer programmer as woman is homemaker? debiasing word embeddings. Advances in neural information processing systems, 29, 2016.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXic:2108.07258 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakanian, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurel information processing systems, 33:1877 1901, 2020.
Marc-Etienne Brunet, Colleen Alkalay- Ashton Anderson, and Richard Zemel. Understanding the origins of bias in word embeddings. In International conference on machine learning, pp. 803-811, PMT.R, 2019.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on faürness, uccounlability and transporency, pp. 77-91. PMTR, 2018.
Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356(63344:183-186, 2017.
Zuohui Chen, Yao Lu, Jinxuan Hu, Wen Yang, Qi Xuan, Zhen Wang, and Xiaoniu Yang. Graph-based similarity of neural network representations. arXiv preprint arXiv:2111.1116; 2021.
Radoslaw Martin Cichy, Aditya Khosla, Dimitrios Pantazis, and Aude Oliva. Dynamics of scene represen- tations in the human brain revealed by maggeetoenceephallggaaggy and deep neural networks. Neurolmage,
153:346-3555 358, 2017.
Elliot Creager, David Madras, Jörn-Henrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. International conference on machine learning, pp. 1436-1445 PMLR, 2019.
Brian d' Alessandro, Cathy O'Neil, and Tom LaGatta. Conscientious classification: A data scientist's guide to discrimination-awarrr classification. Big data, 5(2):120- 134, 2017.
Emily Diana, Wesley Gill, Michael Kearns, Krishnaram Kenthapadi, and Aaron Roth. Convergent algorithms for (relaxed) minimax fairness. arXiv preprinl 1xXi::2011.03100 2020.
Michele Donini, Luca Oneto, Shai Ben-David, John Shawe-Taylor, and Massimiliano Pontil. Empirical risk minimization under fairnoss constraints. In NIPS, pp. 2796-2806, 2018.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In ITCS, pp. 214 226, 2012. ISBN 9781450311151. doi: https://doi.orgg/ 10/g/10..10 1141/2090230 2090255
2090225.. URL
Harrison Edwards and Amos Storkey. Censoring representations with an adversary. In ICLR, 2016. URL http://arxiv.org org/ abs/ /111/11111.1 05897
Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. Understanding undcsirable word embedding associ- ations. In Auna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1696-1705, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. L8855/v11/19-111
URL
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkalasubramanian. Certifying and removing disparate impact. In KDD, pp. 259-268, 2015.
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635 E3635 2018.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Robert Geirhos, Carlos R. M. Temme, Jonas Rauber, Heiko H. Schütt, Matthias Bethge, and Felix A. Wichmann. Generalisation in humans and deep neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL 087775566e000fff33366ff666662223 pdf
Naman Goel, Mohammad Yaghini, and Boi Faltings. Non-discriminatory machine learning through convex fairness criteria. AAAT, 32(1), 2018. URL https://ojs.aant http:///ss..aa..orr///eeeee.eee htgg:/nnee..pp//AAA//tttttlallll
11662
Hila Gonen and Yoav Goldberg. Lipstick on a pig: Debiasing methods cover up systematic gender biascs word embeddings but do not remove them. In Proceedings of NAACL-HLT, pp. 609 -61, 2019.
John C Gower. Generalized procrustes analysis. Psychometrika, 40:33-51, 1975.
William Hamilton, Jure Leskovec, and Dan Jurafsky. Cultural shift or linguistic drift? comparing two computational measures of semantic change. In Proceedings of the conference on empirical methods in natural language processing. Conference on empirical methods in natural language processing, volume 2016, pp. 2116. NIH Public Access, 2016.
Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In NIPS, vohume 29, pp. 3323, 2016. URL httpp::/prroceedings neurips wwurips.cc/papeeeeeee cc/papper/2016ffille/ pdf
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Hoda Heidari and Andreas Krause. Preventing disparate treatment in sequential decision making. In IJCAI, pp. 2248 2248-2254, 2018
Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
Matthew Joseph, Michael Kearns, Jamie H Morgenstern, and Aaron Roth. Fairness in learning: Classic and contextual bandits. Advances in neural information processing systems, 29, 2016.
Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrimination. Knowledge and information systems, 33(1):1 33, 2012.
Masahiro Kancko and Damushka Bollegala. Gender-presarving debiasing for pre-trained word embeddings. Proceedings of the 57th Annvual Meeling of the Associalion for Compulational Linguistics, pp. 1641 -1650, 2019.
Masahiro Kaneko and Danushka Bollegala. Debiasing pre-trained contextualised embeddings. arXiv preprinl arXiv:2101.09523 2021.
Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing fairness gerrymandering: Au- diting and learning for subgroup fairness. In Proceedings of the 35th International Conference Machine PMLR, 10-15 Jul 2018.
Learning, volume 80 of Proceedings of Machine Learning Research, pp. URL
mlr,
html
Joon Sik Kim, Jiahao Chen, and Ameet Talwalkar. Fact: A diagnostic for group fairness trade-offs. In International Conference on Machine Learning, pp. 5264 5274. PMLR, 2020.
Simon Kornblith, Mohammad Norouzi, Honglak Lec, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519 3529. PMLR, 2019.
Nikolaus Kriegeskorte and Rogier A Kievit. Representational geometry: integrating cognition, computation, and the brain. Trends in cognilive sciences, 17(8):401-411 2013.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Nikolaus Kriegeskorte, Maricke Mur, and Peter A Bandettini. Representational similarity analysis the branches of systems neuroscience. Fronliers in systems neuroscience, pp. 4, 2008.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. Measuring bias in contextualized word representations. arXiv preprint arXic: 10.6. 166000976. 2019.
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva, Counterfactual fairness. Advances in neural information processing systems, 30, 2017.
Aarre Laakso and Garrison Cottrell.
Content and cluster analysis: assessing representational similarity in neural systems. Philosophical psychology, 13(1):47-700 76, 2000.
Preethi Lahoti, Krishna P Gummadi, and Gerhard Weikum. ifair: Learning individually fair data represen-
tations for algorithmic decision making. In 2019 ieee 35th international conference on data engineering (icde), pp. 1334-1345. IEEE, 2019.
Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed H. Chi. Fairness without demographics through adversarially reweighted learning. arXiv preprint arXiv:2006. 13114, 2020.
Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. Data and analysis for 'How we analyzed the COMPAS recidivism algorithm'. https://gitthub.com/ http://hhubcooopppppbicaaa/////e http//bbiccccoopppeeeeeeeeee /compas-analysis, 2016.
Yann LeCun, Ido Kanter, and Sara Solla. Second order properties of error surfaces: Learning time and generalization. Advances in neural information processing systems, 1990.
Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. A survey on fairness in large language models. arXiv preprint arXiv:2308.10143 2023.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? arXic preprint arXi::1511.0755 1.7544,, 2015.
Znaonui Liang, Gang Zhang, Jimmy Xiangji Huang, and Qmming Vivian Hu. Deep learning for healthcare docision making with emrs. In 2014 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 556-559 IEEE, 2014.
Weiyang Liu, Rongmei Lin, Zhen Liu, James M Rehg, Liam Paull, Li Xiong, Le Song, and Adrian Weller. Orthogonal over- over-voraaaae parameterized training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7251-7260, 2021.
Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. The variational fair autoen- coder. arXiv preprint urXiv:1511.00830, 2015.
Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Amupam Datta. Gender bias in neural natural language processing, Logic, Language, and Security: Essays Dedicated to Andre Scedrou on the Occasion of His 65th Birthday, pp. 189-202, 2020.
David Madras, Elliot Creager, Toniann Pitassi, and Richard S. Zemel. Learning adversarially fair and transferable representations, In ICML, volume 80, pp. 3381-3390, 2018. URL http://proceedingss mlr. press//80/mmdrra11Baa
Natalia Martinez, Martin Bertran, and Guillermo Sapiro. Minimax pareto fairness: A multi objective perspective. In ICML, volume 119, pp. 6755 6764, 2020. URL http::/pproceddngg..u http://pcceeciti.....r.rrrre////
martinez20a.htai
Chandler May, Alex Wang, Shikha Bordia, Samuel R Bowman, and Rachel Rudinger. On measuring social biases in sentence encoders. Xixio preprint or Xin: 100:1903.1111 10561, 2019.
16

---

Published in Transactions on Machine Learning Research (05/2025)
Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy. An empirical survey of the effectiveness of debiasing techniques for pre-trained language models. arXiv preprint 1rX:::2110008222 10.08527, 2021.
Ninarch Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6):1-35, 2021.
Johannes Mehrer, Courtney Spoerer, Emnr C Jones, Nikolaus Kriegeskorte, and Tim Kietzmann. An ecologically motivated image dataset for deep learning yields better models of human vision. Proceedings of the National Academy of Sciences, 21801)120011417118, 2021.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems, 31, 2018.
Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodola. Relative representations enable zero-shot latent space communicttion. arXiv preprinl arXiv:2209.1544 2022.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In International Conference on Learning Representations, 2021.
Cathy O'neil. Weapons of math destruction: How big data increases inequality and threatens democracy. Crown, 2017.
Manisha Padala and Sujit Gujar. Fnne: Achieving fairness through neural networks. In IJCAI-20, pp. 2277-2283, 7 2020. doi: /19663/jjaat.222/3311 URL https://doi org/olgg///o 20..4493/ijcaii................. 2020//15
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE Europeon symposium on security and privacy (FuroSBP), pp. 372-387. IEFE, 2016.
Dana Pessach and Erez Shmueli. A review on fairness in machine learning. ACM Computing Surveys (CSUR), 55(3):1-44, 2022.
Novi Quadrianto, Viktoriia Sharmanska, and Oliver Thomas. Discovering fair representations in the data domain. In CVPR, pp. 8227 8236. Computer Vision Foundation / IEEE, 2019. doi: 10.1109/CVPR.2011 00842. URL htttp://openaccess thecvf. wwtecvffcom/cae www/cooto/tttcocooooooooommo. _______________________ 2019/htmm/Quadriaatts Distooerriiccinnn.... Fair www.Reeereeettiinnss..m in the Data Dataa_Doaaainnnnne CVPR__CCPR_____ 2019 2wwp__a1e91a_____111111_111 html
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv:1909.0911 09157, 2019.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems. 34:12116 12128, 2021.
Vinay V Ramasesh, Ethan Dyer, and Maithra. Raghu. Anatomy of catastrophic forgetting: Hidden repre- sentations and lask semantics. arXiv preprinl or arXiv:2007.077400 2020.
Hee Jung Ryu, Hartwig Adam, and Margaret Mitchell. Inclusivefacenet: Improving face attribute detection with race and gender diversity. arXie preprint Xiv: 2018.
17

---

Published in Transactions on Machine Learning Research (05/2025)
Yash Savani, Colin White, and Naveen Sundar Govindarajulu. Post-hoc methods for debiasing neural networks. Xiv preprint or arXix200060088 2020.
Harshay Shah, Sung Min Park, Andrew Ilyas, and Aleksander Madry. Modeldiff: A framework for comparing learning algorithms. In International Conference on Machine Learning, pp- 30646-3000 33646-30688. PMLRR 2023.
Mohammad Ahmad Sheikh, Amit Kumar Goel, and Tapas Kumar. An approach for prediction of loan approval using machine learning algorithm. In 2020 international conference on electronics and sustainable commmmication systems (ICESC), pp. 490 494. IEEE, 2020.
Andrew Silva, Pradyumna Tambwekar, and Matthew Gombolay. Towards a comprehensive understanding and accurate evaluation of societal biases in pre-trained transformers. In Proceedings of the 2021 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2389, 2021.
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequential rocommendation with bidircctional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management, pp. 1441 1450, 2019.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskover, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. aiv preprinl urXiv:1312.610 2013.
Yi Chern Tan and L Elisa Celis. Assessing social and intersectional biases in contextualized word represen- tations. Advances in neura informalion processing systems, 32, 2019.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. In Advances in Neural Information Processing Systems, 2020. URL https://prroceedings iew.roceedinggs.euiiiss..... wwt.:eneurrrscpppppeeeeee pew.ripp.cc/ppppr/220//f.... pdf.
Liwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, and John Hopcroft. Towards under- standing learning representations: To what extent do different neural networks learn the same represen- tation. Advances in neural information processing systems, 31, 2018.
Mei Wang and Weihong Deng. Mitigating bias in face recognition using skewness-awan reinforcement learning. In CVPR, pp. 9322 9331, 2020.
Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wet Chang, and Vicente Ordonez. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In 1CCV, pp. 5310 5319, 2019.
Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness in visual recognition: Effective strategies for bias mitigation, 2020.
Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: a survey. ACM Computing Surveys, 55(5)31337, 2022.
Di Xie, Jiang Xiong, and Shiliang Pu. All you need is beyond a good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6176-6185, 2017.
Depeng Xu, Shuhan Yuan, Lu Zhang, and Xintao Wu. Fairgan: Fairness-awart aware generative adversarial networks. In 2018 IEEE International Conference on Big Data (Big Data), pp. 570-575. IEEE, 2018.
Junliang Yu, Hongzhi Yin, Xin Xia, Tong Chen, Jundong Li, and Zi Huang. Self-supervised learning for recommender systems: A survey. IEEE Transactions Knowledge and Date Enginecring. 2023.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P. Gummadi. Fairness beyond disparate treatment & disparate impact. WWW, Apr 2017a. doi: 10.1145/303891 12.088991....2000 URL
http://dx doi http://dt..ddl.orr//0011445////1 wttp:///...../0099222.2300
18

---

Published in Transactions on Machine Learning Research (05/2025)
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez- Rodriz-o, and Krishna P. Gummadi. Fairness con- straints: Mechanisms for fair classification. In ATSTATS, volume 54, pp. 962-970. PMT.R, 2017b. URL
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. Fairness be- yond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In Proceedings of the 26th international conference world wide web, pp. 1171-1180, 2017c
Muhammad Bilal Zafar, Isabel Valera, Manuel Komez-Rodrigee and Krishna Gummadi. Fairness con- straints: A flexible approach for fair classification. JMLR, 20(75): 2019. URL http::/jmlr.orr org/ papers/v20/18-2655 21218-2222hmm
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. ICML, number 3, pp. 325-333, 17-19 Jun 2013. URL http://procceddings. waiggss..r...oo... wttp.//....eee//////eeea........ html
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 335-340, 2018.
Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang, Learning gender-neutral word embed- dings. In Conference on Empirical Methods in Natural Language Processing, pp. 4847 4853, 2018.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. Gender bias in contextualized word embeddings. ar preprint or XrXiv:1944.03311 2019.
19

---

Published in Transactions on Machine Learning Research (05/2025)
A Properties of PNKA
A.1 Overlap of neighbors
In Scction 2.1 we argue that for an input example to be similarly represented in two representations, its neighborhood should be similar across both of them. Here, we empirically show that this intuition applies to PNKA, and that if the PNKA score point is higher than that of j, then i's nearest neighbors in reprosentations and Z' overlap more than those of j. To show this, we train two models that only differ in their random initialization and compute their representations on the test set (10K instances). We the penultimate layer (i.e., the layer before logits) for the analysis. For each model, nearest neighbors by ranking point's representation distance (via either cosine similarity or Le distance) to every other point in that representation. We then compute the fraction of those two sets of lo neighbors that intersect.
determine point's k
the following plots we depicts the relationship between PNKA similarity scores (x-axis) and the fraction of overlapping & nearest neighbors of each point (y-axis), i.e., means all & nearest neighbors are shared between both representatious. We report the analysis on CIFAR-10 10 and CIFAR-100 Krizhevsky Krizhvvyykkizzeevskkyy al. (2009) for ResNct-18 He ct al. (2016), VGG-16 and Inception-V3, for different h values, up to k 20% of the dataset size. All the results are reported over 3 runs. In all cases of Section A.1 we see a clear relationship between high PNKA scores and high overlap of nearest neighbors across representations. indicating that PNKA captures how similar the neighborhoods of the points are. We further show in Figures g and | that the direct comparison of representations through cosine similarity does not exhibit the same trend, i.e., there is not a positive correlation between cosine similarity scores and the fraction of overlap of the k nearest neighbors.
20

---

Published in Transactions on Machine Learning Research (05/2025)
Fraztoo Overlap Neighbons
Fraction Dieriap Nfighton
(0.12)
10.20.1
(0.304)
10.40.51
(..30.0)
[6.60.77
10.7..81
(0.09)
10.8-101
103031
10.044)
10405)
1034611
(06.07)
10.70.
(08031
[c. 91
Average PNKA
Average
(a) ResNet-18, k50 (2.5% total points)
(b) ResNet-18 18, 2=2000 (20% total points)
Fraction Overlap of Neighbors
Fraction of Dueriap of Neightors
0.0
0.0
0.0
0.6
0.4
0.4
0.2
0.2
(6003)
1020.3)
(0.344)
10.a0.5)
(0.00.0)
(G.00
10.7
(0.09))
09)
10.0 1.01
10.9011
(01102)
1022031
10.3..4)
10400)
103
10.6
077.771
10a 1o)
Auerage
Average
(o) VGG-16, k=2 (2.5% total points)
(d) VGG-16, k0500 (5% total points)
Fracion of Overiap of Neighbos
Fracton di Overlap of Neighbors
0.0
10.00.1
(0102)
(0.003)
1033041
(0.405)
10.0001
10.0771
10 10.10
10.911001
cerae average PIKKA
(0300)
10 404
10s0 10.000)
to 1001
10s09)
11.00
Auerage PAKA
(e) VGG-16, 101000 (10% total points)
(I) VGG-16, k=2000 (20% total points)
Fraction Overlap Neightbrrs
Praction at Overiap Neighhors
10.0.11
[6:102]
10.2031
10.004)
(0405)
10.5
(0.60071
10 10..0
10.9:101
100021
10.20.3 (0.12)
10.3 10.3
(0402)
10.5...
104.071
(0.10))
10.8..91
10.911
Average ANKA
Average PNKA
(g) Inception- V3, 250 (2.5% total points)
(h) Inception- V3, k 500 (5% total points)
Fraction of Overtap of Neighbors
Fraction of Overlap of Neighbors
(0001]
10.1221
10.20 (0.30.) 10402)
31
(0.50.)
(066771
10 1.11
1000 2]
(0.30 (0.3903) (0.1022)
to 40.)
t0 (0
to reat
ta 91
91
Average
average
(i) Inception-Va. 101000 (10% lotal points)
(i) Inception- V3, k=2000 (20% total points)
Figure 5: PNKA (with linear kernel) captures the overlap of & nearest neighbors between two representa- tions, i.e., the higher PNKA scores, the higher the fraction of overlapping neighbors. Results are an average over 3 runs, each one containing two models trained on CIFAR-10 Krizhevsky et al. (2009) dataset with the same architecture but different random initialization.
21

---

Published in Transactions on Machine Learning Research (05/2025)
fractio of Neighbors
fradtion Overita Neightors
10.00.11
(0.10.2)
(0.20.3)
(0.0
0...0.31
0.780.8)
(0.8..))
(0.91.01
(0.00.1)
[0
31
10.39.4)
(0.0..)
(0.500.))
10.6..71
(0.70.8)
00.0...1
(0.8.0))
Average PNKA
Average PNKA
(a) ResNet-16 18, 250 (2.5% total points)
(b) ResNet- 18, k=500 (5% total points)
fraction Neighbors
frachonn of Overiop Neightorn
0.2
1000.11
(6.1011
(0.20.3)
10.20.4
(0.43))
(0.5..))
(0.00.71
(0.0
(0....9)
10.01.01
(c000.1)
(0.101]
(0200.1
(020.4)
(0.00.)
(0.00.))
(0.07111
(0.04)
(031.00 1 mm
PNKA
PNKA
(c) ResNet-18 18, 1=1000 (10% total points)
(d) ResNet- 18, k=2000 (20% total points)
Fraction Neighbos
Frattion Dxertap wnightorss
0.0
0.0
0.6
0.6
0.4
0.4
0.2
(o.n).)
(02022)
(0200.)
(0.844) (0.00.))
(0.5.6))
(ea00.)
7:088
(0.9..0) (0.899)
(an-)
10100
(0.2033) (0.244)
(040.5)
(0.50)4)
7.71
(070.8)
(e.0100 (o..9)))
Average
Average
(e) VGG-16, k=2 (2.5% total points)
(f) VGG-16, k000 (5% total points)
Fraction of Overiap of Neighbers
Fraction of Overlap of Neighbers
0.
0.8
0.4
10..g (.1.2))
(0303)
(0.04)
(0403)
(0.50.6)
(0.60.)
(0.708)
(.
t0. (..0..0))
(o.0.1)
10.122 (0203))
103004)
10.40.))
105 03)
10.00.7 (0.788
(0.8.9)
10.9 (..)..0)
Average PNKA
Average PNKA
(g) VGG-16, 1000 (10% total points)
(h) VGG-16, 2000 (20% total points)
Neighbors
Praction Dveritsp Neightors
10.0 8.11
[0.00.]] 10200.1
(0.10.4)
[0A03) (0.5..)
(6.8..)
(0.008)
0.)
(0.8.10)
(000.1)
10.30.31 (0100.))
10.3 90.3
(040))
(84.. 10.5..8)
(0.7..))
10.8
9.91
[2.9 LO)
Average PNKA
Average PNKA
(i) Inception- V3, 250 (2.5% total points)
(j) Inception V3, 5=500 (5% total points)
Faztion of Overiap of Neighbors
Fraction of Oper of Neightors
[0.01]]
(0.10.))
[0.20.3]
10.30.4)
(0007))
10.0 (0.
1.01
(0o01] (...2.
10.1022 (0.20.3)
(0.3..)
10.20.5)
(0.60.1 (0..00)
a 1.11
91
Awerage
average
(k) Inception- V3, 1000000 (10% total points)
(I) Inception V3, k=2000 (20% total points)
Figure 6: PNKA (with linear kernel) captures the overlap of & nearest neighbors between two representa- tions, i.e., the higher PNKA scores, the higher the fraction of overlapping neighbors. Results are an average
over runs, each one containing two models trained on CIFAR-100 Krizhevsky et al. (2009) dataset with the same architecture but different random initialization.
22

---

Published in Transactions on Machine Learning Research (05/2025)
fraction Neighbors
frattion Overita Neighbors
10.0..11
(0.10.2)
10.20.3)
(0.4
0...0.21
00.88
(0.0..))
(0.91.0)
(0.00.1)
[0
(0.20.31
10.3..4)
(0.0..)
(0.5
1o.a..71
(0.70.0)
10.8..21
(0.1.0))
Average PNKA
Average PNKA
(a) ResNet-18, 18, 250 (2.5% total points)
(b) ResNet- 18, k=500 (5% total points)
fraction Neighbors
frachoon of Overiap Neightors
0.2
1000.11
(6.101
(0220.3
10.20.4)
(0.03:)
(0.50.))
(0.80.7]
(0.00
(0.89.)
10.01.01
(0000.1)
(0102)
(020031
1020.4)
(0
(0300.))
(0.0711
(0.04)
(031.1 ( m
PNKA
PNKA
(c) ResNet- 18, 1=1000 (10% total points)
(d) ResNet- 18, k=2000 (20% total points)
Fraction Neighbos
Frattion Dxertap wnightorss
0.0
0.0
0.6
0.6
0.4
0.4
0.2
(o..1.))
(02.02)
(0200.)
(0.844)
(0.05))
(0....))
(G.00.7
(0.7.0.))
(0.9..0) (0.8.9)
(a.0..1)
(01.02)
(020033
(0.244)
(040.5)
(0550.4)
(0.00
(0.0.0) (0.0
Average PNKA
Average
(e) VGG-16, k=250 (2.5% total points)
(f) VGG-16, k000 (5% total points)
Fraction of Overiap of Neighbers
fraction of Overlap of Neighbers
0.
0.0
0.4
(0.00.1)
(0.1002)
(0.303)
(0204)
(0405)
(0.00.))
(0.70.) (0.60.)
10.9.1. (.
(..0..0))
(o00.1)
10.1021
(03044 (02003)
10.40.))
105 64)
1000771 (0.708)
(0.8..)
10.0.10 1.0)
Average PNKA
Average PNKA
(g) VGG-16, 1000 (10% total points)
(h) VGG-16, 200000 (20% total points)
Figure 7: PNKA (with RBF kernel) captures the overlap nearest neighbors between two representations.. i... the higher PNKA scores, the higher the fraction of overlapping neighbors. Results are an average over
runs, each one containing two models trained on CIFAR-10 Krizhevsky et al. (2009) dataset with the same architecture but different random initialization.
23

---

Published in Transactions on Machine Learning Research (05/2025)
fractio of Neighbors
frattion Overiia Neightors
10.00.11
(03.02)
(0.20.3
().0..)
(0
1...0.21
(0.70.))
(0.91.01 (0.0..))
(0000))
20.20.11
10 (0.3..4)
40.5)
to 0.
1.....11
(0.00.))
10 10.8..21
Average PNKA
Average PNKA
(a) ResNet-18. 18, 250 (2.5% total points)
(b) ResNet- 18, k=500 (5% total points)
fraction Neighbors
fractoon of Overiap Neightors
0.2
1000.11
(61101
(020..)
10.844)
(0
(0.80.1 (0..64)
(0.00))
10.01.01
(0000.1)
(0.221)
(0220.1
(02044)
(0.00.)
(0300.)
(0.87111
(0704)
t03100 t m
ANKA
PNKA
(c) ResNet-18 18, k=1000 (10% total points)
(d) ResNet- 18, k=2000 (20% total points)
Fraction Overtap Neighbos
Frattion Dxertap wnightorss
0.0
0.0
0.6
0.6
0.4
0.4
0.2
(o.0).)
(62.02)
(0.844 (0200.)
(0.05)) (0..66)
(ea0.)
70888
(0.01.0) (0.899)
(a.0..1)
(01.02))
(02003)
(0.944)
(040.)) (0504))
lea0.
10708
(0.0000 (0
a.71
Average
Average
(e) VGG-16, 1=100 (2.5% total points)
(f) VGG-16, k050 (5% total points)
Fraction of Overiap of Neighbers
fraction of Overlap of Neighters
0.0
0.8
0.4
(0.10.2)
(0203)
(0304)
(ca0s)
(0.00.0)
(0007)
10.70.)
te. (.
0.0)
(o00.1)
(0.102))
10.40.) 103004) 10200))
105 64)
1000.7 10.08)
(0.09.)
10.0 1.0)
Average PNKA
Average PNKA
(g) VGG-16, 1000 (10% total points)
(h) VGG-16, 20000 (20% total points)
Neighbors
fraction Overits Neightors
[01122]
102033
(0.1..))
(G403) (0.5..)
(6.8..1)
(0.00.)
f0.
0.9
10.9. (0.8.100
(000.1)
(01002))
10.3..31
103 (048))
12.3..8)
(8. (0..0.7)
(0.7..))
10.8 9.91
[0.9 LG)
Average PNKA
Average PNKA
(i) Inception- V3, 250 (2.5% total points)
(j) Inception V3, 5=500 (5% total points)
Faztion of Overiap of Neighbors
Fraction of Oper of Neightbrrs
[0.01]]
(011.2))
[0.2033)
10.20.4
(e403)
(0007)
10.00 (0.
91
1.0)
(0o01)
(0.12.)
(0.3..) (0.2003)
10.20.5
(
(0.60.7
10.0.0)
[
a 1.11
Awerage
average
(k) Inception- V3, 100000 (10% total points)
(I) Inception V3, k=2000 (20% total points)
Figure 8: Cosine similarity is not able to captures the overlap of A nearest neighbors between two repre- sentations ie., there is not positive correlation between cosine similarity scores and the fraction of overlap of the k nearest neighbors. Results are an average over 3 runs, each one containing two models trained on CIFAR-10 Krizhevsky et al. (2009) datasct with the same architecture but different random initialization.

---

Published in Transactions on Machine Learning Research (05/2025)
Factlin of Nepibas
Frestee of
140. (0:00. 10220
mear
1000.1 10.
(0440.4
Taara 10.00. lu40
lasee. sy
lasear 1a 10.t
pverage PNNKA
pverage FINA
(a) ResNet-18, 1=100 (1% total points)
(b) ResNet-18, 18, k=500 (5% total points)
Fredlin Of of
frestin Of 0 0
Men. P1210
m020111 manna
minn
maat
m000.. 1.11
M.e.a.
ot naat
ma
pverage
Average
(c) ResNet-18 18, k= 1000 (10% total points)
(d) ResNet- 18, k=2000 (20% total points)
freston dferrpor
Tosoaay monn maa
1000 M.kiaa ra200 1a.00 me oy
at nai
pverage
Average
(e) VGG-16, 1=100 (1% total points)
(f) VGG-16, k=500 (5% total points)
Rede Oferrr of Weglbos
0
R.2 Heensy Pusnaa
mese
mana
m0 m1
10:2 N.
Tox mean 21
raa
Nea m0.01. at
Average
(g) VGG-16, 1000 (10% total points)
(h) VGG-16, k=2000 (20% total points)
of Westaas
frestir of
(e031) Teat0 10220. 10 Teenn TaEAs
Teea 2e
10000. (0..8.0) (0.00. 103 (0403.) (a.raal Teea 1a20a faaa.
Toa
pverage PNKA
Average
(1) Inception- V3, k= (1% total points)
(j) Inception- V3, k=500 (5% total points)
Fresiar of Weglbos
Hnctar of of
1am 10.10.2 Tain 31
10. Teen Taxra
Tous ta.aa aa
Toan m1
10.0021 102 Tett of
Tox taen 21
Tean
parch.
(k) Inception- V3, == 1000 (10% total points)
(I) Inception- V3, 2==000 (20% total points)
Figure 9: Cosine similarity is not able captures the overlap of nearest neighbors between two repre- scntations i.., there is not positive correlation between cosine similarity scores and the fraction of overlap of the k nearest neighbors. Results are an average over 3 runs, each one containing two models trained on CIFAR-100 Krizhevsky et al. 2009) dataset with the same architecture but different random initialization.
25

---

Published in Transactions on Machine Learning Research (05/2025)
A.2 Relationship of PNKA with aggregate measures of representation similarity
Datasel
Model
CKA
PNKA
ResNet-18
0.925
(#0.005)
0.925 (#0.002)
CIFAR-10
VGG-16
0.895
(+0.013)
0.893
(+0.039)
Inception-v3
0.916
(#0.001)
0.915
(#0.023)
ResNet-18
0.741
(#0.00)
0.733
(+0.083)
CIFAR-100
VGG-16
0.658
(+0.010)
0.668
(0.004
Inception-v3
0.798
(+00009)
0.792
(+0.032)
Comparison between CKA Kornblith et al. (019 and the aggregate version of PNKA (PNKA). Results are an average over 3 runs, each one with two models that only differ in their random initialization. We capture the representations of the penultimate layer (i.c., the layer before logits) for the analysis. We show that both measures produce similar overall scores.
Table 3:
A.3 Proof of invariances
A.3.1 Invariance to orthogonal transformations
Proof. Given orthogonal matrix Q, it suffices to show that
K(ZQ) TQ(2Q)
= ZQQ ZT
-QQQ 'TT
=Z
K(2)
Here we have used that for an orthogonal matrix Q, QT = Q By substituting K(ZQ) and K(Z'R) in PNKA(ZQ, R, i) (Zs(K ()) with K(Z) and K(Z),, respectively, we obtain PNKA(ZQ, i, PZ i), Thus, PNKA invariant orthogonal transformations.
A.3.2 Invariance to isotropic scaling
Proof. Note that because of the bilinearity of the dot-produch we have K K(aZ); = [(aZ) a2)(2), By substituting into PNKA, we get
K(aZ))
PNKA(aZ,82) (2,',,,
a (2) (2K
a² (²K( 3²K(Z')
PNNKAZZ, Z',
Thus, PNKA is invariant to isotropic scaling.
26

---

Published in Transactions on Machine Learning Research (05/2025)
B Investigating Debiased Tabular Data Representations
B.1 Results with RBF kernel
CKA-0.52
Number Poins
Number Point
300
250
230
200
200
130
Number
100
PNKA
(a) U U-CF (COMPAS, race).
(b) U U-IF (COMPAS, race).
Uc) U-CF-IF (COMPAS, race).
4000
3000
CKA 0.59
4000
CKA...C D.53
Number Poims
33000
Number Points
2500
3500
2000
3000
2300
2509
2000
1500
numbe:
2000
1500
1000
(d) U U-GF (Adult, gender).
U U-IF (Adult, gender).
(f) U U-GF-IF (Adult, gender).
Figure 10: Distribution of PÑKA similarity scores with RBF kernel. The first row (blue plots) shows results for the COMPAS dataset, debiased with respect to race, while the second row (green plots) displays results for the Adult dataset, debiased based on gender. The vertical dotted line shows the overall similarity scores provided by CKA Kornblith et al. (2019). We compare the representations obtained from the baseline (U, only utility) with each the three other debiased options: (1) utility, group and individual fairness (C-GF- TF), utility and group fairness (U-GF), and utility and individual fairness (U-IF).
Preportion
Regortion
0.0
0.8
0.2
CAucatine
Fensaee
Caucation
Femate
Categories
Categories
Categories
(a. U U-GF (COMPAS, race).
(b) U U-IF (COMPAS, race).
(c) U x U-GE-IF (COMPAS, race).
0.2
0.8
0.5
Proportion
Proportion
0.4
0.4
0.4
Semme
Cauratine
Femate
Caurcasion
Female
Eaurasan
Categories
Categories
Categorias
(d) U U-GF (Adult, gender).
(e) U x U-IF (Adult, gender).
(f) U U-GF-IF (Adult, gender).
Distribution of the binary attributes for the 10% most affected individuals (ie., lowest PNKA score) using the RBF kernel. The first row (blue plots) shows results for the COMPAS dataset, debiased with respect to race, while the second row (green plots) displays results for the Adult dataset, debiased based on gender. We compare the data representations of the baseline (U, only utility) with each of the three other debiased options: (1) utility, group and individual fairness (U-GF-IF), utility and group fairness (U-GF), and utility and individual fairness (U-IF).
Figure 11:
27

---

Published in Transactions on Machine Learning Research (05/2025)
B.2 Distribution of attributes for the data points whose representation changed the most
In this section, present graphical illustrations of the distribution of all the altributes within the COMPAS and Adult datasets using series of concentric rings. Each ring functions as a pie chart, depicting the distribution specific attribute. For instance, the innermost two rings represent the gender (male/femall) and racial (Caucasian/ wwa.a/onnoonCniaannnn Caucasian) distributions both datasets. The subsequent rings are tailored to each dataset. For the COMPAS dataset, the third, fourth, and fifth outer rings display the distributions of age categories (under 25, 25-45, over 45 years old), prior counts (0, 1-3, more than 3), and type of charge (felony misdemeanor). respectively. For the Adult dataset, the third and fourth outer rings showcase the age categories (grouped by decades) and education levels. Additionally, the first plot of Figure 12 and Figure 13 located in the top left, illustrates the overall distribution of these attributes across the entire dataset. The subsequent plots provide a detailed view of the attribute distributions for the bottom 10% of data points, which exhibited the most significant changes in their representation.
Khing to
Coucaharr
GRIVES
Mi 25
to
ztom, Segnent
(a) Distribution of dataset (baseline).
(b) U U-GF-IF.
Cahare stameea
Coupp Harreeo
counte
more mootte
safer'
to
No
Speny
Cesratio
Carvlaah than
Dugge
25 Seree
chnrg Sagann
U U-GF.
(d) U U-IF
Figure Distribution of all the attributes of the 10% of instances with the lowest PNKA scores for COMPAS dataset with the protected attribute as race.
28

---

Published in Transactions on Machine Learning Research (05/2025)
Eoucador m=12
(docade)
toocalee
Agp denoriiii
Merromee ote
Glocaran
Age GOe Lpeuue
Cassaian
Holf
Rah
Fidco Cociooi Mori 20
CHICEVOn
Age decallt
Educaban Baar
(a) Distribution of dataset (baseline).
(b) U U-GE-IF
Eliirtion
Ap decaneeerg
Age
Eociaiinn TorLd
Age odecadal- 30
Age 1nor
Edyciion 6a0x-10
U U-GE.
(d) U U-IF
Figure 13: Distribution of all the attributes of the 10% of instances with the lowest PNKA scores for Adult dataset with the protected attribute gender.
29

---

Published in Transactions on Machine Learning Research (05/2025)
C Investigating Debiased Word Embeddings
C.1
PNKA results with RBF kernel
Pf Points
Percentage of Points
80
80
of of
80
Perceraage
40
40
40
0.4
0.5
0.0
PNKK
PAKCA
PNKKA
(a) GloVe x GP-GloVe.
(b) GloVe x GN-GloVe.
(c) GloVe GP-GN-GloVe.
Figure 14: Distribution of PNKA scores (with RBF kernel) per group of words for SemBias dataset Zhao et al. 2018). We compare the baseline (GloVe) model and its debiased versions. Words with the lowest similarity scores are the ones that change the most from the baseline to its debiased version. Across all debiased embeddings, the words whose embeddings change the most are the gender-dcfinttin words.
C.2 Evaluation of Debiasing Methods on SemBias dataset
For cach of the four word pairs (a, b) in a SemBias instance, GP- and GN-Glove mcasure its cosine similarity with the canonical gender vector, i.c., cos( d - D he shé). The word pair with the highest cosine similarity is selected as the "predicted" answer. If the word embeddings are correctly debiased, then the cosine similarity of the he she vector with the gender-definition words should be high, and the similarity with the gender- stereotype words should be low, i.e., the frequency of predictions for these categories should be high and low, respectively. Table 4 depicts the results for the GN- and GP-Glove (Zhao et al. 2018 Kaneko & Bollegala, 2019h methods.
SemBias
Embeddings
Definition
Stereotype 4
Neutral
GloVe
80.2
10.9
8.9
GN--loVe
97.7
1.4
0.9
GP-GloVe
84.3
8.0
7.7
GP-GN-GloVe 98.4
1.1
0.5
Table 4: Frequency of predictions for gender relational analogies Kaneko & Bollegala 2019). Each column shows the frequency with which the respective word-pair category (gender-definitionna, gender-stereotype, gender-neutral) is predicted as having the highest cosine similarity with the canonical gender vector he she. The more often gender-defiition words are predicted as being most gender-aligned, opposed to gender-stereotype words, the less biased an embedding approach can be considered.
30

---

Published in Transactions on Machine Learning Research (05/2025)
D Investigating Debiased Contextual Embeddings
Transformer-basee language models are currently the leading approach for many NLP lasks. In contrast to the previously discussed word embeddings, these models use contextualized embeddings where the rep- rosentation of word token also depends the preceding and possibly succoeding tokens. However, just like their non-contextuullzze predecessors, these models have also been found to embed several unfair biases (Zhao et al. 2019 Tan & Celis 2019 Kurita et al. 2019). The predominant approach to using transformer-based (large) language models is to adapt pre-trained models to downstream tasks, either with or without fino-tuning. Any biascs that the pre-trained base-models cxhibit might thus be inherited by the downstream lasks. Since a small number of base-models are adapted for many different downst ream lasks, it is infeasible to audit models on each of those tasks separately. Rather, analyzing biases in the representations that base models use provides a scalable way to study their fairness.
Given the importance of addressing these biases for transformer- based language models, researchers have started proposing methods aimed to debias the contextualized embeddings of these models. One method by Kannko & Bollegala la (2021) proposes to remove a potential gender bias by fine-tuning a contextual baseline model to "preserve semantic information with respect to sentences with sensitive attributes (e.g., 'she', 'he"), while removing any discriminatory biases with respect to sentences with stereotypical words (e.g., "poetry', "math")". In other words, the goal of this method is to remove gender bias from the representations of sentences containing potentially stereotypical words, such ones related to poetry (with a presumed female bias) and ones related to math (with presumed male bias).
We leverage PNKA to investigate whether this debiasing method changes the representations of sentences in the intended manner. For our analysis, we use the Albert ('albert-base-v2") model as baseline, and evaluate it and its debiased versions on the same dataset used to study the gender bias in the original work of Kaneko & Bollegala| (2021), specifically the SEAT-7 and SEAT-8 datasots (May et al.] 2019). These datasets are composed of simple sentence templates such as "This is a [BLANK]", and create gender defining and gender stereotypical sentences by substituting " "[BLANK]" with gender defining (e.g., 'she', 'he') and target stereotypical words (e.g., 'poetry', 'math'), respectively. The specific words were chosen based on the WEAT measure {Caliskan et al. 2017 which measures the associations between concepts like math/ art and science/art with male/female female attributes in SEAT-7 and SEAT-8, respectively. A more detailed explanation on WEAT and SEAT is provided in Appendix D.1 and D.2.
Figure 15 shows the distribution of PNKA similarity scores for the two categories of sentences in SEAT- 7 (Figure 15a) and SEAT-8 (Figure 15b,, respectively. In both cases, we observe an overall high PNKA score, with more than 80% of points obtaining PNKA scores higher equal to 0.8, and CKA of 0.88. This suggests that most of the instances of both gender defining as well as gender stereolypical sentence categories have not drastically changed from the baseline to the debiased model. More surprisingly, there is no clear distinction between the two groups of sentences, i.e., both gender defining (e.g., "This is a woman." or "This is a man.") and gender stereotypical (c.g., "This is poetry." or "This is math.") sentences change in a similar proportion.
To investigate whether the high representation similarity indicates a lack of change in the gender properties of the sentence, we follow the procedure used for the non-contextual embeddings described in Section Again, we project the sentence representations onto the gender vector he in projection magnitude from the baseline to the debiased version. We follow Kaneko & Bollegala (2021) and obtain representations of the [CLS] token at the last layer. Figure 16 displays the relationship between
+ - she and measure the change
PNKA scores and the percentage difference w/ (baseline) for the baseline ("albert-base-v2) as well as w/ (debiased) for debiased model. In accordance with the high PNKA scores, we observe that all the context ual embeddings exhibit a minimal shift along the gender direction. Once again, PNKA is a reliable indicator that the debiasing technique not effective in the intended way, and that the bias of gender stercotypical sentences is not reduced as expected. Our insights complement previous work, which also identified that using SEAT alone is insufficient to evaluate debiasing methods (Meade et al. 2021 Silva et al. 2021 May et al. 2019). Future work could explore the effects of this method further to understand why it does not significantly alter the representations in the intended way.
Tn Appendix D.3 also explore using the average contextual gender vector and observe similar pattern.
31

---

Published in Transactions on Machine Learning Research (05/2025)
CKA=0.88
CKA=0.87
100
100
Percentage of Points
Gender Defining
Percentage Points
80
Gender Stereotypical
80
60
60
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
PNKA
PNKA
(a) SEAT-7.
(b) SEAT-8.
Figure 15: Distribution of PNKA scores (linear kernel) group of sentences in SEAT-7 and SEAT-8 dataset May et al. 2019). We compare the baseline ("albert-base-v2) model and its debiased version (Kaneko & Bollegala, 2021p. Sentences with the lowest similarity scores are the ones that change the most from the bascline to the debiased version. Across all the datasets, we observe that most of the sentences maintain high PNKA scores, which indicates that they have not substantially changed their representations. Moreover, there is no clear difference between the groups of gender defining and gender stereotypical sentences in how they change their representations.
Percentage Difference
Gender Defining
Percentage Difference
Gender Stereotypical
2
0.2
0.6
0.2
0.4
0.6
0.8
PNKA
PNKA
(a) SEAT-7.
(b) SEAT-8.
Figure 16: Relationship between PNKА scores (x-axis) and percentage difference (y-axis) in magnitude of the projection on the gender direction he she. A positive or negative percentage difference value indicates shift in magnitude along the gender direction. Overall, the contextual embeddings exhibit low shift along the gender direction, with no clear distinction between groups of sentences.
D.1 Word Embedding Association Test (WEAT)
The description below provided by Li et al. (023).. Word Embedding Association Test (WEAT) (Caliskan et al. 2017) measures the association between sets of attributes words (e.g., male and female) and two sets of targets words (e.g., family and career). Formally, the sets of attribute words are indicated by A and B, and the sets of target words are denoted by X and Y. Then, the WEAT test is as follows:
s (A,, (A,
Σ ((x,A, xeX
Σ s(x, A, B), DEY
(2)
32

---

Published in Transactions on Machine Learning Research (05/2025)
CKA=0.88
CKA=0.87
100
100
Percentage of Points
Gender Defining
Percentage of Points
80
Gender Stereotypical
80
60
60
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
PNKA
PNKA
(a) SEAT-7.
(b) SEAT-8.
Distribution of PNKA scores (RBF kernel) per group of sentences in SEAT-7 and SEAT-8 dataset May et al. (019). We compare the baseline ('albert-base-v2) model and its debiased version (Kaneko & Bollegala, 2021p. Sentences with the lowest similarity scores are the ones that change the most from the bascline to the debiased version. Across all the datasets, we observe that most of the sentences maintain high PNKA scores, which indicates that they have not substantially changed their representations. Moreover, there is no clear difference between the groups of gender defining and gender stereotypical sentences in how they change their representations.
Figure 17:
where s(w, A, B) represents the difference between the average of the cosine similarity of word w with all words in A and the average of the cosine similarity of word w to all words in B, and it defined as follows:
sw, X,Y)
1 A
Σ aEA
cosine(w,o) a)
1
cosine(w, b),
(3)
B
bE
where X or Y, and cosine(,,) represents the cosine similarity. The normalized effect size is as follows:
d (((s(1, A, B)exx)) p(ts(y, w)x)) affs(t, X, (EEABB))
(4)
where ((-) the mean function and o()) is the standard deviation.
D.2 Sentence Embedding Association Test (SEAT)
Sentence Embedding Association Test (SEAT) (May et al. 2019 adapts WEAT contextual embeddings,
which uses simple sentence templates such as "This is a BLANK TM substitute attribute words and target words to obtain context-indlependent embeddings. Then the SEAT test. statistic between the two sets of embeddings (represented by the '[CLS]' of the last layer) is calculated similarly to Equation 4
33

---

Published in Transactions on Machine Learning Research (05/2025)
D.3 Results of Projection with Average Contextual Gender Direction
To invesligate whether the high represenlation similarity indicates a lack of change in the gender properlies of the sentence, we follow the procedure used for the non-contextual embeddings described in Sections 1 However, in this case, instead of just using the gender directions of the words he and she, we use a gender direction from an aggregated embeddings of male and female sentences. In other words, we project the sentence representations of the SEAT dataset onto the average contextual gender vector originally used by Kaneko & Bollegala| (2021) to debias the model. This average contextual gender vector is obtained follows. First, obtain an average representation for the sentences S(w) where the gendered-word appears. This gendered-we word belongs to an attribute A E A, where A [A,, Am), and A denotes a set of words associated with the corresponding gender, where f stands for female and m stands for male.
ca(w)=
Σ c(s).
(5)
[S(w] *ES(m)
Next, we take the final average contextual gender vector for attribute A, which can be As for female and Am for male, as follows:
ε = A
Σ ca(w). WE
(6)
Thus, the projection will be computed using the average contextual gender direction eA... CAy"
Figure 18 displays the relationship between PNKA scores and the percentage difference for the bascline ('albert-bases well as for debiased model according to the average contextual gendered direction. As before, and in accordance with the high PNKA scores, we observe that most of the contextual embeddings exhibit minimal shift along the gender direction.
Percentage Difference
Gender Dafining
Percentage Difference
Gender Stereotypical
-20
-40
-40
-60
-60
PNKA
PNKA
(a) SEAT-7.
(b) SEAT-8.
Figure 18: Relationship between PNKA scores (x-axis) and percentage difference (y-axis) in magnitude of the projection on the average gendered direction. positive or negative percentage difference value indicates shift in magnitude along the gender direction. Overall, the context tual embeddings exhibit a low shift along the gender direction, with no clear distinction between groups of sentences.
34


Reviews:
[{"review_id": "dur33JeZWB", "reviewer": "Reviewer_GnRZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the paper resides in its clear and well plan out investigation. Indeed, the paper is clearly written and well motivated. The investigation is well thought out and the experiments are well executed. I think the overall question of how individual representations of data points are impacted by debiasing methods is somewhat interesting with the application to understanding how models trained on these representations will perform.\n\nIn terms of weaknesses, I have 3 main concerns:\n1) While the authors have shown how looking at the changes in representations can inform downstream performance, it is unclear what novel insight this provides. For example, in the first example of group and individual fairness for tabular data, the insights around how group and individual fairness technique impact individual representations are not surprising. Indeed, it is expected that methods to satisfy individual fairness would lead to most changes in the representation of most data points compared to group fairness technique which would mostly impact the subset of points that lead to group disparities. Similarly for the insights on using the similarity score to understand performance, the results are not surprised. It can be expected that the debiasing method will increase utility for the minority group. \n\n2) Proof for using relative position as the similarity metric: while the authors provide an intuitive understanding of why they use the relative position of the data point as the similarity metric between the representation before and after debiasing, it is unclear why this is the right choice of metric. The second paragraph of section 2.1 should be expanded to clearly address why this is a correct measure of similarity. \n\n3) Limited methods: the experiments mainly look at one method for each of the case studies. For each case, it is unclear how the insights generalize to other debiasing methods. For example, the experiments on group and individual fairness only look at Zemel et al 2013 which is only an example of method for learning fair representations. Looking at a broader set of fair representation learning methods and individual fairness methods can help assess the validity the insights provided.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to study the effect of debiasing methods on the intermediate representations of individual training data points. \n\n1) They use a measure of representation similarity to assess the impact of fairness procedures on individual data points before and after debiasing. The similarity measure used is based on the relative position of the data points compared to the other training points within the same representation space. Specifically, an individual data point's representations before and after debiasing of the model are deemed similar if the data point maintain the same position relative to the other training points before and after debiasing. \n\n2) The author use this metric to study the overall individual effect of a group fairness and individual fairness procedure proposed by Zemel et al 2013 and debiasing of word embeddings as proposed by Kaneko and Bollegala 2019. By examining the distribution of similarity scores before and after debiasing as well as the distribution of data points with the lowest scores (indicating the most impacted by the debiasing procedure), they provide several insights into how the debiasing methods studied affect the representation of individual data points. \n\n3) Finally, the authors propose using the results of the similarity measure to understand performance on subsequent tasks using the representations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Request:\n- Expand the discussion on the validity of using position as a proxy for similarity\n\nMinor changes:\n- In section 3.1, the description of figures 1c and 1e is confusing. Is it the other way around?: the pattern for the combination of group fairness and individual fairness for COMPAS resembles that of individual fairness alone and for Adult, it mirrors the distribution when enforcing group fairness only\n- Add the legend for the red dotted line on the caption", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mxtAvldvTu", "reviewer": "Reviewer_twWV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: demonstrating the usefulness of individual representation similarity measures for more fine-grained analysis. \n\nWeaknesses: The method itself does not seem different from previous methods. In fact, it seems that every previous aggregate representation method is averaging over an implicit individual similarity measure. \n\nThe kernel used to compute the similarity measure for the empirical results is unclear. It is also unclear what happens if a different kernel is chosen. And if the choice of the kernel changes outcomes significantly, it is unclear how to appropriately choose the kernel.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a tool for evaluating the similarity of two representations based on a point. That is how similarly both representations represent a point. The method is inspired by previous work that computes an aggregate similarity measure of two representations. The paper empirically shows, across some datasets, that their proposed method satisfies some nice properties such as having extent of intersection of k-nearest neighbors correspond positively with similarity in representation, and invariance to some transformations. \n\nThe paper shows applications of using individual, point-based similarity measures that can’t \nbe solved using aggregate similarity measures. The applications shown in the paper are: identifying subgroups that change through debiasing techniques on representations, predicting downstream consequences of representations debiasing, and auditing debiasing techniques for word embeddings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "To strengthen the paper: \n\nProvide what kernel was used and analyze the effects of using other kernels.\n\nComparison with other individual similarity metrics that are induced by average similarity measure.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "a23irXdMOs", "reviewer": "Reviewer_tRz9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is overall well-written and sound. The authors provide a clear rationale for introducing PNKA and effectively connect it to existing fairness challenges, helping readers understand the relevance and significance of the proposed method. The related work is discussed in detail, and the experiments are clearly explained. Overall, this submission is accessible to a broad audience. Furthermore, the problem studied in this submission is relevant, and of high interest for the ML community. The proposed metric, which measures similarity on the representation space, is a novel approach, and the experiment on language embeddings are particularly interesting. My only concern is the lack of a technical contribution in this work, since the main contribution is just Eq. (1).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission addresses the limitations of existing fairness evaluation measures for machine learning (ML) models, particularly their lack of granularity and task generalizability. The authors propose Pointwise Normalized Kernel Alignment (PNKA), a novel metric that analyzes how fairness interventions impact the intermediate representations of individual data points, offering insights into both group-level and individual-level fairness. Unlike conventional evaluation methods, PNKA allows for a fine-grained assessment by measuring representational similarity at the individual level. The study reveals that group fairness interventions often affect only a small subset of individuals while maintaining representational similarity for most of the population, whereas individual fairness constraints lead to broader, more uniform changes across data points. Additionally, PNKA is shown to predict downstream task behavior accurately and highlights unintended shortcomings of existing debiasing methods in language embeddings, such as their failure to effectively remove biases from stereotypical words and sentences. These findings emphasize the inadequacy of current fairness evaluation measures and underscore the utility of pointwise representational similarity metrics, such as PNKA, for deeper fairness audits across a range of ML applications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have no changes to request, since I found this submission overall well-written.", "broader_impact_concerns": "I do not have any concerns regarding ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers offered mixed recommendations, with some suggesting acceptance and others rejection. The main reason for rejection is the 'limited novelty and lack of a strong technical contribution.' However, all reviewers responded positively to the two acceptance criteria: claim and evidence, and audience. The authors have also provided additional analysis using the RBF kernel and found that the conclusions remain consistent. Please ensure the correct citation style is used for the camera-ready version.
Decision: Accept (poster)