Title: Simple and Effective Specialized Representations for Fair Classifiers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Fairness

Invariant Representations

Characteristic Function

Neural Networks
Abstract: Fair classification is a critical challenge that has gained increasing importance due to international regulations and its growing use in high-stakes decision-making settings.
Existing methods often rely on adversarial learning or distribution matching across sensitive groups; however, adversarial learning can be unstable, and distribution matching can be computationally intensive.
To address these limitations, we propose a novel approach based on the characteristic function distance. Our method ensures that the learned representation contains minimal sensitive information while maintaining high effectiveness for downstream tasks. 
By utilizing characteristic functions, we achieve a more stable and efficient solution compared to traditional methods.
Additionally, we introduce a simple relaxation of the objective function that guarantees fairness in common classification models with no performance degradation. 
Experimental results on benchmark datasets demonstrate that our approach consistently matches or achieves better fairness and predictive accuracy than existing methods. 
Moreover, our method maintains robustness and computational efficiency, making it a practical solution for real-world applications.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a representation learning method that overcomes the unstable issues of adversarial learning and computationally intensive issue of distribution matching. The method first uses CFD to align group-wise representations to a common distribution, and then proposes a more efficient moment-matching approximation that matches low-order moments (N = 2) to guarantee that downstream classifiers cannot access sensitive information.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The proposed method replaces the need to match full distributions (via monte carlo CFD estimation) with a tractable alternative by truncating the characteristic function expansion to a low order (e.g., N = 2), which leads to more stable and efficient training compared to monte carlo based methods.\n- The method uses CFD to align group-wise representations at the distributional level, offering a clearer and more principled formulation of fairness compared to adversarial learning or MMD-based approaches.\n- The visualization of learned representations in Figure 4 is very intuitive to capture the effectiveness of the proposed method.\n\nWeakness:\n- The proposed method aligns representations of each group to a shared target distribution, and in this paper, uses the standard normal distribution (likely for computational simplicity and methodological clarity). However, consider a situation where the target distribution is not well-aligned with any group (e.g., when the data resemble a mixture of Gaussians) and enforcing all groups to align with the shared target may actually distort them unnecessarily.\n- One major limitation of the paper lies in its limited experimental scope. The method is only evaluated on two small tabular datasets, which are low-dimensional and structurally simple. In such cases, aligning low-order moments (N=2) may already suffice for fair representation learning. However, it remains unclear whether the proposed approach can generalize to more complex scenarios, such as high-dimensional, nonlinear, or multi-modal tasks, where group-wise distributional differences are more intricate. This limits the method’s relevance to current practice and raises concerns about its applicability in more realistic, large-scale, or multi-modal learning settings.\n\nQUESTIONS:\n- How sensitive is the method to the choice of the shared target distribution?\n- Have the authors considered whether aligning group-wise representations to a learned barycenter or a task-specific distribution might be more effective than enforcing a shared target distribution, such as the Gaussian used in this paper?\n- Line 230: “…however, we show that truncating to N=2 is sufficient to ensure fair classification,” but I cannot find the evidence to support this claim. Can the authors specify and justify it?\n\nLIMITATIONS:\nYes, the authors acknowledge the limitation regarding high-dimensional settings in the conclusion section. However, I respectfully disagree with the claim that \"current fairness benchmarks largely focus on low-dimensional datasets, limiting the ability to comprehensively evaluate and compare methods in more realistic and complex settings.\" In the domain of computer vision, for instance, there exist several well-established image datasets for fairness evaluation, such as CelebA, FairFace, and LFW.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a fair representation approach. It minimizes the characteristic function distance of the different demographic groups' representation and a Gaussian distribution in the latent space. To keep the prediction power, a cross-entropy loss is used. The author also shows some theoretical insights into their methods. In the experiments section, it compares with Fair Normalizing flow methods, which is also a fair guarantee method. The results show that the method proposed by the author achieved a good tradeoff between fairness and utility.\n\nSTRENGTHS AND WEAKNESSES:\n**Stengths**\n\n(1) This paper is well-presented; it introduces the prerequisites in detail, and the notation is neatly arranged.\n \n(2) The algorithm is clear and easy to reproduce. I personally think this algorithm is correct.\n\n(3) It is indeed more flexible than FNF, as it can be used as any backbone model, instead of just flow flow-based model. \n\n**Weakness**\n\n(1) The major concern I have is the tradeoff between utility and fairness. As in Algorithm 1 line 11, different demographic groups' representations are aligned with the same distribution (Gaussian in the text). Although a cross-entropy loss is added to guide the learning procedure, the fairness constraint may overwhelmingly dominate the learning procedure, resulting in less effective performance of the classification task. \n\n(2)  There exist many fair representation learning methods. For example, MMD-B Fair [1] uses MMD as the statistical independence criterion to learn a fair representation. I am wondering why the author's proposed method is more needed than existing fair representation works. A rigorous experiment comparison would help to illustrate that.\n\n(3) Datasets are narrowed in tabular datasets, it narrows the scope of this work.\n\n\n[1]Deka, Namrata, and Danica J. Sutherland. \"Mmd-b-fair: Learning fair representations with statistical testing.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2023.\n\nQUESTIONS:\n1. Why do the authors select a Gaussian distribution as the reference? While I understand it is discussed in the context of stability training, I am still unclear why aligning with different demographic groups would be considered a suboptimal choice\n\nI don't have further questions. Please see the weakness section for my major concerns.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe area of fair representation learning (FRL) seeks to learn a hidden representation which “looks the same” with respect to multiple different sensitive attributes. That is, for example, we would like to learn embeddings of individuals such that if a downstream user had access to our embedding for a particular input x, they would not be able to determine (with good accuracy) whether the user can predict what sensitive attribute x belongs to. This is what the current paper calls adversarial evaluation.\n\nIn addition, FRL methods may fall into two categories: “specialized” or “generalized”. Specialized methods learn representations useful for a particular task or classification problem, while generalized methods seek to learn representations which may be used by any downstream learner on any task, and should allow for fairness guarantees to be inherited.\n\nFair normalizing flows (FNF) is positioned as a state-of-the-art method which achieves this “generalized” FRL. However, the submitted paper claims FNF has various flaws. For example, it requires training a separate model for each group, and also requires the protected subgroup at inference time.\n\nThe paper instead introduces a FRL method based on characteristic functions. Define $P_Z$ as the marginal distribution over the learned latent space. The proposed method, FmCF, seeks to match the conditional distributions for $P_{Z|S}$ for all sensitive attribute values $S$. Intuitively, if the latent distributions are indistinguishable, then the FRL problem is solved. FmCF does this via matching the characteristic functions of the distributions $P_{Z|S}$. Importantly, characteristic functions and probability distributions map one-to-one. Therefore, if we have efficient methods for matching the characteristic functions, then we can solve FRL (equation 7).\n\nWe would like to regularize loss minimization in order to penalize deviation from equal characteristic functions of $P_{Z|S}$. Instead of penalizing this for all pairs of sensitive attributes, the paper suggests instead penalizing the distance of each characteristic function of $P_{Z|S}$ to the characteristic function of the normal distribution $\\mathcal{N}$. This can be done with a montecarlo approach.\n\nIn Section 5, the paper expands on FmCF by introducing a refinement for classification, FmSS. Instead of matching the entire characteristic functions for $P_{Z|S}$, it turns out that for classification with logistic regression, it suffices to match only the first moments $\\mathbb{E}[Z | S]$ in order to achieve FRL. This is possible due to a fact relating characteristic functions and the moments of the distributions they represent (Theorem 5.1). In fact, a stronger result is obtained for the predictive power of a feature $Z_i$ in obtaining the sensitive attribute $S$ in Theorem 5.3. In particular, matching the mean and variance of $P_{Z|S}$ for all $S$ is sufficient to restrict the predictive power of all features.\n\nFmSS utilizes this result and works by minimizing the KL divergence between the standard normal and the empirical conditional distributions $P_{Z|S}$.\n\nEmpirical experiments are conducted in section 6, where the paper demonstrates that FmCF and FmSS both preserve standard accuracy of the original task (relative to existing FRL methods), and decrease the adversarial accuracy of a predictor attempting to infer the sensitive attributes $S$ from the hidden representations $Z$.\n\nSTRENGTHS AND WEAKNESSES:\nI know nothing about fair representation learning and am unfamiliar with most (if not all) of the papers cited by this paper. Hence, my confidence is low.\n\nStrengths\n1. The characteristic function matching approach is novel for FRL (to my knowledge) and seems to really simplify the optimization problem in a convenient manner. The proposed methods FmCF and FmSS also require no architectural adjustments, which seem to be quite prevalent in previous works such as variational fair autoencoders and fair normalizing flows.\n2. The fact that one can only care about matching the mean and variance of the conditional distributions $P_{Z|S}$ for classification is quite surprising and seems to simplify the computation of the regularized loss by quite a bit.\n\nMinor Weaknesses:\n1. Are low latent dimensions like 1,2,3 standard for the FRL literature (line 652)? This seems far lower than one would expect to, for example, learn a sigmoid function on. The final hidden layer of NNs is usually much larger. Is this because of computational constraints? Using such low dimensions for the latent representations to me detracts from strength 2. In particular, if we are in only two or three dimensions, it’s not clear that the higher order moments really have anything interesting to say.\n2. Some of the details in table 1 were unclear. For example, was $\\mathcal{G}$ defined earlier? Also, what I don’t think $\\mathcal{F}$ is defined? I could be missing something, however.\n3. Missing discussion of runtime of FmCF and FmSS vs. standard methods like training fair normalizing flows or VAEs.\n4. It seems like FmCF has a runtime exponential in the dimension of the latent space. However, FmSS seems much more efficient, since we are only matching the mean and variance. \n\nThe paper seems very well motivated and presents what seems like a much simpler approach for achieving FRL, and hence, I believe that it will be a valuable contribution.\n\nOther comments:\n1. As someone outside of FRL, I was a bit confused about the distinction between generalized and specialized FRL methods. I think providing some examples of models or tasks in lines 30-36 could improve clarity.\n2. The paper shows that FmCF extends to settings other than classification in Appendix A.5. To my knowledge, this experiment is not explicitly highlighted in Section 4 or anywhere else in the paper. You should mention it because it is an interesting experiment!\n3. Typo equation (2), parens in subscript.\n4. Consider including a down arrow for adversarial accuracy, to show the reader at a glance that it should be minimized.\n5. No legend for figures 5-8 in appendix, although explained in line 652, could be clearer if you included them in the figure descriptions or as a legend in the plots.\n6. Line 240/241: Maybe remind the reader what the $\\beta$ coefficient is again, as you do in the description of Figure 2.\n7. Line 250-252: How did we move from matching only the mean to matching the mean and variance of $P_{Z|S}$? More explanation may be warranted here if possible.\n8. Relatedly, how did we go from matching the mean of $P_{Z|S}$ (line 239-242) to matching the mean of each feature in Theorem 5.2? A bit more explanation here could help bring some clarity.\n9. Period missing end of line 286.\n10. How does this paper relate to prior work with characteristic functions (e.g., citation [1] from the submission)\n\nQUESTIONS:\n1. To estimate the empirical CF in equation (9) of $P_{Z|S}$, must we do rejection sampling? Is this why the cost of monte carlo is so high? Then, does the computational complexity scale with the size of the smallest group?\n\nLIMITATIONS:\nSome drawbacks of FmCF are discussed, but discussion of computational complexity could be enhanced.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper develops a method for fair representation learning which focuses on \"specialized\" (i.e., task-specific) representations. It aims to avoid the issues with previous approaches such as fair extensions of VAEs, GANs, and normalizing flows. Basically, the approach is to incorporate a penalty term based on characteristic function distance. The method is general, but the case of classification is specifically handled. Some theoretical results are provided, including fairness guarantees in the case of logistic regression. Experiments verify the efficacy of the method.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n\n - The proposed method is quite novel, particularly in the context of fairness. The use of characteristic function distance is an emerging approach to model fitting which has previously been employed for GANs [1], and has some interesting relationships to maximum mean discrepancy (MMD), as well as the method of moments.\n \n - This work could impact the field by providing a new paradigm for fair representation learning.\n \n - The proposed method is algorithmically simple (even while being technically deep), which is elegant and increases the chance that it will be widely used.\n\n - The technique does not need access to the sensitive attribute during test time, which is useful for privacy.\n \n - This manuscript did a good job of contextualizing the research in terms of competing approaches for fair representation learning.\n\n\n*Weaknesses*\n\n - I would have liked to have seen an exploration of sample size versus estimation/fairness performance, with comparison to baselines. Since the proposed characteristic function distance is closely related to moment matching (Theorem 5.1), it may share some of its downsides such as statistical inefficiency and biased estimation with small sample sizes. (Perhaps the results would be unflattering, which could be why this experiment was not reported!)\n \n - The paper could be improved by adding more discussion on the method's potential practical value for concrete real-world applications.\n \n - This research seems to be heavily inspired by Ansari et al. (2020) [1]. The paper should further discuss its relation to that work, as well as other papers that use characteristic functions for model estimation. As well as giving proper credit to prior work, such a discussion would help contextualize the work within that subarea, and explain the extent to which characteristic function distance is an established and/or emerging estimation principle.\n\n\n*Minor suggestions*\n\n- Fix subscripted \"(\" in Equation 2\n\nQUESTIONS:\n- Does it matter that the P(Z|S) distributions are matched to a common target distribution instead of each other? How does the choice of target distribution affect the results?\n\n- Can you please more fully explain the relationship to Ansari et al. (2020) [1]? In what ways does this work build on that paper, and in what ways is it different?\n\n- How well does the method work with small datasets? Does the characteristic function distance strategy suffer from statistical inefficiency and biased estimation, as suggested by its relation to the method of moments?\n\nLIMITATIONS:\nA more thorough discussion on the downsides of the characteristic function distance methodology vs. other fair representation learning techniques is warranted. The paper spends a lot of time talking about the downsides of other methodologies, but does not dedicate much time to discussing the downsides of their own one (except briefly in the conclusion).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}