SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zheng Zhang $^{1,2}$  Qi Liu $^{1,2*}$  Hao Jiang $^{1,2}$  Fei Wang $^{1,2}$  Yan Zhuang $^{1,2}$
Le Wu $^{3}$  Weibo Gao $^{1,2}$  Enhong Chen $^{1,2}$
1: Anhui Province Key Laboratory of Big Data Analysis and Application
3: Hefei University of Technology
{zhangzheng,jianghao0728, wf314159,zykb, weibogao}@mail.ustc.edu.cn;
{qiliuql, cheneh}@ustc.edu.cn;
lewu.ustc@gmail.com
User modeling techniques profile users' latent characteristics (e.g., preference) from their observed behaviors, and play a crucial role in decision-making. Unfortunately, traditional user models may unconsciously capture biases related to sensitive attributes (e.g., gender) from behavior data, even when this sensitive information is not explicitly provided. This can lead to unfair issues and discrimination against certain groups based on these sensitive attributes. Recent studies have been proposed to improve fairness by explicitly decorrelating user modeling results and sensitive attributes. However, most existing approaches assume that fully sensitive attribute labels are available in the training set, which is unrealistic due to collection limitations like privacy concerns, and hence bear the limitation of performance. In this paper, we focus on a practical situation with limited sensitive data and propose a novel FairLISA framework, which can efficiently utilize data with known and unknown sensitive attributes to facilitate fair model training. We first propose a novel theoretical perspective to build the relationship between data with both known and unknown sensitive attributes with the fairness objective. Then, based on this, we provide a general adversarial framework to effectively leverage the whole user data for fair user modeling. We conduct experiments on representative user modeling tasks including recommender system and cognitive diagnosis. The results demonstrate that our FairLISA can effectively improve fairness while retaining high accuracy in scenarios with different ratios of missing sensitive attributes.
User modeling is a fundamental task in many applications (e.g., recommender system), which aims to infer latent characteristics through analyzing users' behavioral information, such as capability and preference [56]. The results from user modeling serve as an important basis for decision-making, which might affect users' lives to some extent [17, 15]. However, recent studies have demonstrated that biases related to sensitive attributes (e.g., gender, race) can be unconsciously captured from behavior data, even when this sensitive information is not explicitly provided. This can lead to unfair user modeling results [3, 23, 45]. For example, college admissions based on user modeling may underestimate underrepresented demographic groups [48]. Career recommendation shows apparent gender discrimination even for equally qualified men and women [19]. As such, it is crucial to consider fairness issues and ensure that users with different sensitive attributes are treated similarly.
Following previous studies [3, 23, 45], the fairness requirement for user modeling in this paper is that the results should not expose any sensitive information. This requirement can be formalized as minimizing the mutual information between user modeling results and sensitive attributes [2]. Along this way, many approaches have been proposed, such as constraint optimization [1], adversarial learning [3, 23, 45, 20] and regularization methods [49, 42], among which adversarial methods show their theoretical elegance and receive widespread attention.
While many methods have shown promising results, most of them require explicit sensitive attributes to guide fair model training. However, in real-world scenarios, users are not always willing to share sensitive information due to privacy concerns. For example, only  $14\%$  of teen users expose their complete profiles on Meta [31]. The limited sensitive attributes situation means that only a limited amount of data with known sensitive attributes can participate in model training, resulting in insufficient training data problem for existing methods. The most practical solution is to utilize data with unknown sensitive attributes to facilitate model training. Nevertheless, several obstacles must be addressed, including: 1) Efficient data utilization: Currently, some researchers have utilized data with unknown sensitive attributes by predicting missing sensitive labels based on data with known sensitive attributes [5]. However, the accuracy of these predicted labels cannot be guaranteed, especially in extremely limited sensitive situations, which results in suboptimal data utilization. 2)
Theoretical guarantee: The fair goal in our paper is to minimize the mutual information between the sensitive attributes and the user modeling results. When we utilize data with unknown sensitive attributes to improve fair user modeling, how can we establish a good theoretical guarantee? 3)
Framework generalization: Several fair adversarial methods have been introduced to handle data with known sensitive attributes [3, 23, 45, 20]. Each method has its advantages and disadvantages in different scenarios. In this case, we believe that a general framework should be proposed that can incorporate data with unknown sensitive attributes into existing fair adversarial methods.
To address these obstacles, in this paper, we propose FairLISA, a general framework for Fair user modeling with Limited Sensitive Attributes, which can efficiently utilize data with known and unknown sensitive attributes to facilitate fair model training. We first propose a novel theoretical perspective to build the relationship between two types of data and the fair mutual information objective. Specifically, we establish a connection between data with unknown sensitive attributes and the fairness goal through shannon entropy, while data with known sensitive attributes and the fairness goal are connected through cross-entropy. By doing so, we can directly leverage user data with unknown sensitive attributes without the need for predicting missing attributes, reducing information loss caused by predictions and ultimately increasing the utilization rate of data with unknown sensitive attributes. Then building on this foundation, we propose an adversarial learning approach to effectively leverage the entire user data for fair model training. Since existing fair adversarial approaches requiring fully known sensitive data can be interpreted from a mutual information perspective [3, 23, 45], FairLISA can easily generalize most existing adversarial methods and expand them to limited sensitive attribute situations. Finally, to validate the effectiveness of our proposed FairLISA framework, we conduct extensive experiments on two representative user modeling tasks including the recommender system and cognitive diagnosis. Empirical results show that our framework can enhance fairness with satisfactory accuracy performance, even in scenarios with different ratios of missing sensitive attributes. The main contributions of this work are as follows:
- Efficient Data Utilization and Theoretical Guarantee. We provide a novel theoretical perspective to build the relationship between data with known and unknown sensitive attributes with fairness objective. This allows for the effective leveraging of data with unknown sensitive attributes, reducing information loss caused by predicting missing attributes.
- Framework Design and Generalization. We propose a general framework, FairLISA, which can expand most existing fair adversarial methods to limited sensitive situations.
- Experimental Evaluations. We conduct experiments in two representative user modeling tasks (i.e., recommender system, cognitive diagnosis) with six representative models and two datasets, the results demonstrate the effectiveness and robustness of FairLISA.
User Modeling. User modeling is a fundamental task that captures users' useful potential characteristics, such as capability, preference, and so on [44, 56, 28, 50]. It has been applied in various
applications. For example, based on user capability fitting, researchers employed cognitive diagnosis [30, 9] to model student proficiency [4, 30]; based on user preference mining, researchers applied collaborative filtering [39] for mining user interest preferences to different tasks including news recommendation [26], social network [46]. Most of the existing user modeling methods focused on getting more precise results [4, 30, 16, 15, 27, 29]. However, as the results directly affect the opportunity of users [32, 54, 55], it can easily lead to unfair outcomes. In this paper, we explore fairness-aware user modeling, which aims to ensure fairness during the user modeling process.
Fairness in Machine Learning. Fairness in machine learning can be divided into three categories: (1) individual fairness, which requires similar individuals to be treated similarly [7, 6]; (2) group fairness, which requires different groups divided by sensitive attributes should be treated similarly [35, 7, 13]; (3) max-min fairness, which aims to maximize the utility of worst-off group [37]. In this work, we focus on group fairness since it can measure how the underrepresented group is treated.
In group fairness research, many fairness metrics and unfairness mitigation techniques had been proposed [22, 36, 45, 38, 47, 3, 23, 43, 10, 21]. For example, Avishek et al. [3] discovered the learned user representations would capture biases of sensitive information and proposed to minimize the mutual information between representations and sensitive attributes through an adversarial framework. Li et al. [23] also proposed this goal from the causal perspective and provided two similar adversarial strategies. Wu et al. [45] argued the user-centric structure would expose users' sensitive attributes and proposed an adversarial framework from a graph perspective to obtain more efficient results.
Despite their ability to improve fairness, all of these methods are based on enough user sensitive attribute information. While in many real-world applications, it is difficult to collect such data since users may care about privacy and are reluctant to divulge their sensitive information [31]. The lacking of sensitive information challenges most existing solutions. Exploring fair models with missing sensitive attributes is an important and challenging issue, and it is still under-explored. There are only a few works in this direction. One branch of approaches investigates fairness without sensitive attributes. For example, some researchers achieved this via solving a max-min problem [25, 14, 18]. However, they were only effective for max-min fairness [37], which was different from our group fairness goal. Some researchers tried to solve this problem with the help of related features [51, 53]. However, obtaining such related features is not always feasible in practical applications, thereby limiting the utilization of these methods. The other branch explores improving fairness in limited sensitive attribute situations, Dai et al. [5] proposed FairGNN, the main idea was to predict missing sensitive attribute labels. However, their reliance on the accuracy of the predicted pseudo labels resulted in suboptimal data utilization, particularly in extremely limited sensitive situations. Different from previous papers, we explore group fairness in limited sensitive attribute situations and can efficiently utilize data with known sensitive attributes without predicting missing sensitive attributes.
In user modeling, we denote  $U = \{u_{i}\}_{i=1}^{N_{u}}$  and  $V = \{v_{i}\}_{i=1}^{N_{v}}$  as the user set and the item set, respectively. The interactions between users and items generate interaction records  $R$ , whose element is denoted as triplet  $(u, v, r)$  where  $u \in U$ ,  $v \in V$ , and  $r$  is the corresponding interaction decided by the corresponding task. As we have assumed that sensitive attributes are only available for a limited number of users, we split the user set  $U$  into two disjoint sets as  $U_{L} = \{u_{i}\}_{i=1}^{l}$  and  $U_{N} = \{u_{i}\}_{i=l+1}^{l+n}$  so that the sensitive label  $s$  is only accessible for each user in  $U_{L}$  instead of  $U_{N}$ . Let  $S$  be the sensitive attribute that takes value in  $S$ , whose values for users in  $U_{L}$  are available, i.e.,  $\{s_{i} | s_{i} \in S\}_{i=1}^{l}$ . Our goal is to learn a user model  $f$  that generates a fair representative characteristic  $\theta$  for each user  $u$ . Following previous studies [3, 21, 42], the fairness requirement for user modeling is that the generated outcome should not expose any sensitive information, which can be depicted as the result of zero mutual information between the user's generated characteristic  $\theta$  and sensitive attribute  $S$ , i.e.,  $I(\theta, S) = 0$ . In this way, we have the following problem definition:
Problem 1. Given interaction records  $R$ , users  $U$ , items  $V$  and the sensitive label set  $\{s_i\}_{i=1}^l$  for  $U_L \subset U$ , the goal is to learn a user model  $f: U \to \Theta$  that generates a fair representative characteristic  $\theta \in \Theta$  for each user  $u \in U$ , such that  $I(\theta, S) = 0$  where  $S$  is the corresponding sensitive attribute.
4 FairLISA
FairLISA is an adversarial learning-based framework designed to promote fair model training by efficiently utilizing data with known and unknown sensitive attributes. The architecture is shown in Figure 1. It consists of two modules including a filter  $\mathcal{F}$  and a discriminator  $\mathcal{D}$ . Specifically, for a given user with the original representation  $\theta$  generated by any user models, we use the filter  $\mathcal{F}$  trained on data with known sensitive attribute  $U_{L}$  and data with unknown sensitive attribute  $U_{N}$  to filter out the information of sensitive attributes. The filtered user modeling result of a user  $u$  is denoted as  $\hat{\theta} = \mathcal{F}(\theta)$ . To guide the filter  $\mathcal{F}$  training, we use the idea of adversary learning to train a discriminator  $\mathcal{D}$ . The goal of the discriminator  $\mathcal{D}$  is to fail the filter  $\mathcal{F}$ , which is achieved by predicting the corresponding sensitive attribute from the filtered user representations  $\hat{\theta}$ . Through adversarial training between  $\mathcal{F}$  and  $\mathcal{D}$ , we can get a filter to remove the effect of sensitive attributes. Unlike traditional adversarial methods [3, 23, 45] that only rely on  $U_{L}$ , FairLISA can train a filter
Figure 1: The architecture of FairLISA. For a given user with the original representation  $\theta$  generated by any user models, we first use the filer  $\mathcal{F}$  trained on  $U_{L}$  and  $U_{N}$  to remove the sensitive information and get the filtered representation  $\hat{\theta}$ . Then we use the discriminator  $\mathcal{D}$  trained on  $U_{L}$  to predict the sensitive feature of  $\hat{\theta}$ . Through adversarial training and the better utilization of  $U_{N}$ , we can get a better filter to remove sensitive information.
with  $U_{L}$  and  $U_{N}$ , i.e.,  $\hat{\theta} = \mathcal{F}_{U_L + U_N}(\theta)$ . Following the previous study [5], we assume the distribution of users' filtered representations in  $U_{L}$  and  $U_{N}$  to be the same and use  $p(\hat{\theta})$  to represent them. We will introduce how FairLISA achieves the fairness goal (i.e.,  $I(\hat{\theta}, S) = 0$ ) through  $U_{L}, U_{N}$  respectively and how to utilize these two types of data to form the final architecture. For simplicity of theoretical analysis, we assume there is one sensitive attribute  $S$  (e.g., gender) and  $s$  (e.g., female) is an attribute value of  $S$ .
4.2 Fairness with Known Sensitive Attributes
We will begin by explaining how FairLISA achieves  $I(\hat{\theta}, S) = 0$  with data with known sensitive attributes  $U_{L}$ , i.e.,  $\hat{\theta} = \mathcal{F}_{U_L}(\theta)$ . More specifically, we will describe how to establish a connection between  $U_{L}$  and  $I(\hat{\theta}, S)$ , as well as how to leverage data with known sensitive attributes to minimize  $I(\hat{\theta}, S)$  based on this connection. In this regard, we define  $\mathcal{D}_s(\hat{\theta})$  as the probability that the discriminator  $\mathcal{D}$  recognizes the value of the sensitive attribute  $S$  as  $s$  based on  $\hat{\theta}$ ,  $H$  denotes the shannon entropy [24]. According to the following lemma, mutual information  $I(\hat{\theta}, S)$  can be expressed as the optimization function shown below, when the optimal discriminator can achieve  $\mathcal{D}_s^*(\hat{\theta}) = p(s|\hat{\theta})$ .
Lemma 4.1. Given the optimal discriminator  $\mathcal{D}_s^* (\hat{\theta})$  and let  $p(\hat{\theta} |s)$  be the distribution of users' filtered representations whose sensitive attribute value is  $s$ , then  $I(\hat{\theta},S)$  can be derived as,
$$
I (\hat {\theta}, S) = \sum_ {s} p (s) \mathbb {E} _ {\hat {\theta} \sim p (\hat {\theta} | s)} \left[ \log \mathcal {D} _ {s} ^ {*} (\hat {\theta}) \right] + H (S). \tag {1}
$$
The detailed proof is provided in Appendix A.1. We have established a connection between  $U_{L}$  and  $I(\hat{\theta}, S)$  through Lemma A.1. Consequently, fairness can be achieved by minimizing Eq. (11) with a fixed optimal discriminator  $\mathcal{D}^*$ . It is important to note that estimating  $p(\hat{\theta} | s)$  in Eq. (11) requires access to sensitive information. This implies that leveraging  $U_{N}$  (data with unknown sensitive attributes) to enhance fairness becomes impossible. However, in real-world applications, users are often concerned about privacy and may be unwilling to disclose their sensitive information. The limited availability of sensitive attributes means that only a restricted amount of data with known
sensitive attributes can participate in model training, leading to an insufficient training data problem for existing methods. In the following sections, we will introduce how to alleviate this problem by effectively utilizing data with unknown sensitive attributes.
4.3 Fairness with Unknown Sensitive Attributes
An intuitive approach to leverage data with unknown sensitive attributes  $U_{N}$  is to predict the missing sensitive labels based on  $U_{L}$ . However, the accuracy of these predicted pseudo-labels cannot be guaranteed, which results in suboptimal data utilization. Therefore, it is crucial to explore new methods that fully harness the potential of  $U_{L}$  to enhance fairness. Given that the suboptimal data utilization stems from predicting missing attributes, a natural idea is to directly employ  $U_{N}$  without predicting missing attributes, which can avoid information loss caused by prediction and maximize the utilization of  $U_{N}$  to a greater extent. To achieve this objective, it is necessary to directly establish a relationship between  $I(\hat{\theta}, S)$  and  $U_{N}$ . Drawing inspiration from Lemma A.1 and structural insights related to the KL divergence [11], we conclude that shannon entropy  $H$  can build this relationship.
Lemma 4.2. Given the optimal discriminator  $\mathcal{D}_s^* (\hat{\theta})$  and let  $p(\hat{\theta})$  be the distribution of users' filtered representation, we have,
$$
\begin{array}{l} I (\hat {\theta}, S) = \mathbb {E} _ {\hat {\theta} \sim p (\hat {\theta})} \sum_ {s} \left[ \mathcal {D} _ {s} ^ {*} (\hat {\theta}) \log \mathcal {D} _ {s} ^ {*} (\hat {\theta}) \right] + H (S) \tag {2} \\ = - H \left(\mathcal {D} _ {s} ^ {*} (\hat {\theta})\right) + H (S). \\ \end{array}
$$
Proof. According to the relationship between mutual information and JS divergence, the relationship between JS divergence and KL divergence [24], we have
$$
I (\hat {\theta}, S) = J S _ {p (s)} (p (\hat {\theta} | s)) = \sum_ {s} p (s) K L (p (\hat {\theta} | s) | | p (\hat {\theta})). \tag {3}
$$
By calculating KL divergence, we get,
$$
\sum_ {s} p (s) K L \left(p (\hat {\theta} | s) \right\lVert p (\hat {\theta}) = \int \sum_ {s} p (s) p (\hat {\theta} | s) \log \frac {p (\hat {\theta} | s)}{p (\hat {\theta})} d \hat {\theta}. \tag {4}
$$
Then we expand the formula and use bayes' theorem,
$$
\begin{array}{l} \int \sum_ {s} p (s) p (\hat {\theta} | s) \log \frac {p (\hat {\theta} | s)}{p (\hat {\theta})} d \hat {\theta} \tag {5} \\ = \int p (\hat {\theta}) \sum_ {s} p (s | \hat {\theta}) \log p (s | \hat {\theta}) d \hat {\theta} + H (S). \\ \end{array}
$$
Noted that  $p(s|\hat{\theta})$  is exactly  $\mathcal{D}^*$ , we replace it in the formula, and get the result,
$$
\begin{array}{l} \int p (\hat {\theta}) \sum_ {s} p (s | \hat {\theta}) \log p (s | \hat {\theta}) d \hat {\theta} + H (S) \tag {6} \\ = - H \left(\mathcal {D} ^ {*} (\hat {\theta})\right) + H (S). \\ \end{array}
$$
Here we give the key steps of proof, more details are provided in Appendix A.2. In Lemma A.2, the term  $p(\hat{\theta} | s)$  disappears, removing the obstacle that hindered the direct utilization of  $U_N$  as discussed in Lemma A.1. Thus, we have successfully established the relationship between  $I(\hat{\theta}, S)$  and  $U_N$  through Lemma A.2. We also provide an intuitive illustration. By maximizing the entropy of the discriminator, we can ensure that the prediction probability of sensitive attributes becomes uniform. For example, in the case of gender prediction, both male and female probabilities are set to 0.5. This shows that the discriminator becomes unable to predict the sensitive features from the learned user embedding. Consequently, we can minimize our fairness criterion  $I(\theta, S)$  directly by minimizing Eq. (15) using  $U_N$ . This allows for the effective utilization of  $U_N$  to enhance fairness without the need to predict unknown sensitive attribute labels.
4.4 Architecture
In this section, we now introduce how to combine  $U_{L}$  and  $U_{N}$  together to form the final architecture (shown in Figure 1) in order to obtain a higher improvement of fairness, i.e.,  $\hat{\theta} = \mathcal{F}_{U_L + U_N}(\theta)$ . By leveraging the insights from Lemma A.1 and Lemma A.2, we have established the connection between  $U_{L}, U_{N}$ , and the fairness goal  $I(\hat{\theta}, S)$ . Thus, fairness can be achieved through minimizing Eq. (11) with  $U_{L}$  and minimizing Eq. (15) with  $U_{N}$  with a fixed optimal discriminator  $\mathcal{D}^*$ . To achieve this goal, we first optimize the  $\mathcal{D}$  to the best, then fix  $\mathcal{D}$  to optimize  $\mathcal{F}$ . FairLISA consists of a filter and a discriminator. We will introduce them respectively as well as the training algorithm.
Filter. We use the whole data to optimize the  $\mathcal{F}$ . For data with known sensitive attributes, we optimize  $\mathcal{F}$  by minimizing the right side of Eq. (11). The empirical estimate can be computed as:
$$
\mathcal {L} _ {L} = \frac {1}{l} \sum_ {s} \sum_ {1 \leq i \leq l: s _ {i} = s} \log \mathcal {D} _ {s} (\hat {\theta} _ {i}), \tag {7}
$$
where  $\hat{\theta}_i$  is the filtered representation of  $u_i$ . Here we omit  $H(S)$  in Eq. (11) since it is a constant.
For data with unknown sensitive attributes, we optimize  $\mathcal{F}$  by minimizing the right side of Eq. (15). The empirical estimate can be computed as:
$$
\mathcal {L} _ {N} = \frac {1}{n} \sum_ {l + 1 \leq i \leq l + n} \sum_ {s} \mathcal {D} _ {s} \left(\hat {\theta} _ {i}\right) \log \left(\mathcal {D} _ {s} \left(\hat {\theta} _ {i}\right)\right), \tag {8}
$$
where  $\hat{\theta}_i$  is the filtered representation of  $u_i$ . Here we also omit  $H(S)$  since it is a constant.
Furthermore, in addition to the fairness requirement, we need to maintain users' useful potential characteristics for different user modeling tasks (e.g., recommender system, cognitive diagnosis). To achieve this, we use a task-specific loss function to mine useful information of users. With the combination of Eq. (7) and Eq. (8), we get the following loss function:
$$
\min  _ {\mathcal {F}} \mathcal {L} _ {\mathcal {F}} = \lambda_ {1} \mathcal {L} _ {\text {t a s k}} + \lambda_ {2} \mathcal {L} _ {L} + \lambda_ {3} \mathcal {L} _ {N}, \tag {9}
$$