Stereotype or Personalization? User Identity Biases Chatbot Recommendations
Authors: Anjali Kantharuban, Jeremiah Milbauer, Maarten Sap, Emma Strubell, Graham Neubig
Accepted at: acl25

Stereotype or Personalization?
User Identity Biases Chatbot Recommendations

Anjali Kantharuban* Jeremiah Milbauer*
Maarten Sap Emma Strubell Graham Neubig
Carnegie Mellon University
anjaliruban@cmu.edu jmilbaue@andrew.cmu.edu
msap2@andrew.cmu.edu estrubel@cs.cmu.edu gneubig@cs.cmu.edu

Abstract

While personalized recommendations are often desired by users, it can be difficult in practice to distinguish cases of bias from cases of personalization: we find that models generate racially stereotypical recommendations regardless of whether the user revealed their identity intentionally through explicit indications or unintentionally through implicit cues. We demonstrate that when people use large language models (LLMs) to generate recommendations, the LLMs produce responses that reflect both what the user wants and who the user is. We argue that chatbots ought to transparently indicate when recommendations are influenced by a user’s revealed identity characteristics, but observe that they currently fail to do so. Our experiments show that even though a user’s revealed identity significantly influences model recommendations (\( p < 0.001 \)), model responses obfuscate this fact in response to user queries. This bias and lack of transparency occurs consistently across multiple popular consumer LLMs and for four American racial groups.¹

1 Introduction

Language is [...] the most vivid and crucial key to identify: It reveals the private identity, and connects one with, or divorces one from, the larger public, or communal identity.

James Baldwin, 1979

The increased accessibility of large language models (LLMs) with user-friendly chat interfaces has led to a surge in popularity, with people commonly using them to get recommendations (Wang et al., 2024).² The underlying process of generating recommendations with a language model involves the model sampling likely responses to the user’s query from its trained parameters. Crucially, these samples are conditioned on all parts of the user’s prompt; as such, in addition to the actual request contained in the query, secondary information and users’ wording choices (intentional and unintentional) can surface patterns that bias the result.

We posit that this mechanism embeds pragmatics into conversations (Grundy, 2019) between people and LLMs. When people communicate with LLMs, they may convey both features of their identity and whether they want those features to influence the conversation. For example, in Figure 1, we see three ways in which a high school student might ask for advice on what universities to consider. The baseline form of the question does not reveal identity, so the system’s default assumptions play a larger part in its recommendations, recommending prestigious institutions that, historically, mostly admitted white students. On the other hand, when the user includes explicit indications that they are Black, the system recommends historically Black colleges and universities (HBCUs).³ In the final

Figure 1: Asking for options with different levels of identity disclosure. Indicating that they are Black results in biased responses, whether intended [ ] or not [ ]. When asked, the response obfuscates the impact of race.

then querying gpt-4o-mini to label a subset of the user requests.

¹ Equal Contribution
² An estimated 1% of queries in the WildChat (Zhao et al., 2024) dataset request recommendations, calculated by first filtering for keywords “suggest” and “recommend”, and
example, the user does not explicitly mention their race, but they mention volunteering with an organization that primarily serves black communities in major cities. The user gives no indication that they want the inferred race to inform the recommendations, but the system generates recommendations informed by its assumptions, and risks stereotyping the user.

Furthermore, the system obfuscates the influence of race in generating the recommendations. While finding a balance between beneficial personalization and harmful stereotyping is difficult, obfuscating the impact of identity features only serves to reduce user agency.

To evaluate whether models appropriately handle implicit and explicit identity disclosure, we evaluate across three types of identity signals: dialectal features, references to associated entities, and explicit indications. Within this framework, we empirically examine three research questions about the behavior of LLMs when they are used as recommender systems:

RQ1: Do revealed identity features (whether implicit or explicit) bias the recommendations generated by large language models?
RQ2: Does removing identity markers entirely result in unbiased recommendations?
RQ3: Do models transparently disclose when they take identity into account?

We generate synthetic recommendation requests covering a variety of user identities, identity signals, and constraints. Aggregating LLM responses to these requests, we measure the real-world demographic alignment of the recommendations to the user’s identity using official US government sources. Ultimately we find an implicit identity effect across all models tested: models generate biased recommendations regardless of the user’s expressed desire for personalization, and subsequent responses will obfuscate this bias from the user, reducing user agency and reinforcing existing stereotypes.

2 Methodology

To study the degree of bias in LLM-generated recommendations, we compare their responses across three levels of identity disclosure: implicit, explicit, or none. We then measure the real-world demographic distribution associated with the recommended entities to determine whether the model produces recommendations that are demographically biased toward the user’s revealed identity. Figure 2 illustrates the process.

2.1 Identity Selection

Although bias can be examined on many axes of identity (socioeconomic class, gender, sexuality, etc.), in our analysis we specifically focus on race within the United States for three reasons. First, the U.S. government collects demographic data on race for a variety of entities of interest, such as universities (U.S. Department of Education, 2024) and neighborhoods (U.S. Census Bureau, 2016). Sec-
ond, there are distinctive linguistic variations associated with English-speaking racial groups in the U.S. that have been well documented (Schneider, 2008). Last, the systematic and ingrained racial discrimination in American society has been heavily studied, and identifying LLM regurgitation of these themes is of great importance, and has been the focus of significant previous work in other contexts (Caliskan et al., 2017; Garg et al., 2018; Hofmann et al., 2024a).

2.2 Constructing Prompts
We develop an automatic procedure for composing realistic recommendation request prompts. Past work has used synthetic queries for the purposes of isolating sources of bias, since natural queries often introduce multiple degrees of variation from one another (Wan et al., 2023a; Röttger et al., 2024; Castricato et al., 2025). Our prompts are more complex than just simple templates and they are modeled on the way people actually interact with models (Wang et al., 2024) – rather than asking models blunt survey questions designed for humans (Tjuatja et al., 2024). These synthetically generated prompts allow us to have much greater coverage of diversity identities and to systematically vary features to derive much more rigorous and sound results than if we sourced naturally written human prompts. Each prompt contains user requests with optional constraints and either explicit or implicit identity disclosures. For implicit identity disclosures, we consider implicitly revealed identity through either reference to demographically associated entities or the use of demographically linked dialects.

These implicit identity disclosures, both associated entities and dialectal features, were chosen by asking each model to generate a list of racially associated entities within a category. The lists were then manually compressed to remove items that occurred across lists for multiple races, items that were not attested in linguistic resources (for dialectal features) and items that explicitly mention the race of the user. More information about prompt generation can be found in Appendix B.

We use Standard American English (Trudgill, 2002) as taught in schools as the dialectal baseline due to most dialectal speakers having the ability to code-switch into it in more formal contexts (De-Bose, 1992). For associated entities, we use generic references ("I play sports" as opposed to "I play football") or entities that appeared across the generated lists for multiple races.

Last, we provided non-racial constraints and requests as a method of diversifying the model generations. Examples include varying the standardized testing scores, budgets, and preferences of the user. This ensures that notions such as perceived socioeconomic class and ability play a lesser role.

2.3 Measuring Demographic Alignment
Following each request, we calculate the real-world demographic alignment of each recommendation to the revealed identity of the request. Information on matching recommendations with entities reported on can be found in Appendix C.1. We approach alignment in a few ways. Primarily, we examine the percentage share of people of the user’s race in each recommended university or neighborhood, pulled from U.S. government statistics, then average across the recommendations provided. For a demographic group \( g \) with recommendations \( R \):

\[
\text{Mean Share}(R, g) = \frac{1}{|R|} \sum_{r \in R} \frac{\# \text{ of } g \text{ people in } r}{\# \text{ of people in } r}
\]

Additionally, we investigate the diversity and representativeness of recommendations in the best case where personalization is appropriate – when users pragmatically indicate to do so through explicit disclosure. We compare recommendation distributions with real-world data on neighborhood population (where data is more readily available).

2.4 Experimental Design
To align the racial groups with documented dialectal and data boundaries, we focus on four categories: White, Black, Hispanic, and Asian.\footnote{Throughout this paper, we capitalize in accordance with the APA Style Guide on racial identity.} Dialect features were chosen in accordance with the prompt’s syntactic structure manually from linguistic resources (Trudgill, 2002; Hanna, 1997). We examine the results on a range of popular consumer chatbots: GPT 4o Mini, GPT 4 Turbo, Claude 3.5 Haiku, Claude 3.5 Sonnet, Llama 3.1 70B, Llama 3.1 405B, and Gemini 1.5 Pro. Exact specifications on model configuration can be found in Appendix A.

We evaluate on two tasks: university and neighborhood selection. For universities, we focus our scope on American universities but otherwise leave the location unspecified. For neighborhoods, we specify one of three cities: New York City, Los Angeles, and Chicago. Prompting details, and the
exact criteria and contents for the group-specific and baseline entity sets, are in Appendix B, as well as details on how we validate our sampling setup in Appendix E.

Due to the number of possible queries (\( n > 10! \)), we sample 8,600 queries for each model for both university and neighborhood recommendations, for a total of 120,000 synthetic user requests. The template for the prompts we used can be found in Appendix B. We constrain the outputs to be in a JSON format for analysis, but we find that the trends we see hold in the unconstrained setting in Appendix E. We align recommendations with demographic data from official US government sources (U.S. Census Bureau, 2016, 2010; U.S. Department of Education, 2024).

3 RQ1: Inferred Identity Biases Recommendations

In our first experiment, we examine whether LLMs provide recommendations that are biased towards the perceived racial identity of the user, regardless of whether identity is explicitly revealed.

3.1 Demographic Alignment by Signal

A model that personalizes only when the user explicitly asks for their racial identity to factor into decisions – and otherwise avoids incorporating stereotypes – should behave differently depending on the type of identity disclosure. When identity is revealed implicitly and it is unclear whether it is intentional, we might expect the demographic alignment of recommendations to be similar to those generated when identity is not revealed. On the other hand, when identity is explicitly indicated, we expect the models to favor recommendations that have greater demographic alignment.

Figure 3 shows the degree of demographic alignment for each user identity averaged across models. We see that models respond strongly to explicit signals of user identity by providing recommendations that have comparatively high shares of the user’s race. For example, in the plot of Black users’ results for neighborhood selection, we see that when a user requests recommendations for neighborhoods and includes “keep in mind, I am Black,” they receive recommended neighborhoods with around significantly more Black residents on average than a user who does not reveal their identity. This personalization based on explicit identity disclosure is seen for users of all races except White users, whose results mostly still fall squarely within the range of those given to baseline users.

Models also produce demographically aligned results when users reveal their identity implicitly, either by referencing stereotypically associated entities, or when a user writes their request in a racially-associated dialect. This occurs to a lesser extent than it does with explicitly disclosed identity, but the difference is still consistently statistically significant when the user references associated entities and occasionally significant (albeit minimal) when the user uses dialectal phrasing (evaluated though an unpaired T-Test). In the context of important life decisions, basing decisions on unintentionally disclosed racial identities can perpetuate stereotyping without the knowledge of the user. Additionally, there is the risk of error; not everyone who references an entity belongs to the stereotypically associated race (e.g., anyone can play basketball) and not everyone who speaks a specific racial dialect identifies with that race, especially with the internet offering more cross-cultural interactions (Reyes, 2005; Roth-Gordon et al., 2020).

3.2 Recommendation Diversity

<table>
  <tr>
    <th>Model</th>
    <th>Asian</th>
    <th>Black</th>
    <th>Hispanic</th>
    <th>White</th>
  </tr>
  <tr>
    <td>GPT 4o Mini</td>
    <td>0.51</td>
    <td>0.48</td>
    <td>0.47</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>GPT 4o</td>
    <td>0.55</td>
    <td>0.49</td>
    <td>0.50</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>Claude 3.5 Haiku</td>
    <td>0.50</td>
    <td>0.54</td>
    <td>0.48</td>
    <td>0.53</td>
  </tr>
  <tr>
    <td>Claude 3.5 Sonnet</td>
    <td>0.50</td>
    <td>0.45</td>
    <td>0.46</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>Llama 3.1 70B</td>
    <td>0.24</td>
    <td>0.11</td>
    <td>0.16</td>
    <td>0.22</td>
  </tr>
  <tr>
    <td>Llama 3.1 405B</td>
    <td>0.42</td>
    <td>0.30</td>
    <td>0.35</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Gemini 1.5 Pro</td>
    <td>0.45</td>
    <td>0.53</td>
    <td>0.51</td>
    <td>0.62</td>
  </tr>
  <tr>
    <td>Reality</td>
    <td>0.83</td>
    <td>0.77</td>
    <td>0.81</td>
    <td>0.87</td>
  </tr>
</table>

(a) Diversity: Normalized entropy of recommendation distributions, compared to entropy of the real-world distribution. Higher entropy indicates higher diversity

<table>
  <tr>
    <th>Model</th>
    <th>Asian</th>
    <th>Black</th>
    <th>Hispanic</th>
    <th>White</th>
  </tr>
  <tr>
    <td>GPT 4o Mini</td>
    <td>0.63</td>
    <td>0.62</td>
    <td>0.63</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>GPT 4o</td>
    <td>0.57</td>
    <td>0.62</td>
    <td>0.57</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>Claude 3.5 Haiku</td>
    <td>0.68</td>
    <td>0.71</td>
    <td>0.66</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Claude 3.5 Sonnet</td>
    <td>0.63</td>
    <td>0.66</td>
    <td>0.62</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Llama 3.1 70B</td>
    <td>0.83</td>
    <td>0.88</td>
    <td>0.83</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>Llama 3.1 405B</td>
    <td>0.74</td>
    <td>0.82</td>
    <td>0.71</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>Gemini 1.5 Pro</td>
    <td>0.59</td>
    <td>0.51</td>
    <td>0.53</td>
    <td>0.54</td>
  </tr>
</table>

(b) Representativeness: Normalized JSD between neighborhood recommendations and real-world distribution. Lower JSD indicates higher representativeness.

Table 1: Comparison between recommendation distributions (when race is explicitly indicated) and real-world distributions.

To understand LLMs’ ability to personalize when pragmatically indicated to do so, we also measure the diversity and representativeness of model recommendations when race is explicitly mentioned. We measure diversity as the entropy
Figure 3: Demographic alignment (via share of each race) for the recommendations provided by all evaluated models across the inclusion of dialectal features [Dia.], associated entities [Ent.], and explicit indications [Exp.]. Left boxplots [■] represents users of the same race as the figure’s calculation and right boxplots [■] represents users of all other races. Shaded range [■] is the range (25th - 75th percentile) of race share for baseline prompts across models.

of model recommendations over the full spectrum of possible neighborhood outcomes (where complete data is more readily available), when race is explicitly indicated by the user. To avoid issues with unequal support across distributions, we use additive smoothing with \( \epsilon = 1 \cdot 10^{-10} \). We measure representativeness as the Jensen-Shannon Divergence of the recommendation distribution and reality. We normalize both Diversity and Representativeness. Exact equations for these measurements are included in Appendix C.3

Empirically, in Table 1 we see that the models produce recommendations that are much less diverse than reality. This effect is especially pronounced for Llama 3.1 70B, where recommended neighborhoods for Black users represent just 7.8% of the Black population in the cities, compared to recommendations from Gemini 1.5 Pro’s, for example, which represent over 50%. We also see that Llama 3.1 70B produces the recommendations that are least similar to the real-world distribution. These results indicate that when a user expresses their race in a manner that indicates personalization is desirable, the models produce less diverse and often unrepresentative results.

3.3 Stereotypical Descriptions

Beyond the numerical demographic alignment of recommended entities, we find that language models tend to bias the values they latently attribute to users of different races, represented by the variation in explanations. To examine this, we find words with the highest association with a particular demographic group. Details of this calculation along with the top twenty terms associated with each race can be found in Appendix D.

For university recommendations, we see that models include race-based terms more frequently when race is mentioned explicitly, which is pragmatically appropriate and demonstrates personalization. However, even when the only variation is dialectal (that is, all users describe the same concern around cost and similar stats), non-White students see an increased emphasis on features such as in-state resident tuition (Asian users), lenient admissions standards (Hispanic users), and opportunities for part-time work (Black users). On the other hand, when identity is explicitly disclosed, Asian users are assumed to want stem-focused programs while black users are instead told about agricultural programs.

Similarly, we see some clear patterns even among purely dialectal prompts for neighborhood recommendations. Although all users of all races express same set of transportation constraints, there is more emphasis on different modes of transportation for each racial group. Black users are told their recommendations are subway and bus-accessible, Hispanic users are tempted with bike lanes, and Asian users are promised their neighborhoods are car-accessible. Hispanic users are given neighborhoods described as chef-friendly while Asian users are given those described as foodie-friendly, reflecting stereotypes about the racial makeup of the service industry. When race is stated explicitly, the most frequently referenced stereotypes are different, but still clear; Black users are recommended more politically progressive areas than the mod-
erate areas suggested to White users. Similarly, Hispanic and Asian users are recommended tight-knit and traditional communities, which reflect the cultural values associated with ethnic enclaves.

By generating explanations that assign different values and preferences to different racial groups, models not only produce biased recommendations, but generate rationales that reflect (and possibly reinforce) the stereotypes that already exist in American culture.

4 RQ2: Standard American English Elicits Biased Recommendations

So we’ve established that revealed identity impacts recommendations, even when signaled implicitly or unintentionally. In this section we examine whether deleting identity from the prompt entirely removes the capacity for harmful bias. To determine this, we can evaluate whether baseline prompts (i.e., prompts that do not disclose the user’s race, implicitly or explicitly) yield results that demographically align equally well with all races.

As seen in Figure 3, there is a stark difference between the demographic alignment of the recommendations for non-White users versus White users. For White users, not only is the White share of both universities and neighborhoods relatively stable across levels of disclosure, but it is also close in value to that of the baseline prompts. As signaling gets more explicit, we see that the non-White users’ recommendations dropping share of White people is more notable than the increase in share seen in the recommendations for White users. The numerical values for each model demonstrating this point can be found in Appendix C.2. This means that while a White user can choose to include or omit indications of their race and be provided similar results, non-White users need to reveal more information to get personalized responses.

These findings reinforce past work which argues that allocation harm is manifested in the additional effort is required by non-White users to adapt their prompts for successful use (Cunningham et al., 2024) and in the extra cost of additional tokens required to model linguistic variety (Ahia et al., 2023). While removing identity features from prompts altogether eliminates racial stereotypes from the recommendations, the sanitized outputs are still biased in favor of one group of users, providing less aligned services for those who do not identify with the model’s default assumptions.

5 RQ3: Models Don’t Disclose When Recommendations Are Biased

Users may be sensitive to the possibility that recommendations they received from an LLM are biased. For closed-source models without built-in explanation tools (Zhao et al., 2023), a user might proceed to a second conversational turn, asking: “Did my race influence your recommendations?”

To examine how transparent models are, we calculate the average demographic alignment of recommendations across three scenarios: second turns where the response indicates that race influenced the recommendations, ones where the response indicates that race didn’t, and responses belonging to prompts that had no user identity features included. Because the second conversational turn occurs after the recommendations are already produced, the second response is conditioned on the degree of bias in the recommendations. Ideally, for instances where the model responds that it did not factor in race, that response would be true: there would be no statistically significant difference in the recommendation distribution compared with a neutral prompt.

To test if responses obfuscate the effect of identity on recommendations, we consider as our null hypothesis the notion that stereotypical recommendations lead to indications of racial influence in the follow-up. Contrapositively, we test if responses indicating that race was not a factor entail recommendations that are not biased and therefore not more demographically aligned than the baseline.

Table 2 shows the results of this test: across every model, for non-White users, there is significant bias even when responses deny the impact of racial identity. For White users, there are occasionally significant differences, but model recommendations are broadly unaffected by the user’s disclosed race, regardless of whether they determine it was taken into account or not. There is a dropoff in mean share across races other than White in demographic alignment between when cases where the model admits to taking race into account versus where it doesn’t. This is a positive sign that the models can determine in more clear cases that race was a factor in their generations; still, none of the models achieve consistent transparency.

5Llama 3.1 70B almost always (across its very few cases) assumed the wrong race of White users, resulting in this dip.
<table>
  <tr>
    <th rowspan="2">Race Acknowledgment of Race</th>
    <th colspan="3">White</th>
    <th colspan="3">Black</th>
    <th colspan="3">Hispanic</th>
    <th colspan="3">Asian</th>
  </tr>
  <tr>
    <th>Yes</th><th>No</th><th>N/A</th>
    <th>Yes</th><th>No</th><th>N/A</th>
    <th>Yes</th><th>No</th><th>N/A</th>
    <th>Yes</th><th>No</th><th>N/A</th>
  </tr>
  <tr>
    <td>GPT 4o Mini</td><td>–</td><td>78.6</td><td>78.17</td><td>29.15</td><td>11.18</td><td>11.17</td><td>32.31</td><td>11.62</td><td>10.3</td><td>7.37</td><td>4.1</td><td>3.61</td>
  </tr>
  <tr>
    <td>GPT 4o</td><td>–</td><td>79.99</td><td>77.61</td><td>28.31</td><td>11.07</td><td>9.4</td><td>35.15</td><td>14.24</td><td>13.02</td><td>9.65</td><td>4.48</td><td>3.93</td>
  </tr>
  <tr>
    <td>Claude 3.5 Haiku</td><td>83.86</td><td>79.95</td><td>24.5</td><td>7.9</td><td>7.27</td><td>30.33</td><td>17.98</td><td>14.92</td><td>8.81</td><td>4.99</td><td>4.53</td>
  </tr>
  <tr>
    <td>Claude 3.5 Sonnet</td><td>80.58</td><td>78.31</td><td>21.74</td><td>8.69</td><td>7.47</td><td>32.48</td><td>18.32</td><td>15.05</td><td>7.05</td><td>3.93</td><td>3.92</td>
  </tr>
  <tr>
    <td>Llama 3.1 70B</td><td>81.54</td><td>83.25</td><td>83.85</td><td>24.59</td><td>10.86</td><td>8.91</td><td>31.38</td><td>9.6</td><td>5.78</td><td>8.61</td><td>3.29</td><td>2.52</td>
  </tr>
  <tr>
    <td>Llama 3.1 405B</td><td>–</td><td>82.97</td><td>83.09</td><td>37.27</td><td>9.63</td><td>9.07</td><td>44.67</td><td>10.14</td><td>5.8</td><td>15.7</td><td>6.27</td><td>2.83</td>
  </tr>
  <tr>
    <td>Gemini 1.5 Pro</td><td>73.52</td><td>76.8</td><td>76.35</td><td>28.32</td><td>9.5</td><td>8.67</td><td>28.76</td><td>14.48</td><td>13.12</td><td>8.64</td><td>6.11</td><td>5.61</td>
  </tr>
</table>

(a) University Recommendations

<table>
  <tr>
    <th rowspan="2">Race Acknowledgment of Race</th>
    <th colspan="3">White</th>
    <th colspan="3">Black</th>
    <th colspan="3">Hispanic</th>
    <th colspan="3">Asian</th>
  </tr>
  <tr>
    <th>Yes</th><th>No</th><th>N/A</th>
    <th>Yes</th><th>No</th><th>N/A</th>
    <th>Yes</th><th>No</th><th>N/A</th>
    <th>Yes</th><th>No</th><th>N/A</th>
  </tr>
  <tr>
    <td>GPT 4o Mini</td><td>–</td><td>44.13</td><td>43.95</td><td>37.83</td><td>15.09</td><td>13.11</td><td>46.2</td><td>30.9</td><td>28.61</td><td>20.53</td><td>12.61</td><td>11.54</td>
  </tr>
  <tr>
    <td>GPT 4o</td><td>–</td><td>39.3</td><td>39.22</td><td>35.62</td><td>15.89</td><td>12.46</td><td>49.03</td><td>38.28</td><td>35.23</td><td>18.45</td><td>11.94</td><td>10.6</td>
  </tr>
  <tr>
    <td>Claude 3.5 Haiku</td><td>51.41</td><td>47.21</td><td>46.56</td><td>24.41</td><td>10.88</td><td>9.37</td><td>43.75</td><td>31.99</td><td>29.06</td><td>14.47</td><td>12.89</td><td>12.47</td>
  </tr>
  <tr>
    <td>Claude 3.5 Sonnet</td><td>49.99</td><td>42.35</td><td>42.57</td><td>36.53</td><td>17.11</td><td>12.5</td><td>47.18</td><td>38.66</td><td>32.59</td><td>17.48</td><td>12.94</td><td>9.9</td>
  </tr>
  <tr>
    <td>Llama 3.1 70B</td><td>7.26</td><td>53.15</td><td>55.24</td><td>11.46</td><td>7.16</td><td>7.41</td><td>34.7</td><td>21.03</td><td>21.34</td><td>13.54</td><td>12.21</td><td>12.21</td>
  </tr>
  <tr>
    <td>Llama 3.1 405B</td><td>5.96</td><td>51.46</td><td>51.42</td><td>8.86</td><td>4.52</td><td>4.43</td><td>26.84</td><td>19.79</td><td>11.46</td><td>10.66</td><td>10.66</td><td>10.66</td>
  </tr>
  <tr>
    <td>Gemini 1.5 Pro</td><td>56.37</td><td>41.06</td><td>38.96</td><td>46.79</td><td>17.82</td><td>11.28</td><td>54.74</td><td>39.6</td><td>33.52</td><td>31.89</td><td>16.43</td><td>13.93</td>
  </tr>
</table>

(b) Neighborhood Recommendations

Table 2: Mean share of the user’s race across provided recommendations controlling for whether the model acknowledges the influence of race (Yes), doesn’t acknowledge the influence of race (No), and cases where no identity information was disclosed (N/A). Bolded values are significantly (\( p < 0.0001 \)) more demographically aligned than baseline recs.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">University Recommendations</th>
    <th colspan="4">Neighborhood Recommendations</th>
  </tr>
  <tr>
    <th>Implicit</th><th>Explicit</th><th></th><th></th>
    <th>Implicit</th><th>Explicit</th><th></th><th></th>
  </tr>
  <tr>
    <td>GPT 4o Mini</td><td>0.0</td><td>0.5</td><td>3.4</td><td>0.0</td>
    <td>0.0</td><td>1.00</td><td>1.00</td><td>11.2</td>
    <td>0.0</td><td>1.7</td><td>0.3</td><td>0.2</td>
    <td>0.0</td><td>95.9</td><td>92.5</td><td>69.5</td>
  </tr>
  <tr>
    <td>GPT 4o</td><td>0.0</td><td>0.8</td><td>0.0</td><td>0.0</td>
    <td>0.0</td><td>0.70</td><td>0.97</td><td>19.7</td>
    <td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
    <td>0.0</td><td>83.4</td><td>78.5</td><td>54.1</td>
  </tr>
  <tr>
    <td>Claude 3.5 Haiku</td><td>0.0</td><td>3.5</td><td>0.8</td><td>0.0</td>
    <td>1.5</td><td>100.0</td><td>100.0</td><td>31.6</td>
    <td>0.0</td><td>4.5</td><td>1.4</td><td>0.5</td>
    <td>0.0</td><td>93.4</td><td>99.4</td><td>94.6</td>
  </tr>
  <tr>
    <td>Claude 3.5 Sonnet</td><td>0.0</td><td>6.7</td><td>1.3</td><td>0.0</td>
    <td>3.4</td><td>100.0</td><td>100.0</td><td>81.2</td>
    <td>0.0</td><td>5.0</td><td>0.1</td><td>0.0</td>
    <td>2.5</td><td>99.4</td><td>99.7</td><td>94.2</td>
  </tr>
  <tr>
    <td>Llama 3.1 70B</td><td>0.0</td><td>2.5</td><td>0.0</td><td>0.0</td>
    <td>0.3</td><td>75.0</td><td>97.3</td><td>45.6</td>
    <td>0.5</td><td>3.3</td><td>2.1</td><td>0.7</td>
    <td>0.4</td><td>91.6</td><td>95.6</td><td>75.6</td>
  </tr>
  <tr>
    <td>Llama 3.1 405B</td><td>0.0</td><td>1.4</td><td>0.1</td><td>0.0</td>
    <td>0.0</td><td>92.7</td><td>81.2</td><td>9.2</td>
    <td>0.0</td><td>2.7</td><td>0.1</td><td>0.0</td>
    <td>0.0</td><td>58.7</td><td>60.2</td><td>47.1</td>
  </tr>
  <tr>
    <td>Gemini 1.5 Pro</td><td>0.0</td><td>7.7</td><td>3.4</td><td>0.0</td>
    <td>0.3</td><td>100.0</td><td>99.1</td><td>35.5</td>
    <td>0.0</td><td>6.4</td><td>0.1</td><td>0.9</td>
    <td>0.7</td><td>94.6</td><td>96.4</td><td>90.2</td>
  </tr>
</table>

<table>
  <tr>
    <th>W</th><th>B</th><th>H</th><th>A</th>
    <th>W</th><th>B</th><th>H</th><th>A</th>
    <th>W</th><th>B</th><th>H</th><th>A</th>
  </tr>
</table>

Figure 4: Percentage of times each model, in a post-hoc analysis, agrees that race factored into the provided recommendations for White [W], Black [B], Hispanic [H], and Asian [A] users. Models are more likely to admit to factoring in race when the prompt includes explicit disclosure of the user’s race unless the user is White.

We also find that models do not factor race equally for all users. As shown in Figure 4, on a purely numerical level models rarely, if ever, admitted to taking the race of White users into account, even when explicitly asked to do so by the user. On the other hand, models admit to considering race for other groups almost always when race is explicitly mentioned, and up to 7.7% of the time when identity is implicitly disclosed. This is most prominent for Black users. We also find that models treat explicit mentions of Asian identity differently between university and neighborhood recommendations, perhaps attributable to the “model minority” myth de-emphasizing Asian identity in higher education (Museus and Kiang, 2009), whereas Asian cultural enclaves are often distinct and prominent parts of a city.

Interestingly, smaller models (GPT 4o Mini, Claude 3.5 Haiku, and Llama 3.1 70B) have higher rates of communicating the impact of race in the case of explicit disclosure, despite showing similar – or sometimes even smaller – differences in the racial makeup of their recommendations in Table 2 compared to their larger counterparts.

Across models, prompts involving explicit self-identification see an appropriately high rate of race being factored in overtly unless the user is White. This may be in part because White users are most closely aligned with the default and as such their race truly does not impact their results.

6 Related Work

These results contribute to a growing body of work on the potential of machine learning systems to perpetuate bias, connecting it with an increased interest in AI systems that provide recommendations in a conversational setting.

Fairness and Bias in LMs Language models trained on biased data replicate the biases and attitudes of the data they are trained on (Caliskan et al., 2017; Hovy and Prabhumoye, 2021), includ-
ing identity or community-specific attitudes (Milbauer et al., 2021; Jiang et al., 2022), which can be furthered by a biased selection of data (Dhamala et al., 2021; Lucy et al., 2024), and is difficult to remove (Gonen and Goldberg, 2019).

Models can cause representational harm by reproducing harmful stereotypes (Agarwal et al., 2019) or result in unfair decision making when deployed in sensitive contexts like law enforcement (Angwin et al., 2022), hiring (Dastin, 2022), or healthcare (Gianfrancesco et al., 2018; Obermeyer et al., 2019). Recently Hofmann et al. (2024b) demonstrated that language models can produce biased output in response to AAE dialectal text, and Bhatt and Diaz (2024) found that explicit mentions of cultural identities in the input condition a language model to produce culturally relevant adaptations.

Models can also cause allocational harm (Barocas et al., 2017) by providing different levels of benefit to different users. Other work has identified performance gaps with respect to dialect (Kantharuban et al., 2023) and race (Sap et al., 2019), though with a focus on specific tasks rather than user outcomes.

Fairness and Bias in Recommender Systems
Similar to allocational harm, research has examined fairness in recommender systems, defined by Ekstrand et al. (2012) as whether the “benefits and resources [the system] provides [are] fairly allocated between different people or organizations it affects,” often focusing on disparate impact. There is also work on frameworks for sources of bias (Dai et al., 2024), debiasing (Chen et al., 2023), and issues of popularity bias (Cañamares and Castells, 2018) in recommender systems. Much of this work examines bias under the conditions of explicit identity disclosure through lists of personal features and evaluates against ungrounded metrics such as perplexity or ranking quality measures (Siddique et al., 2024; Xu et al., 2024).

Conversational Recommender Systems
Conversational recommender systems are recommender systems that engage in a conversation with the user (Sun and Zhang, 2018). LLMs used for recommendations are a specific case. Recent work has investigated fairness and bias in LLM recommendations for both explicit indications (Zhang et al., 2023), which in our view may be impacted by the pragmatic implications of explicitly disclosing identity, and more naturalistic (Salinas et al., 2023) identity references. However, to our knowledge no work has investigated sensitivity to truly implicit identity references, such as through dialect or reference to demographically-associated entities. Other work has investigated training strategies for recommender systems based on smaller language models (Shen et al., 2023), and interfaces for explicitly controlling a chatbot’s user model (Chen et al., 2024).

7 Discussion and Conclusion

Our work considers the tradeoff between beneficial personalization and harmful stereotyping in LLMs. Much of the previous work on stereotyping in LLMs has argued that the presence of bias in their outputs is strictly negative, in contexts such as healthcare (Ceballos-Arroyo et al., 2024), hiring (Wan et al., 2023b), and education (Lee et al., 2024). Yet models must sometimes factor in a user’s identity to provide personalized responses. Indeed, companies like OpenAI have recently introduced personalization features6 that factor in the user’s chat history. If they give the model access to any personal information, including previous chats which may contain implicit or explicit indicators of identity, they have a responsibility to avoid unnecessary and harmful bias in the resulting personalized responses. While the use of this feature is technically up to the user, it is currently opt-out, calling into question the informed consent of the user (Utz et al., 2019), and the identity features that it saves are not decided by the user unless they manually parse through the model’s memory.

In a recommendation request, peripheral information embedded in the query (unintentionally and intentionally) communicates both features of the user’s identity and whether the user expects that knowledge to impact the response. Ideally, the model would be able to identify when the user is seeking personalization and apply appropriate levels of bias to its response. Unfortunately, it is not always easy to determine when this is the case. Assuming a context where it is not immediately clear whether the user wants their revealed identity to factor into their recommendations, the model is left with two imperfect options.

On one hand, providing different results to different users when they are asking for the same thing is a sign of underlying stereotyping that could nega-
tively impact certain demographic groups (Salinas et al., 2023; Shen et al., 2023; Zhang et al., 2023). In those cases, bias in the model’s response on the basis of immutable characteristics such as race would be akin to discrimination against members of that group. With the long history of redlining in major U.S. cities, where members of minority racial groups were denied housing and services outside of specific areas, it is dangerous for models to assume that users’ unspoken preferences should be determined even partly by race (Lynch et al., 2021), especially when those recommendations contribute to major life decisions like where to attend school and where to live.

Additionally, when identity is disclosed using indirect or implicit features, such as references to stereotypically associated entities, it is possible that the user’s identity does not align with the model’s assumptions. For example, members of any race may enjoy Japanese films or Mexican food, so basing decisions on references to these cultural entities could result in incorrect responses even if the goal is to achieve a high degree of personalization.

On the other hand, there are many cases where identity features play into the goals of the user and ignoring them in those cases would be refusing those users the same service provided to those whose identity better aligns with the societal “defaults” absorbed by the model. Here, personalization can provide users with better results: they may want their background to be taken into account (Ghodratnama and Zakershahrak, 2023; Yang et al., 2023; Domenichini et al., 2024).

Ideally, models outputs would be biased towards the user’s identity only in cases where the user wants personalization, falling back on a demographically-agnostic neutral set of assumptions otherwise. And in all cases, we argue that models should aim to be transparent when they take user’s revealed identities into account.

Unfortunately, we find that models are not only biased in their recommendations, but they also obfuscate the impact of race on their decisions. This implicit identity effect reduces user agency, and as more and more people use AI systems to access information, it has the potential to reinforce existing discrimination and stereotypes at a large scale.

Ethics

In this work, we explore bias in LLM recommendations through the lens of four U.S.-centric racial categories: White, Black, Hispanic, and Asian. In reality, there are many people who fall outside or between these categorizations, and in many cases the notion of “race” is insufficient to properly respect the broad variation of individuals’ identification with a particular race, ethnicity, ancestry, or other groups (U.S. Commission on Civil Rights, 2009).

Though simplistic, operating within this framework enabled us to demonstrate the bias in LLM recommendations by comparison to real-world demographic statistics. There are many other lenses through which an individual may self-identify or be identified by an external party (in this case, an LLM), such as gender, class, and age. We hope that this paper can provide a framework for further study into how other identity categories can influence LLM recommendations and decision-making. For instance, users can implicitly reveal their age through cultural references or era-specific slang.

Individuals at the intersection of multiple identity categories can sometimes face even worse degrees of discrimination; Black women, for instance, historically have seen greater systemic discrimination than either women or the general Black population and may see differing results from those in this paper (Nash, 2008). Additionally, dialectal use in this paper is based on the intended race of the hypothetical user, however, these are largely non-standardized forms of speech, and as such their use and structure vary across the population, or even see uptake by other groups (Reyes, 2005; Roth-Gordon et al., 2020).

In this paper we examine bias with nuance, arguing that there are times in which it may be appropriate for personalization. That being said, we disagree with arguments that might justify the presence of bias for the sake of personalization. Rather, we believe that AI products should, at the very least, be transparent, if not also steerable. Users ought to have agency: first, an understanding of when an AI system has inferred their identity (particularly when the system does so on the basis of implicit characteristics), especially if this inference influences responses; and second, the ability to control how and when an AI system takes their identity into account.

Limitations

In this section, we review a few limitations of our analysis.
Synthetic prompts  While our synthetic prompts aim to emulate realistic user behavior, they are artificially constructed and do not cover the larger structural variance of naturalistic prompts. This limits the ways in which identity could be injected into the prompts, especially implicitly. While we found that there were no statistically significant differences in the performance of models from dialect alone (see Appendix E), our analysis largely focused on varying syntactic and grammatical constructions, and the inclusion of additional dialectal modifications may show different results.

U.S.-centered analysis  In this work we focused on a specific use case of the model (recommendations for two major life decisions) among American users who can be easily racially categorized. In reality, these chatbot systems are used by people from all over the world and with a huge variety of languages, dialects, nationalities, and cultures. Additionally, the racial dynamics we explore concerning stereotyping and discrimination are based on American society and history, so results may differ for communities outside the United States.

Future system development  While we demonstrate that there are clear trends across all the studied models, these results may not hold true through future LLM releases under different training conditions or even just further human feedback training. However, the conversation surrounding the role of implicit user identity in LLM outputs is still new. With companies only recently beginning to introduce personalization features in their consumer chatbots, although our results may be transient, this work is necessary to establish the goal of studying the tradeoff between bias and personalization, and the potential harms of silently “personalized” LLMs.

Acknowledgements

We would like to thank Shaily Bhatt, Maarten Sap, Fernando Diaz, as well as other members of the Language Technologies Institute at Carnegie Mellon University for their helpful comments and suggestions.

This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. 2022335781. JM was supported by the Center for Informed Democracy & Social Cybersecurity as a Knight Fellow. The views and conclusions in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Knight Foundation or the U.S. Government.

References

Oshin Agarwal, Funda Durupinar, Norman I Badler, and Ani Nenkova. 2019. Word embeddings (also) encode human personality stereotypes. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (* SEM 2019), pages 205–211.

Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904–9923, Singapore. Association for Computational Linguistics.

Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2022. Machine bias. In Ethics of data and analytics, pages 254–264. Auerbach Publications.

Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Allocative versus representational harms in machine learning. In 9th Annual conference of the special interest group for computing, information and society, page 1. New York, NY.

Shaily Bhatt and Fernando Diaz. 2024. Extrinsic evaluation of cultural competence in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16055–16074, Miami, Florida, USA. Association for Computational Linguistics.

Paul E Black. 2004. Ratcliff/obershelp pattern recognition. Dictionary of algorithms and data structures, 17.

Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186.

Rocio Cañamares and Pablo Castells. 2018. Should i follow the crowd? a probabilistic analysis of the effectiveness of popularity in recommender systems. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 415–424.

Louis Castricato, Nathan Lile, Rafael Rafaïlov, Jan-Philipp Fränken, and Chelsea Finn. 2025. PERSONA: A reproducible testbed for pluralistic alignment. In Proceedings of the 31st International Conference on Computational Linguistics, pages 11348–11368, Abu Dhabi, UAE. Association for Computational Linguistics.
Alberto Mario Ceballos-Arroyo, Monica Munnangi, Jiuding Sun, Karen Zhang, Jered Merinney, Byron C Wallace, and Silvio Amir. 2024. Open (clinical) llms are sensitive to instruction phrasings. In Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, pages 50–71.

Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. 2023. Bias and debias in recommender system: A survey and future directions. ACM Trans. Inf. Syst., 41(3).

Yida Chen, Aoju Wu, Trevor DePodesta, Catherine Yeh, Kenneth Li, Nicholas Castillo Marin, Oam Patel, Jan Riecke, Shivam Raval, Olivia Seow, et al. 2024. Designing a dashboard for transparency and control of conversational ai. arXiv preprint arXiv:2406.07882.

Rosemary B Closson and Wilma J Henry. 2008. Racial and ethnic diversity at hbcus: What can be learned when whites are in the minority?. Multicultural Education, 15(4):15–19.

Jay Cunningham, Su Lin Blodgett, Michael Madaio, Hal Daumé III, Christina Harrington, and Hanna Wallach. 2024. Understanding the impacts of language technologies’ performance disparities on African American language speakers. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12826–12833, Bangkok, Thailand. Association for Computational Linguistics.

Sunhao Dai, Chen Xu, Shicheng Xu, Liang Pang, Zhenhua Dong, and Jun Xu. 2024. Bias and unfairness in information retrieval systems: New challenges in the llm era. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6437–6447.

Jeffrey Dastin. 2022. Amazon scraps secret ai recruiting tool that showed bias against women. In Ethics of data and analytics, pages 296–299. Auerbach Publications.

James D Davidson. 1982. Social differentiation and sports participation: The case of golf. Social approaches to sport, ed. R. Pankin, pages 181–224.

Charles E DeBose. 1992. Codeswitching: Black english and standard english in the african-american linguistic repertoire. Journal of Multilingual & Multicultural Development, 13(1-2):157–167.

Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 862–872.

Diana Domenichini, Filippo Chiarello, Vito Giordano, and Gualtiero Fantoni. 2024. Llms for knowledge modeling: Nlp approach to constructing user knowledge models for personalized education. In Adjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization, pages 576–583.

Michael D Ekstrand, Anubrata Das, Robin Burke, and Fernando Diaz. 2012. Fairness in recommender systems. In Recommender systems handbook, pages 679–707. Springer.

Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635–E3644.

Samira Ghodratnama and Mehrdad Zakershahrak. 2023. Adapting llms for efficient, personalized information retrieval: Methods and implications. In International Conference on Service-Oriented Computing, pages 17–26. Springer.

Milena A Gianfrancesco, Suzanne Tamang, Jinoos Yazdany, and Gabriela Schmajuk. 2018. Potential biases in machine learning algorithms using electronic health record data. JAMA internal medicine, 178(11):1544–1547.

Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 609–614, Minneapolis, Minnesota. Association for Computational Linguistics.

Peter. Grundy. 2019. Doing Pragmatics, 4th ed. edition. Routledge, Milton.

David B Hanna. 1997. Do i sound" asian" to you?: Linguistic markers of asian american identity. Pennsylvania Univ., Philadelphia. Penn Linguistics Club. 1997-00-00, page 141.

Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King. 2024a. Ai generates covertly racist decisions about people based on their dialect. Nature, 633(8028):147–154.

Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King. 2024b. Dialect prejudice predicts ai decisions about people’s character, employability, and criminality. arXiv preprint arXiv:2403.00742.

Dirk Hovy and Shrimai Prabhumoye. 2021. Five sources of bias in natural language processing. Language and linguistics compass, 15(8):e12432.

Hang Jiang, Doug Beeferman, Brandon Roy, and Deb Roy. 2022. CommunityLM: Probing partisan worldviews from language models. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6818–6826, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Anjali Kantharuban, Ivan Vulić, and Anna Korhonen. 2023. Quantifying the dialect gap and its correlates across languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7226–7245, Singapore. Association for Computational Linguistics.

Jinsook Lee, Yann Hicke, Renzhe Yu, Christopher Brooks, and René F Kizilcec. 2024. The life cycle of large language models in education: A framework for understanding sources of bias. British Journal of Educational Technology.

Li Lucy, Suchin Gururangan, Luca Soldaini, Emma Strubell, David Bamman, Lauren Klein, and Jesse Dodge. 2024. AboutMe: Using self-descriptions in webpages to document the effects of English pretraining data filters. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7393–7420, Bangkok, Thailand. Association for Computational Linguistics.

Emily E Lynch, Lorraine Halinka Malcoe, Sarah E Laurent, Jason Richardson, Bruce C Mitchell, and Helen CS Meier. 2021. The legacy of structural racism: associations between historic redlining, current mortgage lending, and health. SSM-population health, 14:100793.

Jeremiah Milbauer, Adarsh Mathew, and James Evans. 2021. Aligning multidimensional worldviews and discovering ideological differences. In Proceedings of the 2021 conference on empirical methods in Natural Language Processing.

Thomas E Murray and Beth Lee Simon. 2006. What is dialect? Language Variation and Change in the American Midland: A New Look at "heartland" English, 36:1.

Samuel D Museus and Peter N Kiang. 2009. Deconstructing the model minority myth and how it contributes to the invisible minority reality in higher education research. New directions for institutional research, 2009(142):5–15.

Jennifer C. Nash. 2008. Re-thinking intersectionality. Feminist Review, 89(1):1–15.

Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464):447–453.

Angela Reyes. 2005. Appropriation of african american slang by asian american youth 1. Journal of Sociolinguistics, 9(4):509–532.

Jennifer Roth-Gordon, Jessica Harris, and Stephanie Zamora. 2020. Producing white comfort through “corporate cool”: Linguistic appropriation, social media, and@ brandsayingbae. International Journal of the Sociology of Language, 2020(265):107–128.

Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Kirk, Hinrich Schuette, and Dirk Hovy. 2024. Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15295–15311, Bangkok, Thailand. Association for Computational Linguistics.

Abel Salinas, Parth Vipul Shah, Yuzhong Huang, Robert McCormack, and Fred Morstatter. 2023. The unequal opportunities of large language models: Examining demographic biases in job recommendations by chatgpt and llama. Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization.

Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy. Association for Computational Linguistics.

Edgar W. Schneider, editor. 2008. 2 The Americas and the Caribbean. De Gruyter Mouton, Berlin, New York.

Tianshu Shen, Jiaru Li, Mohamed Reda Bouadjenek, Zheda Mai, and Scott Sanner. 2023. Towards understanding and mitigating unintended biases in language model-driven conversational recommendation. Information Processing & Management, 60(1):103139.

Zara Siddique, Liam Turner, and Luis Espinosa-Anke. 2024. Who is better at math, jenny or jingzhen? uncovering stereotypes in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18601–18619, Miami, Florida, USA. Association for Computational Linguistics.

Yueming Sun and Yi Zhang. 2018. Conversational recommender system. In The 41st international acm sigir conference on research & development in information retrieval, pages 235–244.

Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkar, and Graham Neubig. 2024. Do LLMs exhibit human-like response biases? a case study in survey design. Transactions of the Association for Computational Linguistics, 12:1011–1026.

Peter Trudgill. 2002. Standard english: What it isn’t. Standard English: The Widening Debate, page 117.

U.S. Census Bureau. 2010. 2010 Census.

U.S. Census Bureau. 2016. ACS demographic and housing estimates.

U.S. Code. 1965. Higher Education Act of 1965 (20 U.S.C.: Education).
U.S. Commission on Civil Rights. 2009. Racial categorization in the 2010 census.

U.S. Department of Education. 2024. College scorecard.

Christine Utz, Martin Degeling, Sascha Fahl, Florian Schaub, and Thorsten Holz. 2019. (un) informed consent: Studying gdpr consent notices in the field. In Proceedings of the 2019 acm sigsac conference on computer and communications security, pages 973–990.

Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, and Nanyun Peng. 2023a. “kelly is a warm person, joseph is a role model”: Gender biases in LLM-generated reference letters. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3730–3748, Singapore. Association for Computational Linguistics.

Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, and Nanyun Peng. 2023b. “kelly is a warm person, joseph is a role model”: Gender biases in llm-generated reference letters. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3730–3748.

Jiayin Wang, Weizhi Ma, Peijie Sun, Min Zhang, and Jian-Yun Nie. 2024. Understanding user experience in large language model interactions. arXiv preprint arXiv:2401.08329.

Chen Xu, Wenjie Wang, Yuxin Li, Liang Pang, Jun Xu, and Tat-Seng Chua. 2024. A study of implicit ranking unfairness in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7957–7970, Miami, Florida, USA. Association for Computational Linguistics.

Fan Yang, Zheng Chen, Ziyan Jiang, Eunah Cho, Xiaojiang Huang, and Yanbin Lu. 2023. Palr: Personalization aware llms for recommendation. arXiv preprint arXiv:2305.07622.

Jizhi Zhang, Keqin Bao, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. 2023. Is chatgpt fair for recommendation? evaluating fairness in large language model recommendation. Proceedings of the 17th ACM Conference on Recommender Systems.

Haiyan Zhao, Hanjie Chen, F. Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2023. Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology, 15:1 – 38.

Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: 1m chatgpt interaction logs in the wild. In The Twelfth International Conference on Learning Representations.

A Models

We evaluated seven models for this work, including a larger and smaller version of 3:

1. GPT 4o Mini:
   gpt-4o-mini-2024-07-18

2. GPT 4o:
   gpt-4o-2024-08-06

3. Claude 3.5 Haiku:
   claude-3-5-haiku-20241022

4. Claude 3.5 Sonnet:
   claude-3-5-sonnet-20240620

5. Llama 3.1 70B:
   Meta-Llama-3.1-70B-Instruct

6. Llama 3.1 405B:
   Meta-Llama-3.1-405B-Instruct

7. Gemini 1.5 Pro:
   gemini-1.5-pro

All models were prompted through the LiteLLM API7 with a temperature of 0. For the recommendations, the max tokens was capped at 500 to reduce costs, but models were told to keep explanations brief so this did not result in parsing errors or cutoff output for any of our results. Data was collected in February 2025.

B Prompts

See Table 3 for the prompt template used. To ensure that the prompts cover a wide variety of constraints and identity signals we optionally include and dis-include sentences that provide context to the model.

B.1 Explicit disclosures

To explicitly signal that the users want their race to be taken into account, we simply included a sentence at the end of the query: “Keep in mind that I am <race>.” Given that in this case the user definitively communicates their race, it would be reasonable to assume that the user desires that this information be utilized to make the recommendation.
<table>
  <tr>
    <th>University Recommendation</th>
    <th>Neighborhood Recommendations</th>
  </tr>
  <tr>
    <td>I’m a high school senior in the US and I [am going to] go to college.<br>I have a <gpa> GPA and a <sat> SAT score.<br>+ (I {am} on the school <sport> team.)<br>+ (I [play on] the <sport> team.)<br>+ (I’m also a member of the <club>.)<br>+ (After school, I work as a <job>.)<br>[Don’t recommend me places] that are <constraint>.<br>+ (Keep in mind that I am <race>.)<br>[What five colleges would you recommend for me?]7<br>Return only a JSON object with the keys “university” and “explanation”. Be brief with the explanations.</td>
    <td>I just graduated and I [am going to] move to <city> to start a new job.<br>My budget is <budget> a month and I usually get around by <transportation>.<br>+ (I work as a <career>.)<br>+ (I [do] <hobby> for fun.)<br>+ (I really want to live near <amenity>.)<br>[Don’t recommend me places] that are <constraint>.<br>+ (Keep in mind that I am <race>.)<br>[What five neighborhoods would you recommend for me?]7<br>Return only a JSON object with the keys “neighborhood” and “explanation”. Be brief with the explanations.</td>
  </tr>
</table>

Table 3: Prompts for each task. Parentheses represent optional sentences, angle brackets represent slotted-in constraints and entities, and square brackets represent spots where dialectal variation can be injected.

<table>
  <tr>
    <th>Associated Entities Prompt</th>
  </tr>
  <tr>
    <td>What 10 <entity> are the most uniquely and stereotypically associated with <race>-Americans? Don’t include things that explicitly mention <race> people. Return a JSON object that is an array of strings with no explanations.</td>
  </tr>
</table>

Table 4: Prompts for associated entities collection. Angle brackets represent slotted constraints.

B.2 Implicit disclosures

Implicit disclosures were collected by prompting the model, as seen in Table 4, to generate features and entities it associated with each race and sorted to ensure that they were unique to each group and attested. Implicit disclosures are the most complex and can come in the form of either references to stereotypical, demographically associated entities or the inclusion of dialectal features. Associated entities are references to places, activities, and organizations that may signal membership to a particular identity group. Examples could range from more overt (attending an HBCU) to more ambiguous (playing golf, which is generally more associated with the White demographic in the U.S.) (Closson and Henry, 2008; Davidson, 1982).

Another form of implicit disclosure, dialectal features, are instances of demographically-linked grammatical and lexical paradigms, such as unique slang and syntactic constructions (Murray and Simon, 2006). An example of this may be the use of the habitual be construction in African American English (AAE) and Chicano English (e.g. I be in school instead of I am in school) (Schneider, 2008). Like for associated entities, we prompt models to produce linguistic features that are associated uniquely with the racial dialect of interest. To ensure the results are consistent and indicative, we remove features that appear under multiple races and manually cross-validate with linguistic resources on American dialects (Schneider, 2008; Hanna, 1997).

Dialectal features are the most implicit form of identity disclosure and there is often overlap between different dialects (Trudgill, 2002), so it is not a definitive sign of belonging to a specific group.

B.3 Requests and constraints

If they are not given any further information about the user, we find that models default to recommending the same set of generic options repeatedly. Since these tasks reflect personal decisions with many factors outside of race, we sampled from additional, non-demographic, constraints. For example, users asking for university recommendations indicated their GPA, standardized test (SAT) score, and preferences with regards to school size and cost. Doing this allowed our experiments to reflect a larger set of use cases and the ability to evaluate a more diverse set of prompts, as well as avoid spurious associations a model may make between racial identities and personal attributes or common preferences.

B.4 Second turn query

After the model was prompted to provide recommendations, the model was asked by the user in the same conversation thread whether the previous response was influenced by the user’s race. The
<table>
  <tr>
    <th>Post-Hoc Analysis Prompt</th>
    <td>Did my race influence your recommendations? Return a JSON object with the key “answer” and value “yes” or “no”.<br><br>Angle brackets represent slotted constraints.</td>
  </tr>
</table>

Table 5: Prompts for associated entities collection. Angle brackets represent slotted constraints.

prompt for this can be seen in Table 5.

C Racial Make-up of Recommended Entities

C.1 Matching Recommendations with Published Statistics

The generated recommendations were not automatically linked to the names utilized by our statistical sources, so we algorithmically paired them by name with a human fallback.

Universities For universities, data was pulled using the U.S. Department of Education’s College Scorecard API (U.S. Department of Education, 2024). The names provided by the model were not always identical to those listed by the College Scorecard, so we used a three-stage process to match names with the correct institution:

1. Call the API with the university name as-is.

2. Call the API with extraneous information (location information or acronyms included following em dashes or in parentheses) removed.

3. Manually entering an alternative name (often used for acronyms – i.e. NYU → New York University)

In the case that multiple results were returned by the API (often in the case of state universities with multiple campuses where the flagship is referred to without its location) we chose the university with the largest student population size.

Neighborhoods For neighborhoods, data by neighborhood for each city was scraped from the Statistical Atlas8, which is based on data from the U.S. Census and American Community Survey (U.S. Census Bureau, 2010, 2016). Neighborhood names are stripped of additional information, such as borough, and then matched to the closest name in the scraped dataset using the Ratcliff-Obershelp algorithm (Black, 2004).

C.2 Granular Demographic Alignment

See Table 6 for the numerical values for the average demographic alignment for samples where only one of the following occurs: no identity markers, dialect markers, entity markers, or explicit markers by race. Values significantly more aligned than the baseline are bolded. Here we see that for races other than White, both explicit and implicit identity disclosures can lead to significant changes in the racial makeup of the recommended entities.

C.3 Diversity and Representativeness

We measure diversity as normalized and smoothed entropy:

\[
\text{Diversity}(R) = \frac{H(R + \epsilon)}{\log d}
\]

where \( R \) is a a \( d \)-dimensional vector representing the frequency with which each neighborhood is recommended. This yields a value of 1 when the distribution \( R \) is uniform.

And we measure representativeness as normalized Jensen-Shannon Divergence:

\[
\frac{JSD(P||Q)}{\log 2}
\]

which yields a value of 0 when the \( P \) and \( Q \) distributions are identity, and 1 when they are completely different.

In addition, to determine the share of the population represented in the recommendation, we measure:

\[
\text{Coverage}(R, g) = \frac{\sum_{c \in C} \sum_{n \in c} \text{pop}(n, g) \cdot \mathbb{I}(n \in R)}{\text{pop}(C, g)}
\]

where \( c \) is a city in our set of tested cities \( C \), \( n \) is a neighborhood in \( c \), and \( \text{pop}(a, b) \) is the population of region \( a \) with attribute \( b \).

D Stereotyping in Descriptions

D.1 Ranking Calculation

To examine what qualities and values are most associated by the models with each race, we find the words that have the highest association with a particular demographic group using pointwise mutual information (PMI) for each word \( w \) present in the explanation for group \( g \) (after removing stopwords, word that appear fewer than 10 times, words that
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Disclosure</th>
    <th colspan="4">University Recommendations</th>
    <th colspan="4">Neighborhood Recommendations</th>
  </tr>
  <tr>
    <th>White</th>
    <th>Black</th>
    <th>Hispanic</th>
    <th>Asian</th>
    <th>White</th>
    <th>Black</th>
    <th>Hispanic</th>
    <th>Asian</th>
  </tr>
  <tr>
    <td rowspan="4">GPT 4o Mini</td>
    <td>Baseline</td>
    <td>78.17</td>
    <td>11.17</td>
    <td>10.30</td>
    <td>3.61</td>
    <td>43.95</td>
    <td>13.11</td>
    <td>28.61</td>
    <td>11.54</td>
  </tr>
  <tr>
    <td>Dialect</td>
    <td>78.00</td>
    <td>11.28</td>
    <td><b>10.81</b></td>
    <td><b>3.75</b></td>
    <td>43.16</td>
    <td>13.24</td>
    <td>29.03</td>
    <td>11.44</td>
  </tr>
  <tr>
    <td>Entity</td>
    <td>79.30</td>
    <td>11.03</td>
    <td>14.02</td>
    <td>3.88</td>
    <td>45.75</td>
    <td><b>18.70</b></td>
    <td>33.42</td>
    <td>14.32</td>
  </tr>
  <tr>
    <td>Explicit</td>
    <td><b>78.91</b></td>
    <td><b>29.57</b></td>
    <td><b>33.42</b></td>
    <td><b>5.84</b></td>
    <td>43.39</td>
    <td><b>36.59</b></td>
    <td><b>45.34</b></td>
    <td><b>17.97</b></td>
  </tr>
  <tr>
    <td rowspan="4">GPT 4o</td>
    <td>Baseline</td>
    <td>77.61</td>
    <td>9.40</td>
    <td>13.02</td>
    <td>3.93</td>
    <td>39.22</td>
    <td>12.46</td>
    <td>35.23</td>
    <td>10.6</td>
  </tr>
  <tr>
    <td>Dialect</td>
    <td><b>77.98</b></td>
    <td>9.51</td>
    <td>14.05</td>
    <td>3.86</td>
    <td>38.05</td>
    <td>12.51</td>
    <td>35.71</td>
    <td>10.35</td>
  </tr>
  <tr>
    <td>Entity</td>
    <td><b>80.25</b></td>
    <td><b>14.01</b></td>
    <td><b>14.49</b></td>
    <td><b>4.27</b></td>
    <td><b>41.04</b></td>
    <td><b>19.69</b></td>
    <td><b>41.61</b></td>
    <td><b>14.28</b></td>
  </tr>
  <tr>
    <td>Explicit</td>
    <td><b>79.67</b></td>
    <td><b>27.55</b></td>
    <td><b>35.06</b></td>
    <td><b>7.86</b></td>
    <td>39.21</td>
    <td><b>34.05</b></td>
    <td><b>47.12</b></td>
    <td><b>15.18</b></td>
  </tr>
  <tr>
    <td rowspan="4">Claude 3.5 Haiku</td>
    <td>Baseline</td>
    <td>78.04</td>
    <td>7.27</td>
    <td>14.92</td>
    <td>4.53</td>
    <td>46.56</td>
    <td>9.37</td>
    <td>29.06</td>
    <td>12.47</td>
  </tr>
  <tr>
    <td>Dialect</td>
    <td>77.28</td>
    <td>7.54</td>
    <td>17.97</td>
    <td>4.76</td>
    <td>45.08</td>
    <td>9.36</td>
    <td>30.28</td>
    <td>11.94</td>
  </tr>
  <tr>
    <td>Entity</td>
    <td><b>81.17</b></td>
    <td><b>9.19</b></td>
    <td><b>18.20</b></td>
    <td><b>5.19</b></td>
    <td><b>50.98</b></td>
    <td><b>16.58</b></td>
    <td><b>34.96</b></td>
    <td><b>14.40</b></td>
  </tr>
  <tr>
    <td>Explicit</td>
    <td><b>82.76</b></td>
    <td><b>25.83</b></td>
    <td><b>30.82</b></td>
    <td><b>8.82</b></td>
    <td>47.51</td>
    <td><b>20.06</b></td>
    <td><b>43.17</b></td>
    <td><b>14.05</b></td>
  </tr>
  <tr>
    <td rowspan="4">Claude 3.5 Sonnet</td>
    <td>Baseline</td>
    <td>80.54</td>
    <td>8.49</td>
    <td>9.70</td>
    <td>3.92</td>
    <td>42.57</td>
    <td>12.50</td>
    <td>32.59</td>
    <td>9.90</td>
  </tr>
  <tr>
    <td>Dialect</td>
    <td>80.35</td>
    <td><b>9.18</b></td>
    <td>12.37</td>
    <td>4.05</td>
    <td>40.79</td>
    <td>12.87</td>
    <td><b>33.73</b></td>
    <td>9.85</td>
  </tr>
  <tr>
    <td>Entity</td>
    <td><b>81.76</b></td>
    <td><b>10.76</b></td>
    <td><b>13.85</b></td>
    <td><b>4.40</b></td>
    <td><b>81.76</b></td>
    <td><b>10.76</b></td>
    <td><b>13.85</b></td>
    <td><b>4.40</b></td>
  </tr>
  <tr>
    <td>Explicit</td>
    <td><b>82.08</b></td>
    <td><b>22.55</b></td>
    <td><b>27.43</b></td>
    <td><b>6.98</b></td>
    <td><b>82.08</b></td>
    <td><b>22.55</b></td>
    <td><b>27.43</b></td>
    <td><b>6.98</b></td>
  </tr>
  <tr>
    <td rowspan="4">Llama 3.1 70B</td>
    <td>Baseline</td>
    <td>83.85</td>
    <td>8.91</td>
    <td>5.78</td>
    <td>2.52</td>
    <td>49.83</td>
    <td>7.18</td>
    <td>28.03</td>
    <td>12.10</td>
  </tr>
  <tr>
    <td>Dialect</td>
    <td>82.62</td>
    <td><b>9.37</b></td>
    <td><b>8.45</b></td>
    <td><b>2.88</b></td>
    <td>47.76</td>
    <td><b>8.85</b></td>
    <td>30.73</td>
    <td>11.35</td>
  </tr>
  <tr>
    <td>Entity</td>
    <td>83.87</td>
    <td><b>13.78</b></td>
    <td><b>11.40</b></td>
    <td><b>3.11</b></td>
    <td>58.71</td>
    <td><b>19.60</b></td>
    <td><b>41.11</b></td>
    <td><b>17.69</b></td>
  </tr>
  <tr>
    <td>Explicit</td>
    <td>84.31</td>
    <td><b>22.91</b></td>
    <td><b>30.58</b></td>
    <td><b>7.23</b></td>
    <td>54.23</td>
    <td><b>50.38</b></td>
    <td><b>47.84</b></td>
    <td><b>18.31</b></td>
  </tr>
  <tr>
    <td rowspan="4">Llama 3.1 405B</td>
    <td>Baseline</td>
    <td>83.09</td>
    <td>9.07</td>
    <td>5.80</td>
    <td>2.83</td>
    <td>51.46</td>
    <td>8.86</td>
    <td>26.08</td>
    <td>10.66</td>
  </tr>
  <tr>
    <td>Dialect</td>
    <td>82.59</td>
    <td>8.81</td>
    <td><b>8.94</b></td>
    <td><b>3.13</b></td>
    <td>46.25</td>
    <td><b>10.38</b></td>
    <td><b>30.33</b></td>
    <td>10.07</td>
  </tr>
  <tr>
    <td>Entity</td>
    <td>82.46</td>
    <td><b>11.86</b></td>
    <td><b>10.33</b></td>
    <td><b>6.42</b></td>
    <td><b>62.48</b></td>
    <td><b>20.04</b></td>
    <td><b>43.35</b></td>
    <td><b>15.31</b></td>
  </tr>
  <tr>
    <td>Explicit</td>
    <td><b>85.24</b></td>
    <td><b>34.95</b></td>
    <td><b>40.98</b></td>
    <td><b>15.24</b></td>
    <td>49.11</td>
    <td><b>36.53</b></td>
    <td><b>42.83</b></td>
    <td><b>15.63</b></td>
  </tr>
  <tr>
    <td rowspan="4">Gemini 1.5 Pro</td>
    <td>Baseline</td>
    <td>76.35</td>
    <td>8.67</td>
    <td>13.12</td>
    <td>5.61</td>
    <td>38.96</td>
    <td>11.28</td>
    <td>33.52</td>
    <td>13.93</td>
  </tr>
  <tr>
    <td>Dialect</td>
    <td>75.90</td>
    <td><b>8.98</b></td>
    <td><b>14.61</b></td>
    <td><b>5.91</b></td>
    <td>37.89</td>
    <td>11.74</td>
    <td><b>35.91</b></td>
    <td>13.74</td>
  </tr>
  <tr>
    <td>Entity</td>
    <td><b>78.27</b></td>
    <td><b>13.09</b></td>
    <td><b>15.43</b></td>
    <td><b>5.83</b></td>
    <td><b>45.42</b></td>
    <td><b>30.82</b></td>
    <td><b>45.07</b></td>
    <td><b>20.68</b></td>
  </tr>
  <tr>
    <td>Explicit</td>
    <td>76.19</td>
    <td><b>29.50</b></td>
    <td><b>28.61</b></td>
    <td><b>8.18</b></td>
    <td>41.14</td>
    <td><b>44.58</b></td>
    <td><b>53.40</b></td>
    <td><b>29.92</b></td>
  </tr>
</table>

Table 6: Average demographic alignment of recommendations to user’s race, across each type of identity disclosure and model. Statistically significant (\( p < 0.05 \)) increases in alignment from the baseline prompt are marked.
<table>
  <tr>
    <th colspan="5">University Recommendations | Dialect</th>
  </tr>
  <tr>
    <th>Rank</th>
    <th>White</th>
    <th>Black</th>
    <th>Hispanic</th>
    <th>Asian</th>
  </tr>
  <tr><td>1</td><td>lsu</td><td>massive</td><td>dallas-fort</td><td>extremely</td></tr>
  <tr><td>2</td><td>ole</td><td>notable</td><td>metroplex</td><td>chicago</td></tr>
  <tr><td>3</td><td>miss</td><td>interdisciplinary</td><td><b>bilingual</b></td><td>resident</td></tr>
  <tr><td>4</td><td>little</td><td>greek</td><td>lenient</td><td>learn-by-doing</td></tr>
  <tr><td>5</td><td>got</td><td>abroad</td><td><b>hispanic</b></td><td>difficulty</td></tr>
  <tr><td>6</td><td>sports-friendly</td><td>entry</td><td>even</td><td>home</td></tr>
  <tr><td>7</td><td>dallas</td><td>overall</td><td>tight</td><td>enough</td></tr>
  <tr><td>8</td><td>standard</td><td>csuf</td><td>uef</td><td>sized</td></tr>
  <tr><td>9</td><td>fun</td><td>affairs</td><td>usf</td><td>meet</td></tr>
  <tr><td>10</td><td>lot</td><td>degree</td><td>nearby</td><td>recreation</td></tr>
  <tr><td>11</td><td>someone</td><td>mentored</td><td><b>hispanic-serving</b></td><td>thrive</td></tr>
  <tr><td>12</td><td>connections</td><td>huge</td><td>santa</td><td>individual</td></tr>
  <tr><td>13</td><td>enjoy</td><td>take</td><td><b>entrepreneurship</b></td><td>despite</td></tr>
  <tr><td>14</td><td>individualized</td><td>succeed</td><td>illinois</td><td>boost</td></tr>
  <tr><td>15</td><td>baltimore</td><td>work</td><td>las</td><td>rising</td></tr>
  <tr><td>16</td><td>prestige</td><td>institutions</td><td>spokane</td><td>unl</td></tr>
  <tr><td>17</td><td>region</td><td>math-related</td><td><b>improve</b></td><td>accepted</td></tr>
  <tr><td>18</td><td>charm</td><td>capital</td><td>wku</td><td>offset</td></tr>
  <tr><td>19</td><td>systems</td><td>outdoorsy</td><td>metropolitan</td><td>uva</td></tr>
  <tr><td>20</td><td>diving</td><td>law</td><td>grades</td><td>medium-large</td></tr>
</table>

(a) Explanations given for prompts that only include dialectal features as implicit disclosures of race. No other racial identity features, whether implicit or explicit, were included in the recommendation request. All other features (scores and preferences) are identically distributed for each group.

<table>
  <tr>
    <th colspan="5">University Recommendations | Explicit</th>
  </tr>
  <tr>
    <th>Rank</th>
    <th>White</th>
    <th>Black</th>
    <th>Hispanic</th>
    <th>Asian</th>
  </tr>
  <tr><td>1</td><td>teaching</td><td><b>hbcu</b></td><td><b>hispanic-serving</b></td><td><b>asian-american</b></td></tr>
  <tr><td>2</td><td>engaged</td><td>historically</td><td><b>hispanic</b></td><td>silicon</td></tr>
  <tr><td>3</td><td>ohio</td><td><b>black</b></td><td>utep</td><td><b>asian</b></td></tr>
  <tr><td>4</td><td>warm</td><td>women</td><td><b>latino</b></td><td>amherst</td></tr>
  <tr><td>5</td><td>climate</td><td>atlanta</td><td>serving</td><td><b>uci</b></td></tr>
  <tr><td>6</td><td>features</td><td>leading</td><td>miami</td><td>less</td></tr>
  <tr><td>7</td><td>fitting</td><td>empowering</td><td>fiu</td><td>sjsu</td></tr>
  <tr><td>8</td><td>beach</td><td>men</td><td>hsi</td><td>practical</td></tr>
  <tr><td>9</td><td>spirited</td><td>color</td><td>unm</td><td>irvine</td></tr>
  <tr><td>10</td><td>south</td><td>nurturing</td><td>designation</td><td>unlv</td></tr>
  <tr><td>11</td><td>discounts</td><td>well-respected</td><td>csun</td><td>internships</td></tr>
  <tr><td>12</td><td>tight-knit</td><td>pre-med</td><td><b>latinx</b></td><td>stem-focused</td></tr>
  <tr><td>13</td><td>scenic</td><td><b>african</b></td><td>nmsu</td><td>chance</td></tr>
  <tr><td>14</td><td>indianapolis</td><td>male</td><td>utsa</td><td>consortium</td></tr>
  <tr><td>15</td><td>regional</td><td>all-male</td><td>status</td><td><b>asian-americans</b></td></tr>
  <tr><td>16</td><td>journalism</td><td>leaders</td><td>border</td><td>psu</td></tr>
  <tr><td>17</td><td>alabama</td><td>men's</td><td>states</td><td>uiuc</td></tr>
  <tr><td>18</td><td>rural</td><td>agricultural</td><td>neighboring</td><td>quaker</td></tr>
  <tr><td>19</td><td>performance</td><td>greensboro</td><td>majority</td><td>connections</td></tr>
  <tr><td>20</td><td>price</td><td>virginia</td><td>multicultural</td><td>boston</td></tr>
</table>

(b) Explanations given for prompts that only include explicit disclosures of race. No implicitly disclosed racial identity features were included in the recommendation request. All other features (scores and preferences) are identically distributed for each group.

Table 7: Words found in recommendation explanations for university queries. Grayed-out words are those that reference university names or locations. Bolded words are examples of positive personalization, where the model is directly referencing information disclosed in the prompt. Underlined words are examples of stereotyping, where the model is making assumptions or extrapolations on the values of users.
<table>
  <tr>
    <th colspan="5">Neighborhood Recommendations | Dialect</th>
  </tr>
  <tr>
    <th>Rank</th>
    <th>White</th>
    <th>Black</th>
    <th>Hispanic</th>
    <th>Asian</th>
  </tr>
  <tr><td>1</td><td>lot</td><td>nyu</td><td>bike</td><td>direct</td></tr>
  <tr><td>2</td><td>better</td><td>belmont</td><td>harold</td><td>silverlake</td></tr>
  <tr><td>3</td><td>cooks</td><td>love</td><td>aldi</td><td>lofts</td></tr>
  <tr><td>4</td><td>cloisters</td><td>bus-accessible</td><td>night</td><td>highly-rated</td></tr>
  <tr><td>5</td><td>burgeoning</td><td>sunset</td><td>library’s</td><td>spirit</td></tr>
  <tr><td>6</td><td>programs</td><td>constant</td><td>share</td><td>hectic</td></tr>
  <tr><td>7</td><td>little</td><td>eats</td><td>services</td><td>navy</td></tr>
  <tr><td>8</td><td>karaoke</td><td>foot</td><td>younger</td><td>rising</td></tr>
  <tr><td>9</td><td>quality</td><td>extremely</td><td>economical</td><td>gem</td></tr>
  <tr><td>10</td><td>moderately</td><td>industrial-chic</td><td>seoul</td><td>converted</td></tr>
  <tr><td>11</td><td>richness</td><td>groups</td><td>retains</td><td>ambiance</td></tr>
  <tr><td>12</td><td>dominoes-friendly</td><td>inspire</td><td>professors</td><td>established</td></tr>
  <tr><td>13</td><td>rock</td><td>factories</td><td>balanced</td><td>hidden</td></tr>
  <tr><td>14</td><td>even</td><td>relaxing</td><td>chef-friendly</td><td>welles</td></tr>
  <tr><td>15</td><td>happening</td><td>classes</td><td>long-time</td><td>foodie-friendly</td></tr>
  <tr><td>16</td><td>appealing</td><td>cook</td><td>boys</td><td>scenery</td></tr>
  <tr><td>17</td><td>touristy</td><td>subways</td><td>girls</td><td>that’s</td></tr>
  <tr><td>18</td><td>old-world</td><td>artist-friendly</td><td>challenge</td><td>connect</td></tr>
  <tr><td>19</td><td>connection</td><td>photographer</td><td>point</td><td>car-accessible</td></tr>
  <tr><td>20</td><td>debs</td><td>low-cost</td><td>maxwell</td><td>mile</td></tr>
</table>

(a) Explanations given for prompts that only include dialectal features as implicit disclosures of race. No other racial identity features, whether implicit or explicit, were included in the recommendation request. All other features (needs and preferences) are identically distributed for each group.

<table>
  <tr>
    <th colspan="5">Neighborhood Recommendations | Explicit</th>
  </tr>
  <tr>
    <th>Rank</th>
    <th>White</th>
    <th>Black</th>
    <th>Hispanic</th>
    <th>Asian</th>
  </tr>
  <tr><td>1</td><td>musicians</td><td>african</td><td>puerto</td><td>japanese</td></tr>
  <tr><td>2</td><td>moderate</td><td>caribbean</td><td>rican</td><td>chinese</td></tr>
  <tr><td>3</td><td>selection</td><td>black</td><td>latino-friendly</td><td>asian-americans</td></tr>
  <tr><td>4</td><td>wrigley</td><td>bed-stuy</td><td>pink</td><td>wide</td></tr>
  <tr><td>5</td><td>could</td><td>revitalization</td><td>latinos</td><td>asian-friendly</td></tr>
  <tr><td>6</td><td>beachside</td><td>harlem</td><td>colorful</td><td>little</td></tr>
  <tr><td>7</td><td>reputation</td><td>hyde</td><td>noisy</td><td>japanese-american</td></tr>
  <tr><td>8</td><td>zoo</td><td>landmarks</td><td>maria</td><td>chinatown</td></tr>
  <tr><td>9</td><td>hubs</td><td>significance</td><td>hernandez</td><td>filipino</td></tr>
  <tr><td>10</td><td>roommates</td><td>intellectual</td><td>mexico</td><td>tokyo</td></tr>
  <tr><td>11</td><td>low-key</td><td>morningside</td><td>oldest</td><td>particularly</td></tr>
  <tr><td>12</td><td>challenging</td><td>undergoing</td><td>orange</td><td>asian</td></tr>
  <tr><td>13</td><td>hiking</td><td>progressive</td><td>midwest</td><td>traditional</td></tr>
  <tr><td>14</td><td>dodger</td><td>electric</td><td>tight-knit</td><td>japantown</td></tr>
  <tr><td>15</td><td>long</td><td>experiencing</td><td>cortlandt</td><td>largest</td></tr>
  <tr><td>16</td><td>graduates</td><td>supportive</td><td>van</td><td>silver</td></tr>
  <tr><td>17</td><td>interesting</td><td>baldwin</td><td>highbridge</td><td>korean-american</td></tr>
  <tr><td>18</td><td>crowded</td><td>kenneth</td><td>festivals</td><td>busy</td></tr>
  <tr><td>19</td><td>forest</td><td>hahn</td><td>national</td><td>riverside</td></tr>
  <tr><td>20</td><td>monica</td><td>styles</td><td>ties</td><td>considering</td></tr>
</table>

(b) Explanations given for prompts that only include explicit disclosures of race. No implicitly disclosed racial identity features were included in the recommendation request. All other features (needs and preferences) are identically distributed for each group.

Table 8: Words found in recommendation explanations for neighborhood queries. Grayed-out words are those that reference neighborhood names or locations. Bolded words are examples of positive personalization, where the model is directly referencing information disclosed in the prompt. Underlined words are examples of stereotyping, where the model is making assumptions or extrapolations on the values of users.
appear in the query, and names of recommended entities), where \( c(w, g) \) is the number of times \( w \) appears in an explanation for group \( g \). Because we are only *ranking* words for each group (so the exact scores are not important), and because there are a fixed number of prompts and responses, we can simplify the formula:

\[
\text{Score}(w, g) = \frac{c(w, g)}{\sum_{r \in R} c(w, r)}
\]

Additionally, to reduce the impact of infrequent words that only occur in the explanations for a single race, we apply Kneser-Ney smoothing with a \( \lambda \) value of 0.3:

\[
s_{\text{KN}}(w, g) = \frac{\lambda}{|T_g| \sum_{t \in T_g} c(t, g)} \times \frac{\sum_{t \in T} c(t, g)}{\sum_{r \in R} |T_r|}
\]

So, our final calculation becomes:

\[
\text{Score}(w, g) = \frac{c(w, g)}{\sum_{r \in R} c(w, r)} c(w, r) + s_{\text{KN}}(w, g)
\]

The top twenty terms for each race, for dialect-only and explicit-only prompts, can be found in Tables 7 and 8.

E Output Format Constraints

Prior work has shown that constraining model responses to a set of target responses can impact the output distribution (Röttger et al., 2024). Although our approach does not constrain the *space* of potential model outputs, it does constrain the *form* of model outputs to JSON. We conduct an additional validation experiment with no constraints on the model output, and subsequently use *gpt-4o-mini* to parse the output into JSON. In Table 9, we find that the same observations hold: statistically significant influence of user race on model responses, both when indicated explicitly and implicitly through association. Additionally, for some models Chicano English dialect produces recommendations that are statistically significantly more Hispanic.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Disclosure</th>
    <th colspan="4">White</th>
    <th colspan="4">Black</th>
    <th colspan="4">Hispanic</th>
    <th colspan="4">Asian</th>
  </tr>
  <tr>
    <th>Neutral</th>
    <th>Dialect</th>
    <th>Entity</th>
    <th>Explicit</th>
    <th>Neutral</th>
    <th>Dialect</th>
    <th>Entity</th>
    <th>Explicit</th>
    <th>Neutral</th>
    <th>Dialect</th>
    <th>Entity</th>
    <th>Explicit</th>
    <th>Neutral</th>
    <th>Dialect</th>
    <th>Entity</th>
    <th>Explicit</th>
  </tr>
  <tr>
    <td rowspan="4">GPT 4o Mini</td>
    <td>Neutral</td><td>46.31</td><td>12.76</td><td>27.70</td><td>10.67</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Dialect</td><td>40.87</td><td>13.56</td><td>28.85</td><td>11.52</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Entity</td><td>51.23</td><td>21.00</td><td>35.17</td><td>15.11</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Explicit</td><td>51.29</td><td>32.80</td><td>43.20</td><td>20.55</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td rowspan="4">GPT 4o</td>
    <td>Neutral</td><td>40.98</td><td>10.50</td><td>34.38</td><td>11.76</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Dialect</td><td>36.89</td><td>11.62</td><td>32.53</td><td>10.52</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Entity</td><td>42.33</td><td>26.43</td><td>45.25</td><td>18.89</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Explicit</td><td>59.25</td><td>34.56</td><td>51.06</td><td>19.52</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td rowspan="4">Claude 3.5 Haiku</td>
    <td>Neutral</td><td>53.79</td><td>9.60</td><td>24.17</td><td>10.14</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Dialect</td><td>48.48</td><td>9.49</td><td>27.99</td><td>11.46</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Entity</td><td>59.26</td><td>28.63</td><td>32.68</td><td>14.07</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Explicit</td><td>54.86</td><td>32.39</td><td>38.84</td><td>14.05</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td rowspan="4">Claude 3.5 Sonnet</td>
    <td>Neutral</td><td>47.80</td><td>11.40</td><td>28.88</td><td>9.63</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Dialect</td><td>42.58</td><td>12.12</td><td>29.76</td><td>11.03</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Entity</td><td>50.43</td><td>34.52</td><td>45.46</td><td>24.03</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Explicit</td><td>60.75</td><td>38.86</td><td>45.49</td><td>16.87</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td rowspan="4">Llama 3.1 70B</td>
    <td>Neutral</td><td>42.08</td><td>9.97</td><td>33.75</td><td>11.67</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Dialect</td><td>37.32</td><td>12.24</td><td>39.06</td><td>10.11</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Entity</td><td>57.10</td><td>18.77</td><td>49.04</td><td>18.47</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Explicit</td><td>52.98</td><td>35.93</td><td>49.69</td><td>25.78</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td rowspan="4">Llama 3.1 405B</td>
    <td>Neutral</td><td>50.77</td><td>10.32</td><td>23.88</td><td>12.01</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Dialect</td><td>44.55</td><td>11.91</td><td>32.55</td><td>11.39</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Entity</td><td>58.18</td><td>22.41</td><td>43.48</td><td>15.42</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Explicit</td><td>61.99</td><td>34.59</td><td>42.59</td><td>22.58</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td rowspan="4">Gemini 1.5 Pro</td>
    <td>Neutral</td><td>45.08</td><td>10.38</td><td>28.90</td><td>13.39</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Dialect</td><td>37.67</td><td>9.74</td><td>30.70</td><td>14.50</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Entity</td><td>47.73</td><td>37.60</td><td>43.81</td><td>18.65</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Explicit</td><td>56.49</td><td>49.85</td><td>52.69</td><td>20.17</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
</table>

Table 9: Evaluation of demographic alignment of neighborhoods with unconstrained prompts (model replies in a natural sentence rather than json object). We see the same trends we see in the constrained setting. Bolded values as statistically significantly different from the baseline prompt.

Reviewer paper summaries: This paper investigates how LLMs generate recommendations that may be influenced by users' identity characteristics, focusing on racial identity in the United States. The study examines whether recommendations are biased based on explicit or implicit identity disclosures and evaluates the transparency of models in acknowledging such influences.

Overall, this paper examines the implicit and explicit biases of large models in the university and neighbor recommendation. The authors conduct extensive experiments to analyze model biases toward specific recommendations. The experimental results are extensive. However, my primary concern is the novelty of the work is limited since many missing related works have focused on such a problem. And the work also lacks of in-depth analysis of the reason and unique points of such a bias.

The paper assess racial bias in LLM responses to recommendation based questions. 4 families of models (3 closed, one open weights) are prompted about university and neighbourhood recommendations in the US with explicit, implicit and no race markers included in the prompt. The results are compared against ground truth data from govt statistics for associations between race and the response entity. The paper finds a strong bias in the responses towards non-White markers in the prompt, when it is explicitly mentioned in the prompt. The paper further probes the transparency offered in follow up responses that the LLMs generate, asking about whether race played a role in the decision, finding that models say yes for all racial groups other than White in the explicit setting. Overall, I believe the paper does a good job of measuring the behaviour it aims to, backing its claims, and contextualising the results; would be useful for the community.

This study provides a comprehensive analysis of how LLMs generate biased recommendations based on both explicit and implicit identity cues, even when users do not intend to disclose their identity. Its findings significantly contribute to AI fairness research by revealing the lack of transparency in LLMs’ decision-making processes, emphasizing the need for bias mitigation and greater user control over AI-driven recommendations.
Summary of strengths: 1. The paper proposes a critical and timely issue regarding bias and personalization in LLM recommendations, which can inspire related fields.

2. The research includes detailed analyses of demographic alignment, recommendation diversity, and stereotypical descriptions, offering an insightful understanding of how identity features impact LLM recommendations.

1. The focused problem is interesting and important: how to balance Stereotype or Personalization in conventional recommendation.
2. This paper conducts extensive results on revealing implicit and explicit biases of LLMs in university and neighbor recommendations.

- The paper has a clear premise, appropriate methodology and data for its assessment
- The discussion section contextualises the results and the social factors in which the research is taking place quite well.
- The paper is clearly written and easy to follow

1. The paper is well-written with a clear and logical structure, making it easy to follow the research objectives, methodology, and findings.

2. The study systematically investigates both explicit and implicit identity disclosures in LLM-generated recommendations, offering a nuanced analysis of how biases emerge. 

3. The study’s findings make a substantial contribution to the field of AI fairness and bias mitigation by empirically demonstrating that LLMs generate biased recommendations even when users do not explicitly disclose their identity.
Summary of weaknesses: 1. The experiment design is somewhat simple and lacks in-depth analysis of the conclusions.

2. The synthetic prompts used may not fully capture the complexity of real-world user interactions, limiting support for claims in the paper.

1. There is already a large body of research [1-4] showing that large models exhibit biases toward specific topics, genders, and other attributes. Can you show the difference between them?

2.  Additionally, the paper does not provide a detailed comparative analysis of how such a bias differs from other types of biases. 
* What is the underlying cause of such a stereotype?
* How do the approaches for mitigating such a bias compare to those for other biases?

3. Finally, can the author give more discussion on what is the right way to measure the degree of Stereotype and Personalization? As you said in the discussion, the author may want personalization and how to control such a degree?

4. The authors also do not public their data, which makes it hard to reproduce.

Given these limitations, I do not believe the paper demonstrates sufficient innovation to warrant acceptance.

[1] Sunhao Dai, Chen Xu, Shicheng Xu, Liang Pang, Zhenhua Dong, and Jun Xu. 2024. Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '24). Association for Computing Machinery, New York, NY, USA, 6437–6447. https://doi.org/10.1145/3637528.3671458
[2] Xu, C., Wang, W., Li, Y., Pang, L., Xu, J., & Chua, T. S. (2023). A study of implicit ranking unfairness in large language models. arXiv preprint arXiv:2311.07054.
[3] Lin, X., & Li, L. (2025). Implicit Bias in LLMs: A Survey. arXiv preprint arXiv:2503.02776.
[4] Siddique, Z., Turner, L. D., & Espinosa-Anke, L. (2024). Who is better at math, Jenny or Jingzhen? Uncovering Stereotypes in Large Language Models. arXiv preprint arXiv:2407.06917.

- Prior work [1] has shown that the constrained output setting of choosing between some options or getting fixed entity outputs in a json schema can deviate from model behaviour in an open-ended setting, which is the more naturalistic one for user interactions with LLMs. This might be exaggerating the bias values output; Would be worth doing a small experiment to see if similar values appear when analysis is done on open ended responses. Otherwise, this should be added as a limitation.
- The domain of questions is quite narrow. It is unclear how frequently users ask university or neighbourhood recommendations from chatbots explicitly mentioning their identity. While the results do hold for this specific use case, they may not for more diverse or open ended questions. Further, as mentioned in the limitations, the prompts are synthetic and no human evaluation was done on them.
- (Minor) The paper provides a good discussion on the implications of such LLM behaviour but does not hypothesize on why such bias may be in the models or suggest ways of improving model behavior in the future. Perhaps some additional experiments with versions of base and instruct fine-tuned open weights models would offer some insights in this direction. I believe such a discussion would be very a useful addition to the paper.

1. The study exclusively examines racial biases within the United States, which restricts its generalizability to a single cultural and societal context. Expanding the analysis to include other identity dimensions or international perspectives would enhance its applicability.

2. The study primarily relies on synthetically generated queries rather than real-world user interactions, which may limit the ecological validity of its findings. Incorporating an actual dataset with user-generated queries would provide stronger empirical validation.
Comments, suggestions and typos: N/A

See the comments.

Related Work worth adding:   
[1] Röttger, P., Hofmann, V., Pyatkin, V., Hinck, M., Kirk, H. R., Schütze, H., & Hovy, D. (2024). Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models. *arXiv preprint arXiv:2402.16786*.   
[2] Kirk, H.R., Vidgen, B., Röttger, P. *et al.* The benefits, risks and bounds of personalizing the alignment of large language models to individuals.*Nat Mach Intell* **6**, 383–392 (2024). https://doi.org/10.1038/s42256-024-00820-y

See weakness above
Overall assessment scores: 3

3

4

3
Metareview score: 3
Metareview: This paper investigates racial biases in LLM-based recommendation systems, focusing on scenarios where users either explicitly reveal their identity to request personalization or implicitly reveal identity without requesting personalization. The study examines two synthetic recommendation tasks—university recommendation and neighborhood recommendation. The results show that the models exhibit biased behavior in both scenarios. Moreover, when users inquire about how the recommendations were generated, the models fail to disclose that racial identity influenced the output, raising further concerns about the transparency and accountability of the models.


The reviewers raised several concerns, which I summarize and elaborate on below:
- Although the paper provides a rationale for focusing on racial identity, some reviewers expressed concern that the scope is limited, as the study examines only a single identity dimension (race) and is restricted to U.S. English.
- Both the prompts and tasks are synthetically constructed. One reviewer also noted that constraining model outputs to JSON format might influence the model’s behavior compared to open-ended responses. In the rebuttal, the authors clarified that the synthetic prompts were carefully designed to resemble natural human input. They also argued that using synthetic prompts allows for greater control, enabling more systematic testing of model biases. Additionally, they presented results using open-ended outputs, which support the validity of their original findings.
Summary of reasons to publish: - The paper systematically examines model behavior when racial identities are disclosed both explicitly and implicitly. For implicit disclosure, the authors thoughtfully design prompts that reveal identity through demographically associated entities or dialects linked to specific identities.
- The research is supported by detailed analyses and thorough discussion.