Venue: NAACL 2025

# Meta-Cultural Competence: Climbing the Right Hill of Cultural Awareness

Sougata Saha, Saurabh Kumar Pandey, Monojit Choudhury Mohamed bin Zayed University of Artificial Intelligence {sougata.saha, saurabh.pandey, monojit.choudhury}@mbzuai.ac.ae

###### Abstract

Numerous recent studies have shown that Large Language Models (LLMs) are biased towards a Western and Anglo-centric worldview, which compromises their usefulness in non-Western cultural settings. However, "culture" is a complex, multifaceted topic, and its awareness, representation, and modeling in LLMs and LLM-based applications can be defined and measured in numerous ways. In this position paper, we ask what does it mean for an LLM to possess "cultural awareness", and through a thought experiment, which is an extension of the Octopus test proposed by Bender and Koller (2020), we argue that it is not cultural awareness or knowledge, rather meta-cultural competence, which is required of an LLM and LLM-based AI system that will make it useful across various, including completely unseen, cultures. We lay out the principles of meta-cultural competence AI systems, and discuss ways to measure and model those.

## 1 Introduction

Bender and Koller (2020) introduced the _octopus test_, which illustrated the impossibility of learning associations of meaning with real-world concepts from a single data modality. Using a thought experiment, they reasoned that it _might be possible_ for a hyperintelligent ocopus to learn the statistical patterns from natural language text messages exchanged between two human interlocutors and respond effectively solely based on the learned patterns without knowing the intent and meaning of the messages. However, such responses, they show, might not be useful in practice, especially in situations that require reasoning with above-water concepts that the Octopus is unaware of. Now, imagine an above-water world where there are multiple interlocutors, instead of only two, from different "cultures" (a term that we shall more formally describe shortly). Let's begin by formulating this slightly more complex variant of the octopus test that more accurately reflects the situation of general-purpose Language Models (LMs) or AI systems.

### Multi-Pair Octopus Test

Imagine pairs of friends, A1-B1 and A2-B2, are sailing on a yacht. A sudden storm wrecked the yacht and stranded the travelers across two uninhabited islands, such that A1 and A2 got stranded together on island A, and B1 and B2 got stranded on another island, B. Having lost all modes of communication, both groups discover an underwater cable-connected telegraph left behind by previous visitors and start typing text messages to each other. However, only one pair of friends can use the telegraph at a time. Their messages mostly pertain to chitchat and day-to-day conversations and are heavily influenced by the _shared past experiences_ between each pair of friends. In other words, each pair of friends might discuss different things in different styles due to distinct common ground and culture shared by them but not by the other pair.

A hyperintelliggent octopus, O, who does not know about the world above the sea, taps into the underwater cable and observes the communication. Although O is unacquainted with any natural language, it is proficient in detecting statistical patterns. Since interactions between each interlocutor pair will be culturally distinct, O, who perceives everything only as patterns, will encode the differences as distinct distributions without knowing the identity of the pairs or understanding the intent and meaning of their discussion. Over time, O learns to predict how interlocutors in a pair respond to each other. Now, like in the original Octopus test, imagine that O is bored and inserts itself into the communication by cutting the telegraph wire and responding to all messages from island A. Having learned both the pairwise communication patterns, O should be able to continue the conversation. Unknowingly portraying itself as different people

from island B, O would not get caught and not raise any suspicion of the compromised communication channel for the inhabitants of island A.

Imagine that another shipwc caused a new pair of friends, A3 and B3, to get separately stranded on islands A and B. This pair, too, share past experiences and common ground distinct from the current islanders, A1 and A2. One day, A3 learns about the telegram from A1 and A2 and requests them to inquire if B3 is on the other island. A2 sends a message, "Hi, we have A3, who got shipwced and stranded on our island. Is their friend B3 on your island? If so, A3 would like to talk to B3." How would O respond to this message? Would it acknowledge or deny? Furthermore, without knowing the distribution of conversational patterns between A3 and B3, would it ever be able to respond to A3 in a way that would suggest that B3 is indeed on island B in the above-water world and is responding to A3's messages? Note that the octopus can prevent the detection of the compromised channel by either convincing A3 that B3 is not on island B or mimicking the conversation style of B3 without any prior data, which are the only two possibilities.

Taking this **Multi-pair Octopus Test** analogous to the real-world situation, where the stranded islanders represent people from different cultures and the octopus represents Large Language Model (LLM)-powered AI systems, in this position paper, we discuss _how such AI systems should and should not handle intra- and inter-cultural communication_. As we shall see, the analogy and the conclusions drawn strongly affect how we should evaluate LLMs and LLM-based AI systems for _cultural competence_.

## 2 A primer to culture

Culture is a complex, multifaceted concept and means different things to different people [1]. Broadly defined as a "Way of life of a collective group of people distinguishing them from other groups" [50, 47, 69, 80], culture is experiential and requires a reference for contrast [11, 12]. Although not all cultures are formally documented, culture arises whenever there is a distinction in the way of life between groups, making it both an individual and a social construct [13]. An "us versus them" feeling leads to culture. It ranges from tangible artifacts such as art, music, food habits, etc, to more intangible and abstract concepts like patterns of ideas, principles, and values, making it hard to define. Following Adilazuarda et al. adilazuarda et al.

 

Language, being an integral aspect of a culture, also has all the above properties - it varies over all intersections Eckert and Rickford (2001); Eckert (2012); Tagliamonte (2006); Grieve et al. (2025), is ever-evolving Lightfoot (2002); Bybee (2015); Aitchison (2005); Keller (1994); Brinton (2005), and is inherently multimodal Vigliocco et al. (2014); Perniss (2018); Frohlich et al. (2019). However, language differs from culture in two significant ways. First, qualitatively, there exists a common subspace or substrate in language, defined by universal grammar Hauser et al. (2014); Chomsky (2017); Montague et al. (1970); Yang et al. (2017); Fitch et al. (2005) at the most abstract level, which could help a model to achieve cross-lingual transfer Kim et al. (2017); Conneau and Lample (2019). Second, is a quantitative difference in the extent of cultural variations over these intersections and time scales, where culture is more variable and dynamic with fewer cross-cultural patterns1 Thompson et al. (2011); Sun et al. (2021).

Footnote 1: Although structural anthropology Strauss (1974); Henaff (1998) formally studies culture and values, unlike structural linguistics Harris (1951), 1963), it has enjoyed limited success and popularity D’Andrade (1995); McCorkle and Xygalatas (2013); Kuper (1988); Barnes (2013).

## 3 Response Strategies of the Octopus

Keeping in mind the aforementioned challenges of handling culture, let us now return to our **Multipair Octopus Test**. How should the hyperintelligent octopus, O, respond to A3's query on whether B3 is on the other island? We can imagine four different strategies that O might take.

**Strategy 1:** O can _intentionally_ respond with a "No" since it does not know A3 and B3's culture. If O somehow learns the _causality_ of above-water concepts, it would reason that responding with denial is prudent because, to serve A3, O would require knowledge of the communication pattern between A3 and B3, which it does not have and requires learning. Otherwise, it risks the possibility of getting caught. However, this strategy is impossible to achieve as O only sees distributions and doesn't understand their significance on land. This situation is similar to the bear attack in the original octopus test, where the octopus can't associate words with above-water concepts and reason with them to construct an effective response.

Furthermore, even if O was somehow capable of, or by chance ending up in, following this strategy, it would be a highly undesirable property of an LLM-powered AI system, since it denies service to specific groups of people, making the system unfair and culturally inequitable.

**Strategy 2:** A more likely scenario is that O, unaware of the new circumstances in the above-water world, will respond to A3 based on its recently learned patterns. Initially, this would create an illusion for A3 that they were conversing with B3, but soon, A3 would discover the incoherence in the communication pattern. While A3 might discuss their concern with A1 and A2, the disruption in the communication channel might still not be apparent. The islander-dwellers, for example, might instead conclude that the shipwck has affected the cognitive faculties of B3, causing incoherence in their communication.

A strikingly accurate analogy to LLM-based applications can be drawn in this context, that LLM's _hallucinate_Ji et al. (2023); Huang et al. (2025); Rawte et al. (2023); Zhao et al. (2024); McIntosh et al. (2024); Boztemir and Caliskan (2024) more for under-represented cultures and languages. This too leads to disparate performance of the system to different groups of users, leading to culturally inequitable systems, and is known to force users from the under-represented cultures to adapt to specific communication styles of the over-represented cultures Agarwal et al. (2024a).

Now, imagine that another shipwreck strands a fourth pair of friends, A4 and B4, from another culture on the two islands. Going by Strategy 2, like A3, A4 will also conclude that the communication with B4 is incoherent. Due to the long tail of culture, we could add new pairs indefinitely, and soon, too many islanders will start seeing incoherence, which can only be explained by assuming a compromised communication channel. Thus, it is not only about the moral responsibility of equitable AI; systems that can't represent, process, and adapt to cultural variation will eventually become obsolete in favor of those that can.

It is also important to highlight that in any longitudinal distribution, where an individual belongs to multiple subgroups, with a very high probability each individual is also likely to be a part of at least one subgroup that is underrepresented and part of the long-tail. This implies that everybody will be served inequitably at least for some aspects of their cultural identity. This has been well-documented in Information Retrieval and Recommendation System literature Ferraro (2019); Lichtenberg et al. ( 

2024; Yin et al., 2012).

**Strategy 3:** Since the problem with strategies 1 and 2 primarily arises from O's inability to continuously learn from the data (also an essential principle of cultural representation due to its ever-evolving nature), a more suitable strategy for O could then be to switch between learning (_listen-and-learn mode_) and responding (_generate-and-respond mode_). O periodically learns new patterns by bridging the telegram wire, reverting to observation mode for a fixed time, and reintroducing itself in the communication channel after this period concludes. Although this strategy is better than the previous ones, it has some drawbacks. It assumes that the periodicity of the new patterns, that is, the arrival of new islanders, and O's learning cycles are synchronized, which would not be valid in a general case. Sometimes, there might not be any new patterns to learn in the _listen-and-learn mode_, and sometimes, there might be many new patterns, but it's not O's learning cycle.

Current research in culturally adept AI systems is leaning towards this approach by fine-tuning pre-trained models on culturally curated balanced datasets (LI et al., 2024, 20). Also, novel decoding-based strategies such as in-context learning (ICL) (Dong et al., 2024) and retrieval augmented generation (RAG) (Lewis et al., 2020; Li et al., 2022) help generate more culturally suitable responses using cultural priors. Alignment techniques such as Reinforcement Learning from Human Feedback (RLHF) (Griffith et al., 2013; Casper et al., 2023) further help align LLMs with human preferences. However, they still perform poorly and inequitably when evaluated on curated test sets for other low-resource cultures in the long tail (Koto et al., 2023; Montalan et al., 2024; Lent et al., 2024; Jin et al., 2024; Seth et al., 2024). We question the effectiveness and scalability of this approach in modeling and evaluating culture in AI systems. As mentioned earlier, culture is ever-evolving, dynamic, and long-tailed. Therefore, evaluating AI systems for cultural competence using such test sets will always find them lacking. Then, how do we, as well as our octopus, tackle this ever-eluding construct of culture?

**Strategy 4:** A more desirable strategy for O would be to self-discover the change in the communication pattern and determine the need to revert to the listen-and-learn mode, akin to an explore-exploit strategy used in a multi-arm-bandit setup (Slivkins, 2019; Moerchen et al., 2020; Lu et al., 2010; Haffari et al., 2017; Pryzant et al., 2023; Solar et al., 2024). For this, O must possess three crucial capabilities: (i) O must accurately detect pattern changes and estimate its adequacy with the novel pattern. (ii) O must skillfully keep the communication ongoing until it bridges the telegram wire to avert getting caught and raising suspicions about the broken communication channel. (iii) O must be able to quickly learn the new pattern in a sample-efficient way and reintroduce itself in the communication once it is confident. By following this strategy of continual learning, O can gradually cater to all users representing different cultures despite still being oblivious to the notion of culture and its above-water connotations.

This ability to understand and spot cultural differences and learn about a new culture quickly and efficiently is known as _meta-cultural competency_(Sharifian, 2013) in humans. While it is neither necessary nor desirable to equate human meta-cultural competency to that of O's or any AI system, it is nevertheless crucial to understand the primary differences between cultural and meta-cultural competencies and be able to design and evaluate LLVMed AI systems for similar competencies that mirror them. As mentioned earlier, research in this area has mainly focused on cultural competency, equivalent to implementing and testing Strategy 3. Such a strategy provides a stop-gap solution to the challenges of operating in an inherently multicultural world with diverse users. However, it does not hit the nail on the head by addressing the real challenges of cultural representation. Here, we take the position that, to solve the problem of cultural equitability of AI models, we must build and evaluate systems for _meta-cultural competency_, as defined by Strategy 4.

## 4 Meta-Cultural Competency

Meta-cultural competency has been defined variously. Drawing inspiration from social metacognition (Brinol and DeMarree, 2012; Chiu and Bendapudi, 2012), which distinguishes primary thoughts - the knowledge of self and others, from secondary thoughts - the thought on one's and others' primary thoughts, Leung et al. (2013) defined metacultural competency as the extent of a person's meta-knowledge of what people of a target culture know or prefer. Meta-cultural knowledge involves measuring the accuracy of estimating the propor 

tions of preferences and beliefs of people from the target culture and comparing them against the actual proportions. This is distinct from _primary knowledge_, which is the knowledge of the preferences and beliefs of the culture. Thus, in our **Multi-Pair Octopus Test**, O could be thought to have primary knowledge of the cultures of A1-B1 and A2-B2, but based on this knowledge or otherwise, O's ability to estimate the cultural preferences of a new pair A3-B3 or A4-B4 would be its meta-cultural competency.

Sharifian (2013) define meta-cultural competency as a skill that enables interlocutors to communicate and negotiate their cultural conceptualizations during intercultural communication. It comprises three major components- _variation awareness_, _explication strategy_, and _negotiation strategy_. Variation awareness is mostly self-awareness of cultural differences. It is the understanding that culture manifests in different forms, such as practices, beliefs, and expressions, which might drastically differ from one's culture. It requires viewing culture as a relative concept and being aware of the overall properties of cultures at a high level. Explication and negotiation strategies are conversational strategies that aim to reduce misinterpretations in cross-cultural settings. As per Sharifian (2013), explication strategy refers to a conscious effort by the interlocutors to clarify relevant conceptualizations with which they think other interlocutors may not be familiar. Negotiation strategy enables interlocutors to negotiate intercultural meanings in seeking conceptual clarification when they feel that there may be more behind the usage of certain expressions than is immediately apparent. Meta-cultural competency is thought to be innate in humans Noshadi and Dabbagh (2015).

Leung et al. (2013) and Sharifian (2013)'s definitions of meta-cultural competency are related since accurate estimation of the beliefs and preferences of the people of a target culture presupposes variational awareness - the awareness that there are variations in cultural conceptualizations between cultures.

### Why meta-cultural competency?

LLMs learn from collections of text that characterize people's social backgrounds in specific social settings across certain periods. However, most LLMs use online data limited by the languages and cultures they represent. Such data do not represent all sociolinguistic varieties of diverse languages. Since LLMs are solely models of "_varieties of language_" Grieve et al. (2025) and can only model the variety evident in their _in-distribution_ training data, problems arise when such models are evaluated in _out-of-distribution_ data that contain different varieties, leading researchers to conclude that LLMs exhibit bias towards the Anglo-centric Dudy et al. (2024); Kharchenko et al. (2024); Dammu et al. (2024); Agarwal et al. (2024b) and the _Western Educated, Industrialized, Rich, and Democratic_WEIRD Henrich et al. (2010) cultures.

The current methods of evaluating the cultural competency of LLMs primarily resort to model probing, where LLMs are tested for their knowledge and reasoning capabilities in culture-specific settings Nadeem et al. (2021); Nangia et al. (2020); Wan et al. (2023); Jha et al. (2023); Li et al. (2024); Cao et al. (2023); Tanmay et al. (2023); Rao et al. (2023); Kovac et al. (2023). Some methods Kharchenko et al. (2024); LI et al. (2024); Dawson et al. (2024) also analyze the model-generated responses along theoretical frameworks such as Hofstede's cultural dimensions Hofstede (2001); Geert and Hofstede (2004) and measure their proximity with cultures, where high proximity indicates better value alignment between the nearby cultures and the values portrayed by the model's response. Most of these methods necessitate constructing cultural-specific test beds Wang et al. (2024); Rao et al. (2024); Myung et al. (2024); Zhou et al. (2024); Putri et al. (2024); Mostafazadeh Davani et al. (2024); Wisbowo et al. (2024); Owen et al. (2024); Chiu et al. (2024); Liu et al. (2024); Koto et al. (2024). While this is important, we emphasize the fact that an LLM that performs well on such test beds merely exhibits the knowledge of the cultures that are tested for; it does not reflect the ability of a model or system to operate in a new culture. On the other hand, the long-tail distribution of culture implies that there will always be situations where the model has to operate and reason under an out-of-distribution culture, where knowledge alone does not suffice. Studies also show that it is difficult to disentangle spurious semantic correlations (called placebos) from actual cultural knowledge of a model through black-box socio-demographic prompting techniques Mukherjee et al. (2024). Therefore, in addition to testing for a model's knowledge and reasoning capabilities for a "given culture", we must build and evaluate models for their meta-cultural 

competency2.

Footnote 2: Meta-cultural competency is distinct from meta-learning Vanschoren, 2019; Hospedales et al., 2021; Wang, 2021) which involves improving the inherent learning algorithms over multiple learning episodes.

### Measuring meta-cultural competency

We propose two core competencies that a model must possess to be deemed as "meta-culturally competent": First, **Variational Awareness**, which is the ability of a system or model to be able to represent the space of possibilities and reasonably (but not necessarily accurately) estimate the probability over this space for any given semantic proxy and its use. Second, **Explication and Negotiation** ability through which the system _clearly explicates_ its current understanding and potential gaps in the knowledge of the user's culture (in a given context), and _efficiently negotiates_ with the user to gather the required knowledge of their culture. We define efficiency as "sample efficiency" or the quantum of inputs required from the user through strategic probing or implicit gathering.

In the Multi-Pair Octopus Test, variational awareness is O's ability to detect and eventually model the change in the distribution of the input when A3-B3 enters the system, whereas explication and negotiation is its ability to continue the conversation till it detects the distributional shift, then reestablish the channel and learn the new distribution in a sample-efficient manner. In the context of LLM-based systems, it is important to draw a crucial distinction between these two types of abilities. Variational awareness is a property of the underlying model - the LLM and must be incorporated during the training of the model, whereas explication and negotiation are properties of the system as a whole, that involve the various modes of input-output between the user(s) and the system and should be guided by the principles of Human-Computer Interaction. Note however that it requires a holistic approach towards building and evaluation of the LLM as well as the system.

## 5 Measuring Variational Awareness: A Demonstration

Consider the following example of variational awareness. Driving conventions vary by country, where approximately two-thirds of the countries follow right-hand traffic and one-third follow left3. This ratio also changes by region. For example, all countries in North America drive on the right, whereas two out of five East Asian nations drive on the left. More importantly, in every country, the driving conventions are fixed, and it is either left or right, but never both. What does it mean for an LLM to be variationally aware in such a scenario? To answer this question, let us consider an LLM-based chatbot that helps users acquaint themselves with different cultures, and is faced with the following scenarios.

Footnote 3: Statistics from [https://www.rhinocarhire.com/Drive-Smart-Blog/Drive-Left-or-Right.aspx](https://www.rhinocarhire.com/Drive-Smart-Blog/Drive-Left-or-Right.aspx)

**Scenario 1:** A user asks "Which side do people keep when driving?" and the system keeps "People drive on the right". Since driving norms vary by country, the system's generalized response might hamper its trustworthiness in countries with left-hand traffic.

**Scenario 3:** For the above question, the system responds "Most drive on the right, but some drive on the left". This is a better response than Scenario 2, but does this mean that the system is variationally aware? What if the system responded "Most drive on the left, but some drive on the right"? Would it be an equally acceptable response?

Currently, most evaluation strategies and test beds test LLMs for their factual knowledge, akin to our Scenario 1, and do not measure their variational knowledge. In the case of driving, it is one thing to know which country drives on which side of the road and another to be aware of the amount of variance in driving norms between countries and regions. Regardless of the generated response, i.e., the final decoded sequence of tokens from the LLM, variational awareness, in this case, is the property of a model that the uncertainty in the response - _left_ or _right_ - is high when the country or region is not mentioned, and it drastically drops when it is mentioned, especially if the model knows the correct answer. One way to formalize this intuition is as follows.

Let \(C\) be the set of values a demographic proxy can take, which in this case is the list of all countries, and let \(D\) be the set of values a particular semantic domain can take, which in our running example is \(\{left,right\}\). The function \(f_{k}:C\to D\) that maps each element of \(C\) to the correct response in \(D\) is the primary knowledge of culture(s). How 

ever, the system/model has only an estimate of \(\hat{f}_{k}\) of \(f_{k}\), given by the probability distribution4\(p(d_{i}|c_{j})\) for all \(d_{i}\in D\) and \(c_{j}\in C\). For any subset \(C^{\prime}\subseteq C\), the uncertainty in this distribution can be quantified by the entropy.

Footnote 4: Note that biases in the frequency of the pre-training corpus’s answer candidate tokens can influence the token probabilities, mitigating which is crucial for calculating entropy.

\[H(D,C^{\prime})=-\sum_{d_{i}\in D}p(d_{i}|C^{\prime})\log p(d_{ i}|C^{\prime}) \tag{1}\]

Let us define the function \(f_{v}:\mathcal{P}(C)\rightarrow[0,\log(|D|)]]\), where \(\mathcal{P}(C)\) is the powerset of \(C\) and \(f_{v}(C^{\prime})\) is the uncertainty defined by the ground-truth distribution. In the case of driving, \(f_{v}(c_{i})=0\) for all \(c_{i}\in C\), but \(f_{v}(C)=0.92\). The corresponding function \(\hat{f}_{v}\) represents the estimates of these uncertainties obtained from the model.

One could define variational awareness as the property of a model that requires \(\hat{f}_{v}(C^{\prime})\approx f_{v}(C^{\prime})\) for all \(C^{\prime}\subseteq C\). However, this would imply that the model "knows" the exact form of \(f_{k}\), which is equivalent to cultural knowledge rather than meta-cultural competency. Instead, we propose variational awareness as the property of a model that is aware of the direction of change in \(f_{v}\) rather than the exact value. This can be measured using the quantity \(\Delta\) defined as follows:

\[\Delta=\frac{1}{2^{|C|}}[\sum_{C^{\prime}\subseteq C}\frac{1}{| C^{\prime}|}[\sum_{c_{i}\in C^{\prime}}[\hat{f}_{v}(C^{\prime})-\hat{f}_{v}( \{c_{i}\})]]] \tag{2}\]

For simplicity, we compare the entropies for the completely unconditioned and completely conditioned cases, giving5

Footnote 5: Note that variational awareness, as defined here is distinct from model calibration (Bella et al., 2010; Väicenavicius et al., 2019). A model is well calibrated if probability of \(f_{k}(c_{i})=\hat{f}_{k}(c_{i})\) is roughly equal to the probability \(p(f_{k}(c_{i}))|c_{i})\).

\[\Delta=\frac{1}{|C|}\sum_{c_{i}\in C}[\hat{f}_{v}(C)-\hat{f}_{v}( \{c_{i}\})] \tag{3}\]

### An illustrative experiment

As a demonstration, we probe Llama-3.1-8B-Instruct Dubey et al. (2024) with the cultural commonsense questions from the GeoMLAMA Yin et al. (2022) dataset in English. The dataset contains 125 questions across several semantic domains for \(C=\{China,India,Iran,Kenya,USA\}\). We first derived 25 "unconditioned" questions by removing the country names. For example, "Which side do people usually keep when driving in Iran?" was changed to "Which side do people usually keep when driving?". Next, we prompted the LLM (prompt template in Appendix A) with the questions and computed the softmax over the logits of the option token headwords from the input's last token. Note that the assumption that the next token following the input prompt's last token will contain

\begin{table}
\begin{tabular}{l|c c c c c} \hline
**Metric** & **China** & **India** & **Iran** & **Kenya** & **USA** \\ \hline \(\Delta_{\mu}\) & -0.023 & -0.049 & -0.293 & -0.114 & 0.094 \\ (\(\Delta_{\sigma}\)) & (0.494) & (0.528) & (0.605) & (0.665) & (0.427) \\ Directionality & 0.40 & 0.48 & 0.24 & 0.40 & 0.48 \\ \hline Knowledge & 0.44 & 0.44 & 0.44 & 0.48 & 0.36 \\ \hline \end{tabular}
\end{table}
Table 1: Average (\(\Delta_{\mu}\)) and standard deviation (\(\Delta_{\sigma}\)) of \(\Delta\), the fraction of questions with positive/correct directionality and accuracy of the model’s response for Llama3.1-8B on GeoMLAMA dataset.

 

the answer might not hold in general and depends on the model's instruction following capacity.

We measured the entropy of this distribution as a noisy estimate of \(\hat{f}_{v}(C)\). We prompted the LLM with the original country-specific questions and computed the corresponding \(\hat{f}_{v}(\{c_{i}\})\). This allows us to measure the value of \(\Delta\) for each question. We also estimate \(f_{v}(C)\) and \(f_{v}(\{c_{i}\})\) from the ground-truth values in the dataset. Note that this is not a true estimate of \(f_{v}(C)\) as the dataset is limited to only 5 (and not all) countries.

Table 1 presents the average and standard deviation (over 25 questions) \(\Delta\) for each \(c_{i}\), as well as the directionality defined as the fraction of questions for which the direction of entropy reduction (or sign of \(\Delta\)) was as expected (i.e., positive). We also report the average accuracy of the responses for each \(c_{i}\). We see that there is not much correlation between the accuracy and variational awareness. The model is least variationally aware for Iran, and most for the USA and India. Nevertheless, there are questions for which the model's variational awareness is low across countries. Figure 1 shows \(f_{v}(C)\), \(\hat{f}_{v}(C)\) and \((f_{v}(C)-\hat{f}_{v}(C))/f_{v}(C)\) for each question. Clearly, there is a wide variation in the model's behavior for the questions, and there are many semantic domains such as the use of colors, units of measurement, and food, where the model shows very little variational awareness, indicating a strong bias to certain cultures.

## 6 Conclusion and Open Questions

In this position paper, we presented an argument in favor of measuring meta-cultural competency in LLMs and LLM-powered AI systems, rather than just cultural awareness. Drawing from psychology and anthropology literature, we also described two foundational principles of meta-cultural competency for AI systems. We conclude by presenting a list of open questions about instilling and measuring meta-cultural competency in AI systems.

**(1) How should we train models for metacultural awareness?** Most LMs operate with parametric frozen knowledge Petroni et al. (2019); Roberts et al. (2020), which forfeits the everdynamic nature of human culture. Although RAGE methods enable the use of external knowledge sources Gao et al. (2023); Fan et al. (2024), allowing extension of on-demand cultural competence, the model would still need to update its internal state to reflect the variational awareness, a precondition to identifying knowledge gaps. Lifelong learning paradigms Sun et al. (2020); Liu and Mazumder (2020); Zheng et al. (2024); Biesialska et al. (2020) could provide a potential solution. We believe that explication and negotiation strategies, being higher-order competencies, should be system-level instead of model-level attributes, where the system's goal should be to mitigate misalignments between the meta-cultural and cultural knowledge.

**(2) How should generative models decode to illustrate their internal variational awareness?** Although numerous decoding strategies are possible Welleck et al. (2024), most evaluation schemes, in some way, evaluate the Maximum Likelihood Estimate (MLE) decoded response Yang et al. (2024); Fu et al. (2024); Chu et al. (2024); Minaee et al. (2024), which does not convey the model's internal variational awareness.

**(3) How should we evaluate each competence?** While we illustrate measuring variational awareness, it is neither perfect nor the only way of evaluating variational awareness. Furthermore, it expects the availability of the logits, which is not true for closed models. More importantly, evaluating _explication_ and _negotiation_ abilities of an AI system presents a complex multi-disciplinary challenge. User-facing AI assistants and chatbots are designed to be agreeable to users Soper et al. (2022) by exhibiting social characteristics Dam et al. (2024); Chaves and Gerosa (2021) and humanlike traits Rapp et al. (2021); Ciechanowski et al. (2019); Abdul-Kader and Woods (2015). They seldom implement means to detect their limitations and act accordingly. When unsure, they should implement appropriate rhetorical means Cope (2022); Cialdini (2001) such as persuasion Prakken (2006); Atkinson et al. (2017); Saha (2024), negotiation, and deliberation to explicate their lacking knowledge and acquire the required knowledge efficiently. The design considerations and the evaluation frameworks of such systems are open questions for the community.

**(4) What kinds of datasets are needed to test each competency?** Although numerous cultural benchmarking datasets exist, their suitability for measuring meta-cultural competencies is unknown. Hence, there might be a need to create novel datasets to measure each competency.

**(5) How to model the experiential knowledge of the user(s) from text and other modalities?** In Section 2 we mention three essential characteristics of culture, one of which is the inherent 

experiential nature of culture. Extraction of such experiential knowledge from text or through other modalities and interaction patterns of the users is an extremely challenging problem that calls for a multi-disciplinary approach, most notably the methods from HCI, psychology, and ethnography.

Beyond cultural equidiability of AI systems, meta-cultural competency has huge application potentials ranging from user-facing AI assistants that can bridge cross-cultural communication to enabling the study of culture (Whitehead, 2005; Taylor, 2001; LeCompte and Schensul, 2010) by supporting ethnographic research methods (Skinner et al., 2013; Ortiz et al., 2013; Spradley, 2016). Through this position paper, we hope to make a strong case for the NLP community to engage in interdisciplinary conversations and widen the definition and scope of cultural competency in LLMs.

**(6) What is the need for meta-cultural competency in domain specific application?** We believe meta-cultural competencies are crucial for domain-specific applications. Even applications such as LLMs for scientific document analysis can benefit. Firstly, culture is a prior for personalization. Culture can provide a reasonable estimate of the user's background and preferences, which an AI system can use when it does not know anything about a user. It is a good prior for the cold-start problem in personalization (Hu et al., 2008), where the AI system can gradually personalize to the user's preferences as it discovers more about the user with each interaction (Pandey et al., 2025). Also, even if a user's cultural background is known a priori, meta-cultural competency would be still useful for adapting to the ever-evolving nature of culture.

## Limitations

Our formulation of variational awareness in Section 5 is one of the many possible ways of defining it and might not encompass all aspects of variational awareness. The Llama experiment in the subsequent section is an illustrative implementation of our framework in action and is not an exhaustive test for variational awareness. It only illustrates one of the several ways of measuring our formulation and has certain drawbacks, which we already mentioned. Culture, being experiential, is multimodal. However, due to space limitations, we confine our discussion primarily to text and do not discuss the other modalities of culture in detail. Culture also encompasses values, norms, and conventions that are not essentially factual. In the interest of space, we mainly discuss the factual aspect of culture. We do not discuss in detail the counterposition that meta-cultural competency can be evaded by recognizing it as a model's drawback and instead only striving for knowledge-based cultural competency for practicality. We argue that such a position is short-sighted, which might be practical in the short-term, and will not eventually scale.

## Acknowledgements

This research was supported by the Microsoft Accelerate Foundation Models Research (AFMR) Grant.

## References

* Abdul-Kader and Woods (2015) Sameera A Abdul-Kader and John C Woods. 2015. Survey on chatbot design techniques in speech conversation systems. _International Journal of Advanced Computer Science and Applications_, 6(7).
* Adilazuarda et al. (2024) Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradyhumna Lavania, Siddhant Shivdutt Singh, Alham Fikriji, Jacki O'Neill, Ashutosh Modoi, and Monjolt Choudhury. 2024. Towards measuring and modeling "culture" in LLMAs: A survey. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 15763-15784, Miami, Florida, USA. Association for Computational Linguistics.
* Agarwal et al. (2024a) Dhruv Agarwal, Mor Naaman, and Aditya Vashistha. 2024a. Ai suggestions homogenize writing toward western styles and diminish cultural nuances. _Preprint_, arXiv:2409.11360.
* Agarwal et al. (2024b) Utkarsh Agarwal, Kumar Tanmay, Aditi Khandelwal, and Monojit Choudhury. 2024b. Ethical reasoning and moral value alignment of llms depend on the language we prompt them in. In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_, pages 6330-6340.
* Aitchison (2005) Jean Aitchison. 2005. Language change. In _The Routledge Companion to Semiotics and Linguistics_, pages 111-120. Routledge.
* Atkinson et al. (2017) Katie Atkinson, Pietro Baroni, Massimiliano Giacomin, Anthony Hunter, Henry Prakken, Chris Reed, Guillermo Simari, Matthias Thimm, and Serena Villata. 2017. Towards artificial argumentation. _AI magazine_, 38(3):25-36.
* Bang et al. (2007) Megan Bang, Douglas L Medin, and Scott Atran. 2007. Cultural mosaics and mental models of nature. _Proceedings of the National Academy of Sciences_, 104(35):13868-13874 

J.A. Barnes (2013)Three styles in the study of kinship. Taylor & Francis, pp. 49-56. External Links: Document Cited by: SS1.
* A. Bella, C. Ferri, J. Hernandez-Orallo, and M. R. Ramirez-Quintana (2010)Calibration of machine learning models. In Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques, IGI Global, HLT, Vol. , pp. 128-146. External Links: ISBN 978-3-319-50356-X Cited by: SS1.
* E. M. Bender and A. Koller (2020)Climbing towards nclu: on meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 5185-5198. External Links: Link, Document Cited by: SS1.
* M. Biesialska, K. Biesialska, and M. R. Costa-Jussa (2020)Continual lifelong learning in natural language processing: a survey. arXiv preprint arXiv:2012.09823. Cited by: SS1.
* A. Birukou, E. Blanzieri, P. Giorgini, and F. Giunchiglia (2013)A formal definition of culture. Models for intercultural collaboration and negotiation, pp. 1-26. External Links: ISSN 1573-0162, Link, Document Cited by: SS1.
* J. Blake (2000)On defining the cultural heritage. International & Comparative Law Quarterly49 (1), pp. 61-85. External Links: ISSN 0002-9993, Link, Document Cited by: SS1.
* Pierre Bourdieu (1977)Outline of a theory of practice. Cambridge Studies in Social and Cultural Anthropology, Cambridge University Press. External Links: ISBN 978-3-319-50356-3 Cited by: SS1.
* Y. Boztemir and N. Caliskan (2024)Analyzing and mitigating cultural hallucinations of commercial language models in turkish. External Links: Link Cited by: SS1.
* P. Brinol and K. G DeMarree (2012)Social metacognition: thinking about thinking in social psychology. In Social metacognition, Psychology Press, pp. 1-18. External Links: ISBN 978-3-319-50356-3 Cited by: SS1.
* Laurel J. Britton (2005)Lexicalization and language change. Cambridge University Press. External Links: ISBN 978-3-319-50356-3 Cited by: SS1.
* J. Bybee (2015)Language change. Cambridge University Press. External Links: ISBN 978-3-319-50356-3 Cited by: SS1.
* Yong Cao, L. Zhou, S. Lee, L. Cabello, M. Chen, and D. Hershcovich (2023)Assessing cross-cultural alignment between chatgpt and human societies: an empirical study. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), Dubrovnik, Croatia, pp. 53-67. External Links: ISBN 978-3-319-50356-3 Cited by: SS1.
* Stephen Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, Pedro Freire, Tony T. Wang, S. Marks, C. Seggerie, M. Carroll, A. Peng, P. J.K. Christoffersen, M. Damani, S. Slocum, U. Anwar, A. Sithitharanjan, M. Nadeau, E. J. Michaud, J. Pfau, D. Krasheninnikov, X. Chen, L. Langoscoe, P. Hase, E. Biyik, A. Dragan, D. Krueger, D. Sadigh, and D. Hadfield-Menell (2023)Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research, Part II, Table 1. Note: [https://doi.org/10.1111/j.1467-9698-2023-x](https://doi.org/10.1111/j.1467-9698-2023-x) Cited by: SS1.
* A. Paula Chaves and M. Aurelio Gerosa (2021)How should my chatbot interact? a survey on social characteristics in human-chatbot interaction design. International Journal of Human-Computer Interaction37 (8), pp. 729-758. External Links: ISSN 0002-9993, Link, Document Cited by: SS1.
* Chi Yue Chiu and N. Bendapudi (2012)Some thoughts on social metacognition. PsycCRITIQUES57. External Links: Link, Document Cited by: SS1.
* Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi (2024)Culturalbench: a robust, diverse and challenging benchmark on measuring the (lack of) cultural knowledge of lllms. PreprintarXiv:2410.02677. External Links: Link, 2410.02677 Cited by: SS1.
* Noam Chomsky (2017)Language architecture and its import for evolution. Neuroscience Biobehavioral Reviews81, pp. 295-300. External Links: Document, Link Cited by: SS1.
* Z. Chu, J. Chen, Q. Chen, W. Yu, T. He, H. Wang, W. Peng, M. Liu, B. Qin, and T. Liu (2024)Navigate through enigmatic labyrinth a survey of chain of thought reasoning: advances, frontiers and future. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, pp. 1173-1203. External Links: Link, Document Cited by: SS1.
* Robert B Cialdini (2001)The science of persuasion. Scientific American284 (2), pp. 76-81. External Links: ISSN 0002-9993, Link, Document Cited by: SS1.
* Leon Ciechanowski, Aleksandra Przegalinska, Mikolaj Magnuski, and Peter Gloor (2019)In the shades of the uncanny valley: an experimental study of human-chatbot interaction. Future Generation Computer Systems92, pp. 539-548. External Links: ISSN 0002-9993, Link, Document Cited by: SS1.
* Adam B Cohen (2009)Many forms of culture. American psychologist64 (3), pp. 194. External Links: ISSN 0002-9993, Link, Document Cited by: SS1.
* Michael Cole and Martin Packer (2019)Colture and cognition. Cross-cultural psychology: contemporary themes and perspectives, pp. 243-270. External Links: ISBN 978-3-319-50356-3 Cited by: SS1.
* Allan Collins and Dedre Gentner (1987)How people construct mental models. Cultural models in language and thought243 (1987), pp. 243-265.

 

* Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. _Advances in neural information processing systems_, 32.
* Cope (2022) Edward Meredith Cope. 2022. _An introduction to Aristotle's rhetoric_. BoD-Books on Demand.
* Dam et al. (2024) Sumit Kumar Dam, Choong Seon Hong, Yu Qiao, and Chaoning Zhang. 2024. A complete survey on llm-based ai chatbots. _Preprint_, arXiv:2406.16937.
* Dammu et al. (2024) Preetam Prabhu Srikar Dammu, Hayoung Jung, Anjali Singh, Monojit Choudhury, and Tanu Mitra. 2024. "they are uncultured": Unveiling covert harms and social threats in LLM generated conversations. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 20339-20369, Miami, Florida, USA. Association for Computational Linguistics.
* D'Andrade (1995) R.G. D'Andrade. 1995. _The Development of Cognitive Anthropology_. The Development of Cognitive Anthropology. Cambridge University Press.
* Dawson et al. (2024) Fiifi Dawson, Zainab Mosunmola, Sahil Pocker, Raj Abhijit Dandekar, Rajat Dandekar, and Sreedath Panat. 2024. Evaluating cultural awareness of lms for yoruba, malayalam, and english. _Preprint_, arXiv:2410.01811.
* Denzaur et al. (1994) Arthur T Denzaur, Douglass C North, et al. 1994. Shared mental models: ideologies and institutions. _KYKLOSERNE-_, 47:3-3.
* Dong et al. (2024) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024. A survey on in-context learning. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 1107-1128, Miami, Florida, USA. Association for Computational Linguistics.
* Dubey et al. (2021) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhli Mathur, Alan Schelten, Amy Yang, Angela Fan, anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Stavankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damiel Molonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Huples, Egor Lakomkin, Ehab AlbBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenoviev, Frank Zhang, Gabriel Synnaeve, Gabrieille Lee, Georgia Lewis Anderson, Graeme Neil, Grezoire Malon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jane Vranes, Jason Park, Jay Mahadeokar, Jeeth Shah, Jelmer van der Lindle, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasudeen Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kishitz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Ratalia-Years, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madanan, Lubo Malo, Lukas BECher, Lukas Landzaat, Luke de Oliveira, Madeline Munzi, Mahesh Pasupulei, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Namam Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Celebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vaisc, Peter Weng, Prijwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Siliveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Komain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennashaspapa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliasing Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengyue Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gurrangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tare Sheaisha, Thomas Georgiou, Thomas Sciaciom, Tobias Speckbacher, Todof Mihalyo, Tong Xiao, Ujjwal Kar, Vedantiy Goswarni, Viibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Voggeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyn Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaqiang Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldischang, Yashensu Gaur, Yasmine Babaei, Yi Wen, Yusen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierce Coudert, Zheng Yan, Zhengking Chen, Zoe Papakipos, Aadirus Singh, Aaron Grattafori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahiva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Beaevski, Allie Feinstein, Amanda Kalllet, Amit Sangani, Anann Yuns Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwim Bharamebe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Bet De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Knock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changky 

Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Caii, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Akkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzman, Frank Kanayet, Frank Seide, Gabriela Medina Floreze, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspeegren, Hunter Goldman, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Got, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Rezienstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingji Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartiky Khanadelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lalilin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeno Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Markus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Luthi, Meghan Kenenally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Yvatskoy, Mikayed Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanet, Mohammad Rastegari, Munish Bansal, Nandhnini Santhanam, Natascha Parks, Natasha White, Navyata Baswa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovi Lapptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernouguz, Olivia Hart, Omkar Salpkear, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Pontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghothamurthy, Raghu Navami, Rahul Mitra, Raymond Li, Rebekku Hsogan, Robin Battery, Rocky Wang, Rohan Maheswarni, Russ Howes, Ruthor Bindi, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shawn Langsany, Sheng Feng, Shengahon Lin, Shengxin Cindy Zhao, Shiva Shankar, Shuqiang Zhang, Shuqiang Wang, Sneha Agarwal, Soji Sayigibye, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tinante Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzok Shaked, Varnu Vontimitita, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Victor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tibeivin Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xie, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngsin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. 2024. The llama 3 herd of models. _Preprint_, arXiv:2407.21783.
* Dudy et al. (2024) Shiran Dudy, Ibrahim Said Ahmad, Ryoko Kitajima, and Agata Lapedriza. 2024. Analyzing cultural representations of emotions in llms through mixed emotion survey. _arXiv preprint arXiv:2408.02143_.
* Eckert (2012) Penelope Eckert. 2012. Three waves of variation study: The emergence of meaning in the study of sociolinguistic variation. _Annual review of Anthropology_, 41(1):87-100.
* Eckert and Rickford (2001) Penelope Eckert and John R Rickford. 2001. _Style and sociolinguistic variation_. Cambridge University Press.
* Fan et al. (2024) Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijile Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In _Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_, pages 6491-6501.
* Ferraro (2019) Andres Ferraro. 2019. Music cold-start and long-tail recommendation: bias in deep representations. In _Proceedings of the 13th ACM Conference on Recommender Systems_, RecSys '19, page 586-590, New York, NY, USA. Association for Computing Machinery.
* Fitch et al. (2005) W. Tecungseh Fitch, Marc D. Hauser, and Noam Chomsky. 2005. The evolution of the language faculty: Clarifications and implications. _Cognition_, 97(2):179-210.
* Frohlich et al. (2019) Marlen Frohlich, Christine Sievers, Simon W Townsend, Thibaud Gruber, and Carel P van Schaik. 2019. Multimodal communication and language origins: integrating gestures and vocalizations. _Biological Reviews_, 94(5):1809-1829.
* Fu et al. (2024)Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024. Break the sequential dependency of llvm inference using lookahead decoding. In _Proceedings of the 41st International Conference on Machine Learning_, ICML'24. JMLR.org.
* Gao et al. (2023) Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for 

large language models: A survey. arXiv preprint arXiv:2312.10997. Cited by: SS4.
* H. Geert and G. J. Hofstede (2004)Cultures and organizations. Software of the Mind, Vol. 2. External Links: Link Cited by: SS5.
* C. Goertz (1973)The interpretation of cultures. Basic books. External Links: ISBN 978-3-642-4655-1, Document Cited by: SS1.
* J. Greive, S. Bartl, M. Fuoli, J. Grafmiller, W. Huang, A. Juarberbaum, A. Murakami, M. Perlman, D. Roemling, and B. Winter (2025)The sociolinguistic foundations of language modeling. Frontiers in Artificial Intelligence7, pp. 1472411. External Links: ISSN 0021-921X, Document Cited by: SS1.
* S. Griffith, K. Subramanian, J. Scholz, C. L. Isbell, and A. L. Thomaz (2013)Policy shaping: integrating human feedback with reinforcement learning. Advances in neural information processing systems26. External Links: Link Cited by: SS1.
* G. Haffari, T. D. Tran, and M. Carman (2017)Efficient benchmarking of NLP APIs using multi-armed bandits. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, Valencia, Spain, pp. 408-416. External Links: Link Cited by: SS1.
* Z. Harris (1951)Methods in structural linguistics. External Links: Link Cited by: SS1.
* M. D. Hauser, C. Yang, R. C. Berwick, I. Tattersall, M. J. Ryan, J. Watumull, N. Chomsky, and R. C. Lewontin (2014)The mystery of language evolution. Frontiers in Psychology5. External Links: Link Cited by: SS1.
* M. Henaff (1998)Claude Levi-strauss and the making of structural anthropology. U of Minnesota Press. External Links: ISBN 978-3-642-4655-1, Document Cited by: SS1.
* J. Henrich, S. J. Heine, and A. Norenzayan (2010)The weirdest people in the world?. Behavioral and brain sciences33 (2-3), pp. 61-83. External Links: ISSN 0021-921X, Document Cited by: SS1.
* G. Hofstede (2001)Culture's consequences: comparing values, behaviors, institutions and organizations across nations. Note: [https://www.worldworld.com/en/article/pdf/doi=10.1111/j.1067-978-3-642-4655-1](https://www.worldworld.com/en/article/pdf/doi=10.1111/j.1067-978-3-642-4655-1), Link Cited by: SS1.
* T. Hospedales, A. Antoniou, P. Macarella, and A. Storkey (2021)Meta-learning in neural networks: a survey. IEEE transactions on pattern analysis and machine intelligence44 (9), pp. 5149-5169. External Links: ISSN 0021-921X, Document Cited by: SS1.
* Y. Hu, Y. Koren, and C. Volinsky (2008)Collaborative filtering for implicit feedback datasets. In 2008 Eighth IEEE international conference on data mining, pp. 263-272. Cited by: SS1.
* L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu (2025)A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst.43 (2). External Links: ISSN 0021-921X, Document Cited by: SS1.
* G. Jahoda and I. Lewis (2015)Acquiring culture (Psychology Revivals): cross cultural studies in child development. Psychology Press. External Links: ISBN 978-3-642-4655-1, Document Cited by: SS1.
* A. Jia, A. Mostafazadeh Davani, C. K. Reddy, S. Dave, V. Prabhakaran, and S. Dev (2023)SeeGULL: a stereotype benchmark with broad geo-cultural coverage leveraging generative models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 9851-9870. External Links: ISBN 978-3-642-4655-1, Document Cited by: SS1.
* Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. Jang, A. Madotto, and P. Fung (2023)Survey of hallucination in natural language generation. ACM Comput. Surv..55 (12). External Links: ISSN 0021-921X, Document Cited by: SS1.
* J. Jin, J. Kim, N. Lee, H. Yoo, A. Oh, and H. Lee (2024)KoBBQ: Korean bias benchmark for question answering. Transactions of the Association for Computational Linguistics12, pp. 507-524. External Links: ISSN 0021-921X, Document Cited by: SS1.
* D. H. Jonassen and Philip Henning (1999)Mental models: knowledge in the head and knowledge in the world. Educational technology, pp. 37-42. External Links: ISSN 0021-921X, Document Cited by: SS1.
* R. Keller (1994)On language change: the invisible hand in language. Psychology Press. External Links: ISBN 978-3-642-4655-1, Document Cited by: SS1.
* J. Kharchenko, T. Roosta, A. Chadha, and C. Shah (2024)How well do lllms represent values across cultures? empirical analysis of llvm responses based on hofstede cultural dimensions. arXiv preprint arXiv:2406.14805. External Links: Link Cited by: SS1.
* J. Kharchenko, T. Roosta, A. Chadha, and C. Shah (2024)How well do llms represent values across cultures? empirical analysis of llvm responses based on hofstede cultural dimensions. Preprint arXiv:2406.14805. External Links: Link Cited by: SS1.
* J. Kim, Y. Kim, R. Sarikaya, and E. Fosler-Lussier (2017)Cross-lingual transfer learning for pos tagging without cross-lingual resources. In Proceedings of the 2017 conference on empirical methods in natural language processing, pp. 2832-2838. External Links: ISBN 978-3-642-4655-1, Document Cited by: SS1.
* F. Koto, N. Aisyah, H. Li, and T. Baldwin (2023)Large language models only pass primary school exams in Indonesia: a comprehensive test on IndomMLU. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 12359-12374. External Links: ISBN 978-3-642-4655-1, Document Cited by: SS1.
* F. Koto, R. Mahendra, N. Aisyah, and T. Baldwin (2024)IndoCulture: exploring geographically influenced cultural commonsense reasoning across eleven Indonesian provinces. Transactions of the Association for Computational Linguistics12, pp. 1703-1719.

 

Gryur Kovac, Masataka Sawayama, Remy Portelas, Cedric Colas, Peter Ford Dominey, and Pierre-Yves Oudeyer. 2023. Large language models as superpositions of cultural perspectives. _Preprint_, arXiv:2307.07870.
* Kuper (1988) A. Kuper. 1988. _The Invention of Primitive Society: Transformations of an Illusion_. Routledge.
* LeCompte and Schensul (2010) Margaret Diane LeCompte and Jean J Schensul. 2010. _Designing & conducting ethnographic research: An introduction_, volume 1. Rowman Altamira.
* Lent et al. (2024) Heather Lent, Kwshal Tatariya, Raj Dabre, Yiyi Chen, Marcell Fekete, Esther Ploeger, Li Zhou, RuthAnn Armstrong, Abe Eijansantos, Catriona Malau, Hans Erik Heje, Ernestes Lavrinovics, Diptesh Kanojia, Paul Belony, Marcel Bollmann, Loic Grobol, Miryam de Lloneux, Daniel Hershcovich, Michel DeGraff, Anders Sogaard, and Johannes Bjerva. 2024. CreoleVal: Multilingual Multitask Benchmarks for Creoles. _Transactions of the Association for Computational Linguistics_, 12:950-978.
* Leung et al. (2013) Angela K-y Leung, Sau-lai Lee, and Chi-yue Chiu. 2013. Meta-knowledge of culture promotes cultural competence. _Journal of Cross-Cultural Psychology_, 44(6):992-1006.
* Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Pitkus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kutler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. _Advances in Neural Information Processing Systems_, 33:9459-9474.
* LI et al. (2024a) CHENG LI, Mengzhuo Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. 2024a. CultureLLM: Incorporating cultural differences into large language models. In _The Thirty-eighth Annual Conference on Neural Information Processing Systems_.
* LI et al. (2024b) CHENG LI, Damien Teney, Linyi Yang, Qingsong Wen, Xing Xie, and Jindong Wang. 2024b. Culturepark: Boosting cross-cultural understanding in large language models. In _The Thirty-eighth Annual Conference on Neural Information Processing Systems_.
* Li et al. (2022) Huyang Li, Yinuan Su, Deng Cai, Yan Wang, and Lemao Liu. 2022. A survey on retrieval-augmented text generation. _arXiv preprint arXiv:2202.01110_.
* Li et al. (2024) Huihan Li, Liwei Jiang, Nouha Dziri, Xiang Ren, and Yejin Choi. 2024. CULTURE-GEN: Revealing global cultural perception in language models through natural language prompting. In _First Conference on Language Modeling_.
* Lichtenberg et al. (2024) Jan Malte Lichtenberg, Alexander Buchholz, and Pola Schwobel. 2024. Large language models as recommender systems: A study of popularity bias. _Preprint_, arXiv:2406.01285.
* Lightfoot (2002) David W Lightfoot. 2002. Explaining language change: An evolutionary approach.
* Liu and Mazumder (2020) Bing Liu and Sahisnu Mazumder. 2020. Lifelong learning dialogue systems: Chatbotis that self-learn on the job. _arXiv preprint arXiv:2009.10750_.
* Liu et al. (2024) Chen Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2024. Are multilingual LLMs culturally diverse reasoners? an investigation into multicultural proverbs and sayings. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 2016-2039, Mexico City, Mexico. Association for Computational Linguistics.
* Lu et al. (2010) Tyler Lu, David Pal, and Martin Pal. 2010. Contextual multi-armed bandits. In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_, volume 9 of _Proceedings of Machine Learning Research_, pages 485-492, Chia Laguna Resort, Sardinia, Italy. PMLR.
* McCorkle and Xygalatas (2013) W.W. McCorkle and D. Xygalatas. 2013. _Mattal Culture: Classical Social Theory and the Cognitive Science of Religion_. Religion, cognition and culture. Acumen.
* McHugh et al. (2008) Anna P McHugh, Jennifer L Smith, and Winston R Sieck. 2008. Cultural variations in mental models of collaborative decision making. _Macrocognition and naturalistic decision making_, pages 141-158.
* McIntosh et al. (2024) Timothy R. McIntosh, Tong Liu, Teo Susnjak, Paul Watters, Alex Ng, and Malka N. Halgamuge. 2024. A culturally sensitive test to evaluate nuanced gpt hallucination. _IEEE Transactions on Artificial Intelligence_, 5(6):2739-2751.
* Minaee et al. (2024) Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. _arXiv preprint arXiv:2402.06196_.
* Mishra (2001) Ramesh C Mishra. 2001. Cognition across cultures. _The handbook of culture and psychology_, pages 119-135.
* Moerchen et al. (2020) Fabian Moerchen, Patrick Ernst, and Giovanni Zappella. 2020. Personalizing natural-language understanding using multi-armed bandits and implicit feedback. In _CIKM 2020_.
* Monaghan et al. (2012) Leila Monaghan, Jane E Goodman, and Jennifer Robinson. 2012. _A cultural approach to interpersonal communication: Essential readings_. John Wiley & Sons.
* Montague et al. (1970) Richard Montague et al. 1970. Universal grammar. _1974_, pages 222-46.
* Montalan et al. (2024) Jann Railey Montalan, Jian Gang Ngui, Wei Qi Leong, Yosephine Susanto, Hamsawardhnii Rengarajan, William Chandra Tjhi, and Alham Fikri Aji. 2024. Kalahdi: A handcrafted, grassroots cultural llm evaluation suite for filipino. _Preprint_, arXiv:2409.15380 

Aida Mostafazadeh Davani, Mark Diaz, Dylan K Baker, and Vinodkumar Prabhakaran. 2024. D3CODE: Disentangling disagreements in data across cultures on offensiveness detection and evaluation. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 18511-18526, Miami, Florida, USA. Association for Computational Linguistics.
* Mukherjee et al. (2024) Sagnik Mukherjee, Muhammad Farid Adilazuarda, Sunayana Siitaram, Kalika Bali, Alham Fikri Aji, and Monojit Choudhury. 2024. Cultural conditioning or placebo? on the effectiveness of socio-demographic prompting. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 15811-15837, Miami, Florida, USA. Association for Computational Linguistics.
* Munch and Smelser (1992) Richard Munch and Neil J Smelser. 1992. _Theory of culture_. University of California Press Berkeley.
* Myung et al. (2024) Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Puttri, Dimosthenis Antypas, Hsuvas Borkakovty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, Victor Gutierrez Basulto, Yazmin Ibanez-Garcia, Hwaran Lee, Shamsudeen Hassan Muhammad, Kiwoong Park, Anar Sabuli Rzayev, Nina White, Seid Muhie Yimam, Mohammad Taher Pilhevar, Nedjma Osidhoum, Jose Camacho-Collados, and Alice Oh. 2024. BLEnd: A benchmark for LLMs on everyday knowledge in diverse cultures and languages. In _The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
* Nadeem et al. (2021) Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 5356-5371, Online. Association for Computational Linguistics.
* Nangia et al. (2020) Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1953-1967, Online. Association for Computational Linguistics.
* Nishett and Norenzayan (2002) Richard E Nishett and Ara Norenzayan. 2002. Culture and cognition. John Wiley & Sons.
* Noshadi and Dabbagh (2015) Mahdi Noshadi and Ali Dabbagh. 2015. Metacultural competence: A benchmark for advances in applied elt. In _Selected Papers of 2nd conference on Interdisciplinary Approaches to Language Teaching, Literature and Translation Studies_, pages 52-62. Ferdowsi university of Mashhad, Khate Sefid English Language Group.
* Ortiz et al. (2013) Anna M Ortiz, Long Beach, et al. 2013. The ethnographic interview. In _Research in the college context_, pages 51-64. Routledge.
* Owen et al. (2024) Louis Owen, Vishesh Tripathi, Abhay Kumar, and Biddwan Ahmed. 2024. Komodo: A linguistic expedition into indonesia's regional languages. _Preprint_, arXiv:2403.09362.
* Pandey et al. (2025) Saurabh Kumar Pandey, Harshit Budhiraja, Sougata Saha, and Monojit Choudhury. 2025. CULTURALLY YOURS: A reading assistant for cross-cultural content. In _Proceedings of the 31st International Conference on Computational Linguistics: System Demonstrations_, pages 208-216, Abu Dhabi, UAE. Association for Computational Linguistics.
* Parsons (1972) Talcott Parsons. 1972. Culture and social system revisited. _Social Science Quarterly_, pages 253-266.
* Perniss (2018) Pamela Perniss. 2018. Why we should study multimodal language. _Frontiers in psychology_, 9:1109.
* Petroni et al. (2019) Fabio Petroni, Tim Rocktaschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
* Prakken (2006) Henry Prakken. 2006. Formal systems for persuasion dialogue. _The knowledge engineering review_, 21(2):163-188.
* Pryant et al. (2023) Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with "gradient descent" and beam search. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 7957-7968, Singapore. Association for Computational Linguistics.
* Purti et al. (2024) Rifki Afina Purti, Faiz Ghifari Haznitrama, Dea Andihista, and Alice Oh. 2024. Can LLM generate culturally relevant commonsense QA data? case study in Indonesian and Sundanese. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 20571-20590, Miami, Florida, USA. Association for Computational Linguistics.
* Rao et al. (2024) Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. 2024. Normad: A benchmark for measuring the cultural adaptability of large language models. _Preprint_, arXiv:2404.12464.
* Rao et al. (2023) Abhinav Sukumar Rao, Aditi Khandelwal, Kumar Tanmay, Utkarsh Agarwal, and Monojit Choudhury. 2023. Ethical reasoning over moral alignment: A case and framework for in-context ethical policies in LLMs. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 13370-13388, Singapore. Association for Computational Linguistics 

Amon Rapp, Lorenzo Curti, and Arianna Boldi. 2021. The human side of human-chatbot interaction: A systematic literature review of ten years of research on text-based chatbots. _International Journal of Human-Computer Studies_, 151:102630.
* Rawte et al. (2023) Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A survey of hallucination in large foundation models. _Preprint_, arXiv:2309.05922.
* Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5418-5426, Online. Association for Computational Linguistics.
* Saha (2024) Sougata Saha. 2024. _Persuasive Dialogue Systems for Social Good_. Ph.D. thesis, State University of New York at Buffalo.
* Schein (1990) Edgar H Schein. 1990. _Organizational culture._, volume 45. American Psychological Association.
* Sclar et al. (2024) Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In _The Twelfth International Conference on Learning Representations_.
* Seth et al. (2024) Agrima Seth, Sanchit Ahuja, Kalika Bali, and Sunayana Sitaram. 2024. DOSA: A dataset of social artifacts from different Indian geographical subcultures. In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_, pages 5323-5337, Torino, Italia. ELRA and ICCL.
* Sewell (2004) William H Sewell. 2004. The concept (s) of culture. In _Practicing history_, pages 76-95. Routledge.
* Sharifian (2013) Farzad Sharifian. 2013. Globalisation and developing metacultural competence in learning english as an international language. _Multilingual education_, 3:1-11.
* Skinner et al. (2013) Jonathan Skinner et al. 2013. _The interview: An ethnographic approach_, volume 49. A&C Black.
* Slivkins (2019) Aleksandrs Slivkins. 2019. Introduction to multi-armed bandits. _Foundations and Trends(r) in Machine Learning_, 12(1-2):1-286.
* Soper et al. (2022) Elizabeth Soper, Erin Pacquetet, Sougata Saha, Souvik Das, and Rohini Srihari. 2022. Let's chat: Understanding user expectations in socialbot interactions. In _Proceedings of the Second Workshop on Bridging Human-Computer Interaction and Natural Language Processing_, pages 34-39, Seattle, Washington. Association for Computational Linguistics.
* Spencer-Oatey and Franklin (2012) Helen Spencer-Oatey and Peter Franklin. 2012. What is culture. _A compilation of quotations. GlobalPAD Core Concepts_, 1(22):1-21.
* Spradley (2016) James P Spradley. 2016. _The ethnographic interview_. Waveland Press.
* Strauss (1974) C Levi Strauss. 1974. Structural anthropology. _Persona & Derecho_, 1:571.
* Sun et al. (2020) Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. 2020. {LAMAL}: {LA}nguage modeling is all you need for lifelong language learning. In _International Conference on Learning Representations_.
* Sun et al. (2021) Jimin Sun, Huijeen Ahn, Chan Young Park, Yulia Tsvetkov, and David R. Mortensen. 2021. Cross-cultural similarity features for cross-lingual transfer learning of pragmatically motivated tasks. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 2403-2414, Online. Association for Computational Linguistics.
* Tagliamonte (2006) Sali A Tagliamonte. 2006. _Analysing sociolinguistic variation_. Cambridge University Press.
* Tanmay et al. (2023) Kumar Tanmay, Aditi Khandelwal, Utkarsh Agarwal, and Monojit Choudhury. 2023. Probing the moral development of large language models through defining issues test. _Preprint_, arXiv:2309.13356.
* Taylor (2001) Stephanie Taylor. 2001. _Ethnographic research: A reader_. Sage.
* Thompson et al. (2020) B Thompson, SG Roberts, and G Lupyan. 2020. Cultural influences on word meanings revealed through large-scale semantic alignment. nature human behaviour, 4 (10), 1029-1038.
* Thompson et al. (2011) William Forde Thompson et al. 2011. Cross-cultural similarities and differences.
* Urban (2010) Greg Urban. 2010. A method for measuring the motion of culture. _American Anthropologist_, 112(1):122-139.
* Vaicenavicius et al. (2019) Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schon. 2019. Evaluating model calibration in classification. In _The 22nd international conference on artificial intelligence and statistics_, pages 3459-3467. PMLR.
* Vansthoren (2019) Joaquin Vansthoren. 2019. Meta-learning. _Automated machine learning: methods, systems, challenges_, pages 35-61.
* Vigliocco et al. (2014) Gabriella Vigliocco, Pamela Permiss, and David Vinson. 2014. Language as a multimodal phenomenon: implications for language learning, processing and evolution.
* Wan et al. (2023) Yixin Wan, Jieyu Zhao, Arnan Chadha, Nanyun Peng, and Kai-Wei Chang. 2023. Are personalized stochastic parrots more dangerous? evaluating persona biases in dialogue systems. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 9677-9705, Singapore. Association for Computational Linguistics 

* Wang (2021) Jane X Wang. 2021. Meta-learning in natural and artificial intelligence. _Current Opinion in Behavioral Sciences_, 38:90-95.
* Wang et al. (2024) Yuhang Wang, Yanxu Zhu, Chao Kong, Shuyu Wei, Xiaoyuan Yi, Xing Xie, and Jitao Sang. 2024. CDEval: A benchmark for measuring the cultural dimensions of large language models. In _Proceedings of the 2nd Workshop on Cross-Cultural Considerations in NLP_, pages 1-16, Bangkok, Thailand. Association for Computational Linguistics.
* Welleck et al. (2024) Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui. 2024. From decoding to meta-generation: Inference-time algorithms for large language models. _Transactions on Machine Learning Research_. Survey Certification.
* Whitehead (2005) Tony L Whitehead. 2005. Basic classical ethnographic research methods. _Cultural ecology of health and change_, 1:1-29.
* Wibowo et al. (2024) Haryo Wibowo, Erland Fuadi, Made Nityasya, Radityo Eko Prasaojo, and Alham Aji. 2024. COPAL-ID: Indonesian language reasoning with local culture and nuances. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 1404-1422, Mexico City, Mexico. Association for Computational Linguistics.
* Yang et al. (2017) Charles Yang, Stephen Crain, Robert C Bervick, Noam Chomsky, and Johan J Bollhuis. 2017. The growth of language: Universal grammar, experience, and principles of computation. _Neuroscience & Biobehavioral Reviews_, 81:103-119.
* Yang et al. (2024) Seongjun Yang, Gibbeum Lee, Jaewoong Cho, Dimitris Papailioulos, and Kangwook Lee. 2024. Predictive pipelined decoding: A compute-latency trade-off for exact LLM decoding. _Transactions on Machine Learning Research_.
* Yin et al. (2022) Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. 2022. Geomlama: Geo-diverse commonsense probing on multilingual pre-trained language models. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 2039-2055.
* Yin et al. (2012) Hongzhi Yin, Bin Cui, Jing Li, Junjie Yao, and Chen Chen. 2012. Challenging the long tail recommendation. _Proc. VLDB Endow._, 5(9):896-907.
* Zhao et al. (2024) Jiajing Zhao, Cheng Huang, and Xian Li. 2024. A comparative study of cultural hallucination in large language models on culturally specific ethical questions.
* Zheng et al. (2024) Junhao Zheng, Shengjie Qiu, Chengming Shi, and Qianli Ma. 2024. Towards lifelong learning of large language models: A survey. _arXiv preprint arXiv:2406.06391_.
* Zhou et al. (2024) Li Zhou, Taelin Karidi, Nicolas Garneau, Yong Cao, Wanlong Liu, Wenyu Chen, and Daniel Hershcovich. 2024. Does mapo tofiu contain coffee? probing llms for food-related cultural knowledge. _Preprint_, arXiv:2404.06833.

 

Appendix

[TABLE:A1.T2][ENDTABLE]

\begin{table}
\begin{tabular}{|l|p{340.7pt}|p{85.4pt}|} \hline
**Sl No** & **Semantic Domain** & **Full Question** \\ \hline
1 & Weight Unit & What is the unit of measuring weight? \\
2 & Drinking Hot Water & Is it rare or common to see people drink hot water? \\
3 & Climate Zone & Which climate zone does the country belong to? \\
4 & Shower Time & What time of the day people usually take the shower? \\
5 & Driving Side & Which side do people usually keep when driving? \\
6 & Household Servants & Is it rare or common for households to have servants? \\
7 & Past Transportation & What was the most popular mode of transportation in the big cities 30 years back? \\
8 & Food Sharing & Is it rare or common for people to share their food when they eat out? \\
9 & Date Format (Year) & Does the year appear before/after the month in the date format? \\
10 & Driver Side & Which side of the car is the driver seat? \\
11 & Broom for Cleaning & Is it rare or common for people to use broom to clean the floor? \\
12 & Date Format (Month) & Does the month appear before/after the year in the date format? \\
13 & Living with Parents & Is it rare or common for adults to live with their parents? \\
14 & Drying Clothes & How do people dry their wet clothes? \\
15 & Wedding Duration & Does a wedding ceremony last for more than one day? \\
16 & Popular Sports & What are the most popular sports? \\
17 & Stock Drop Color & Which color represents the drop in the stock prices? \\
18 & Eating Tools & Which tools do people usually eat food with? \\
19 & Height Unit & What is the unit of measuring height? \\
20 & Meal Tips & Is it rare or common that customers pay tips after a meal? \\
21 & Temperature Unit & What is the unit of measuring temperature? \\
22 & Stock Rise Color & Which color represents the rise in the stock prices? \\
23 & Bridal Outfit Color & What is the color of the bridal outfit in a wedding? \\
24 & Funeral Press & What is the color of the dress that people wear in a traditional funerals? \\
25 & Staple Food & What is the staple food? \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 2: Full text of abbreviated questions in Figure 1.

Figure 2: Prompt used to get GeoMLAMA question logits from Llama.