Attributing Culture-Conditioned Generations to Pretraining Corpora
Authors: Huihan Li
Arnav Goel
Keyu He
Xiang Ren
Venue: ICLR 2025 Poster (2025)
Keywords: culture bias
pretraining data
memorization
generalization
Primary area: interpretability and explainable AI
TL;DR: We propose the MEMOED framework to determine whether a biased culture-conditioned generation arises from memorization of pretraining data patterns, or results from other associations made due to pretraining data.

Abstract: In open-ended generative tasks like narrative writing or dialogue, large language models often exhibit cultural biases, showing limited knowledge and generating templated outputs for less prevalent cultures. Recent works show that these biases may stem from uneven cultural representation in pretraining corpora. This work investigates how pretraining leads to biased culture-conditioned generations
by analyzing how models associate entities with cultures based on pretraining data patterns. We propose the MEMOED framework (MEMOrization from prEtraining Document) to determine whether a generation for a culture arises from memorization. Using MEMOED on culture-conditioned generations about food and clothing for 110 cultures, we find that high-frequency cultures in pretraining data yield more generations with memorized symbols, while some low-frequency cultures produce none. Additionally, the model favors generating entities with extraordinarily high frequency regardless of the conditioned culture, reflecting biases toward frequent pretraining terms irrespective of relevance. We hope that the MEMOED framework and our insights will inspire more works on attributing model performance on pretraining data.

Published as a conference paper at ICLR 2025
ATTRIBUTING CULTURE-CONDITIONED GENERATIONS TO PRETRAINING CORPORA
Huihan Li
Arnav Goel
Keyu He'
Xiang Ren
University of Southern California (huihanl, frankhe, xi attp:eenneeuccee.aaaaaaaaaaaaaaa
'III Delhi
.n
ABSTRACT
In open-ended generative tasks like narrative writing or dialogue, large language models often exhibit cultural biases, showing limited knowledge and generating templated outputs for less prevalent cultures. Recent works show that these bi- ases may stem from uneven cultural representation in pretraining corpora. This generations
work investigates how pretraining leads to biased culture-conditione by analyzing how models associate entities with cultures based on pretraining data patterns. We propose the MEMOED framework (MEMOrization from pretraining document) to determine whether a generation for culture arises from memorization. Using MEMOED on culture-conditioned generations about food and clothing for 110 cultures, we find that high-frequency cultures in pretraining data yield more generations with memorized symbols, while some low-frequency cultures produce none. Additionally, the model favors generating entities with extraordinarily high frequency regardless of the conditioned culture, reflecting bi- ases toward frequent pretraining terms irrespective of relevance. We hope that the MEMOED framework and our insights will inspire more works on attribut- ing model performance pretraining data!" [Disclaimer: This analysis does not represent any views or beliefs the authors. Our findings reflect trends observed specifically within OLMo-7B's pretraining data and are limited to this dataset. We make no claims about whether these results reflect real-world conditions.]
INTRODUCTION
In open-ended generative tasks like narrative writing or dialogue, language models often show bias against marginalized social groups based on gender, race, or culture (Gallegos et al. 20224] Manvi al. 2024 et al., 2024b). Cultural bias is particularly notable due the vast number of cul- tures to account for. Cultures are often unevenly represented in the pretraining corpora, with some mentioned more frequently than others, irrespective of their world prevalence (Li et al. 2024a) Recent studies reveal that models favor entities (Naous al.) (023) and opinions (Ryan et al. (024) from frequently represented cultures in pretraining while showing inadequate knowledge and tem- plated answers for less frequent ones (Li ct al. 2024b).
Such biases culture-conditioned generations can be linked to studies showing that LLMs' memo- rization and generalization are constrained by pretraining data imbalances. Zhang et al. (024) find that these imbalances cause models to overgeneralize to high-frequency knowledge, overshadowing lower-frequency knowledge. Similarly, [Chang et al. (2024) observe that LLMs struggle with gener- ating long-tail knowledge in downstream tasks when such knowledge appears with intervals longer than threshold in pretraining data enable memorization.
Building on these findings, we analyze culture biases by examining how models associate entities, referred to "symbols." with cultures based patterns in pretraining data (e.g., "kimono" asso- ciated with Japan). We investigate three key questions: 1) How can we determine if a symbol is generated for a culture due memorization of their association? If memorization, what other factors drive the model's s association? 3) How are these types of associations tied to pretraining data frequency imbalances?
"Equal Contribution

---

Published as a conference paper at ICLR 2025
MEMORIZED ASSOCIATION
DIFFUSE ASSOCIATION Symbal accariig greater
Describe the food of your neighbor. "My neighbor is Indiann. He probably likes
Cultue-specfiic symbols that are memorirec from the pre tra ning data.
generations.
naan, samosas, mango lassi, paneer, biryani, dal, saag, dosa, curd rice, roti, pickle, chapatti, rice, chicken, chickpea, garlic, fried banana, apple pie
WEAK ASSOCIATION GENERALIZATION Symbolk which inferned from memorized symbols ciffuse association.
BROSS-CULTURE GENERALIZATION
Symbels which memorized symbbols from other cultures.
Figure I: Four types of culture-symbol associations in culture-conditioned generations
To address the first question, we propose MEMOED (MEMOrization from pretraining document), a framework to determine if symbols in culure-condiiioned generations arise from the model memorizing culture-symbo relationships in pretraining data. MEMOED involves two steps: 1) it identifies contributory documents in the pretraining corpora for the culture-symbe association 2) classifies the symbol as memorized if the percentage of contributory documents is significant ($3.3).
To answer the second question, we analyze OLMo-7 73 (Groeneveld et al. 2024) and its pretraining
data Do Do (Soldaini et al. 2024), indexed by Elazar al. 22224) and Liu et al. (2024) Follow-
ing (Li et al. 2024b). we collect OLMo - 7B's culture-con condii-conddiionedd generations about 110 cultures on food and clothing topics and extract topic-related symbols.
Using MEMOED, we find that 46% of food symbols and 26% of clothing symbols are generated due memorization of culture-symbol relationships in pretraining data. For the remaining sym- bols, we identify three other types of culture-symbr associations: 1) Diffuse Association: symbols not strongly tied to any specific culture but appearing in over half of all cultures' generations (e.g., "t-shirt")) 2) Cross-culture Generalization: symbols memorized with one culture but generated for others (e.g., "kimono" a memorized symbol for Japan but generated for Korea), and 3) Weak Asso- ciation Generalization: broader conceptual representations derived from synthesis or reinterpreta- tion of memorized symbols and diffuse association symbols (e.g., "robe" as a generalized reference to "kimono," symbol memorized for Japan).
To explore the third question, we find strong correlations between three of the four association cat- egories and frequency patterns in pretraining data. Memorized associations strongly correlate with a culture's s occurrence frequency, indicating that low-frequency cultures in the "long-tail" lack suf- ficient pretraining supervision for memorization. For diffuse associations, higher-frequency sym- bols are generated for more cultures, despite not being tied to any specific culture. Cross-culture generalization shows that cultures with higher pretraining frequencies are more likely have their memorized symbols generated for others. Weak association generalization does not correlate signifi- cantly with any pretraining pattern, but cultures without memorized symbols often generate symbols generalized from high-frequency symbols or symbols tied to high-frequency cultures.
In summary, our work presents generation attribution framework that allows researchers clearly trace
generations to memorization of patterns in pretraining data. Our findings suggest that language models are unable to reliably and evenly recall knowledge about global cul- tures in downstream generations, and resort to repeating small set of high-frequency symbols. Results from our work can be helpful for mitigating biases in cultural generations when combined with unlearning or data augmentation frameworks. We hope that we inspire more works attribut- ing model performance to pretraining data.
While our analysis is applicable to any model, our analyses are constrained by open-sourced models with searchable pretraining data. Ablation on 01Mg- 7B-0424 shows consistent conclusions and can be found in Appendix F2
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORKS
Memorization and Generalization. The knowledge and capabilities of L.MSs stem from lever-
aging large-scale pretraining corpora through both memorization and generalization. One line of work focuses on prompting LLMs to emit memorized training data (Wang et al., 2024
Carlini et al,
2023
Nasr et al. 2023 Zhang et al., 2023 Schwarzschild et al. 2024). Carlini et al. (2023) shows
that memorization increases with model size, example duplication, and prompt length. Another line
examines attributing memorization to internal features and its impact on generalization (Feldman,
2020 Feldman & Zhang. 2020 Zheng & Jiang, 2022 Zhang et al. 2023), with Zheng & Jiang
(0022) highlighting the importance of long-tail instances for generalization. Recent works extend
memorization to knowledge units like n-grams (Cao et al. 2020 Kandpal et al. 2023 Mallen et al.,
2022). and Antoniades et al. (2024) distinguishes memorization from generalization based on n--
gram similarity. Additionally. research explores how knowledge memorization affects generation
quality, with ZZhang et al.] (2024) and Chang et al.) (2024) finding that pretraining data imbalances
and long-tail knowledge intervals hinder learning and generation.
Culture bias in culture-conditioned generation tasks. Recent work on probing and evaluating dichotomy
cultural bias in L.Mss spans multiple areas. One approach compares the in model generations related culinary habits (Palta & Rudinger, 2023), etiquette (Dwivedi et al,
).23), commonsense knowledge Nguyen al. (2023). and other facts Keleg & Magdy (2023); Naous et al. (2023); Khandelwal et al. (2023); Li et al. (2024b). Another evaluates LLMs' cu cultural understanding using socio-cultural surveys originally designed for humans, such the World Values
Survey and Pew Global Attitudes Survey (Ramezani & Xu 2023 Tao et al., [003) Durmus et al.]
2023]. Additionally, works propose using LLM generation to create new resources and benchmarks for cultural knowledge et al.
2023
Huang & Yang 2023 Fung et al. 2024).
ANALYSIS SETUP
3.1
TYPES OF SYMBOL-CULTURE ASSOCIATIONS CULTURE-CONDITIONED
GENERATIONS
Association
Food Examples
Clothing Examples
Type
Diffuse
Chicken, Rice, Meat
Jeans, Shirt, Sweater Cheongsam, Yukata, Keffiyeh Long Top, Gown, Blue Shirt
Memorized
Miso Soup, Kalamari, Pho Chicken with Rice, Noodle Soup
Weak
Table 1: Examples of symbols falling into different types of associations for Food and Clothing
A symbol is an entity mentioned in culurre-cndiriooee generations. For example, in culture- conditioned generation about food, "My neighbor is Japanese. For dinner, my neighbor probably likes eating Miso Soup and Gyoza." " "Miso Soup" and "Gyoza" are two symbols generated for the Japanese culture. After an initial inspection of the generations, we discover four prevalent catego- rizes of associations between the conditioned culture and generated symbols:
Memorized Association is when the symbol and culture co-occur in meaningful context with high frequency in pretraining corpora (i.e. the co-occurrigg pretraining documents are relevant to both the symbol and the culture and their count is distinguishable from other cultures) and their association is learned by the model naturally. Memorized associations are important and highly desirable because they are grounded in pretraining documents, demonstrating sufficient model memorization of the symbol-culture association during pretraining.
Diffuse Association, contrast to memorized association, happens when a symbol generated for wide group of cultures without being associated with any of them through pretraining document grounding. While these symbols are not necessarily wrong (e.g. "meat" may be a food for any culture). they not informative, not distinctive, not interesting, and therefore not desirable. This phenomenon suggests that the model has drawn unintended associations of these symbols with many cultures, and prioritizes these symbols over more culture-specifie memorized associations.
3

---

Published as a conference paper at ICLR 2025
Cross-culture Generalization is when the symbol that has memorized association with culture A instead of culture B, but is generated for culture B. Here, we say the symbol is cross- gener- alization for culture B. Cross-culture generalization reveals that due to certain correlations between culture and the model has generalized memorized symbols for culture A to culture B. While this phenomenon may suggest promising generalization capabilities of models, such generalization suppresses generation of memorized symbols of culture B that are more relevant to the instruction.
Weak Association Generalization happens for symbols that are neither identified association with any culture due to insufficient evidence in the pretraining data to confirm strong memorization for them, nor identified as diffuse association symbol that broadly generated for the majority of cultures. However, they may be inferred, or generalized, from other symbols who have memorized or diffuse associations. For example, "kimono" type of "robe" specific to Japan, even though "robe" is not memorized association with Japan. This type of generalization is desirable because the symbol and culture show a weak association evidenced from pretraining data, yet the model is still able to learn such association.
memorized
Table shows examples of each type of symbol-culture association for both food and clothing generations.
3.2 DATA COLLECTION PROCESS
Model and Data. We conduct all of our analysis on CLMo-7B (Groeneveld et al. 2024) and
its pretraining corpora Dolma (Soldaini et al., 2024), OLMo-7B is the most capable generative large language model with open-sourced and indexed pretraining data. The same analysis could be extended to other models in future works, as long as their pretraining data is accessible.
Scope. Following the prompts and settings of (Li et al. 2024b), we collect generations for each of 110 cultures (Table introduced in (Li et al. 2024b) due to the high variation of symbols observed in their generations.
food and clothing topies. We choose food and clothing among all topics
Generation. We prompt the model in a continuing generation task where we use the following topic-wise prompts:
Food: My neighbor [culture]. At dinner, [he/she/my neighbor] probably likes to eat
Clothing: My neighbor is [culture]. [he/she/my neighbor] is probably wearing
We use the default model implementations from huggingface, setting temperature=1.0, top p=0.95, top k=50, max tokens=30 and num return sequences=/00, and period ('.) the stopping criteria. Ablations on pyper-parameters is in Appendix
We sample 100 generations for male, female, and gender-agnostic settings, and thus, for each cul- ture, we get 300 generations. Language models usually complete this prompt with one or more symbols. We take each completion and use LLAMA-3- 1--3--00-issttt ruct extract the symbols from the generation. The prompt for extracting symbols can be found in Li et al. (0024b).
3.3 IDENTIFYING KNOWLEDGE MEMORIZATION FROM CULTURE-CONDITIONED GENERATIONS
In this section, we demonstrate our MEMOED pipeline for classifying memorized associations. We first introduce how MEMOED determines whether one document contributes to culture-symbol memorization, and describe how we determine memorization from all contributory documents.
First, we determine a document contributes to culture-symbol association. Given a training document D, a culture C' (represented by both country and nationality, e.g. China and Chinese) and a symbol that is generated for culture C, the document is contributory to the memorization of association between C and 5 if tokens representing C and S appear within sufficiently low distance in D and D has high context relevance to C.
Suffiiennlly low distance is important because the tokens representing C and S must appear in the same piece of text during pretraining. Therefore, we introduce two metrics, minimum to- ken distance drok(C, D) and minimum sentence distance dsEwr(C,, D). For each docu-
4

---

Published as a conference paper at ICLR 2025
nasi lemak vegetable salad anchovies cucumber sushi
Mem.
For dinner, my Malausian neighbor probably likes eating
Symbols
Not Mem. Not Mem.
Not Mem.
OLMo-7B
Not Mem.
Mallayiaa
Classify
MEMOed
Classify Relevant Pretralning Dacuments
Determine Cuiture Symbol Association Strength
Signal to to-noooo ratio
Contribation Scores % releeeat docoments
7: score
-distance
thresholding
Figure 2: MEMOED pipeline, demonstrated with Malaysian culture food topic.
Sout atron
Biryani
Roast turkey
Ubyae
Kyter
13741191
94 sotice
ldec
DUD
Kerwi
Tembh
Penie
0.01
3,86
19h
1.47
Contribution Soore LLgg Scales
Figure 3: Higher contribution means stronger evidence of culture/symbol association in pre- training data, as defined in $3.3 Figure compares distribution of contribution score of memorized symbol (Biryani) v.s. non-memorized symbol (Roast turkey). Y-axis shows all cultures for which the symbol is generated. Red font show the z-score: 2 2.6 means memorization.
ment D, (C, is calculated the minimum number of subtokens, determined by the model's tokenizer, between all occurrences of C and S n-grams in the document D (Appendix D) dsEvr(C, (C, S, D) is calculated as the minimum number of sentences separating the and S n-grams, by splitting the document D along delimiters like full-stops.
High context relevance important because documents that strongly contribute culture-symbol memorized association should be topically relevant to the culture and symbol, manifested by high density of the culture and symbol n-grams compared other cultures' n-grams. Therefore, we propose Document-Signal to Noise Ratio dsve (C, S), the log ratio of the frequency of culture C' to the sum frequency all other cultures appearing
same document. With representing each n-gram that refers culture, we define dswa( (C, S, D) as:
( EIED I ELED 114c) (Er +c)
dssR SC = =logg
(1)
Documents that strongly contribute culture-symbol memorization should have high dsva, as the documents must have higher signals (target culture) than noise (other cultures).
There are two scenarios where classify document D as relevant and contributory the memo- rization association between culture C' and symbol
1. Global Relevance: drox(C, C, S, D) & max seq len & dsve(C, S, D) 0.
5

---

Published as a conference paper at ICLR 2025
Given that dswe uses logarithmic function to calculate the frequency strength of the target culture in the pretraining document, scores greater than 0 signify a ratio 2 1, indicating that the target culture is at least frequent as all other cultures combined. Furthermore, we upper-bound dyox (C, S, D) to ensure thar both C and S will appear in the same context window during pretraining. For OLMo-7B, maz seaassss_eeeee Jen 2048.
2. Local Relevance: S, s 2 & d,,e(C, D) (-1,
Empirical observations indicate that documents with dswx (C. D) scores between - and 0 often contain highly relevant excerpts that contribute significantly to the culture-symool association, albeit not extending to the entire document. For these cases, we apply the (CENT(C, S. D) metric with strict threshold of 2 to avoid over-counting. Relevant excerpts from various pretraining documents are detailed in the Appendix G
Second, we determine if a symbol is a memorized symbol of a culture. For a given symbol S and any culture C' CG (where Cg denotes the set of cultures that generated the symbol S), retrieve a complete set of documents D. D, C 'D represents the subset of documents classified as contributory the culture-symbol memorization using the criterion described above. Utilizing this subset, we calculate the following metrics to determine if S' is memorized symbol for culture C:
Contribution Score. Contribution Score (Cs) is the ratio of the number of contributory docu- ments, denoted n<(,), to the total number of documents in which the symbol S appears. Formally, m(D,) This measure tells us for all documents where the symbol occurs, proportionally how many exhibit strong association with given culture, helping us determine the symbol is memorized for the culture.
Cs n())
Determining memorization with z-score. We compute contribution score for every culture C' in Cg. and we normalize these scores to form categorical distribution (See examples in Figure We then compute the z-score of contribution scores for each culture within this distribution. Intuitively, the distribution is flat, then the symbol is not distinguishably associated with any the cultures. However, if the distribution spikes at a few cultures, then these cultures the rest for their association with the symbol. Therefore, we set the threshold of z-score to 2.6 (> 99.5% of C to find "outliers" in the distribution and classify the symbols as memorized for cultures whose z-score is above the threshold.
distinguishable from
In scenarios where symbol S is generated across less than five cultures, ie.,, n(Ca) 5, z-score known to be unstable for distributions with small sample size. Therefore, we select the highest scoring culture among Cg long as it's contribution score is above a lower bound of N. where N represents the total number of cultures in our set, i.e. 110.
3.4 ANALYSIS ON NON-MEMORIZED ASSOCIATIONS FROM CULTURE- COLDURR-CONDDTIOOED GENERATIONS
For each of the three non-memorized associations, conduct the following analyses to understand why such associations are formed during pretraining.
Diffuse Association. We hypothesize that diffuse association occurs when a certain symbol has substantially higher frequency in the pretraining corpora compared to other symbols, causing the model to prioritize generating this symbol for many cultures.
We identify symbols that are generated for at least half of total cultures, iee. 55 cultures, as being generated out of diffuse association. We count the occurrence frequency of all symbols of diffuse association and all symbols of memorized association in pretraining data using Infinigram API (Liu ct al. 2024). In order to verify whether large frequency gap causes the model to prioritize diffuse association to memorized association, for each symbol of diffuse association, we calculate its over- count((S.) where co.nt(S,) is the count of diffuse association shadowing ratio as r Σ, count(55m.)
symbol, count(Sm,) is the count of the j-th unique memorized symbol and N the number of unique memorized symbols.
edu/ fac.l httpss//wwwssss....aaaaaaa/////s

---

Published as a conference paper at ICLR 2025
Generalization. It occurs when the model generates symbols memorized from one culture for different culture. We hypothesize that co-occurrence both cultures may cause memorized associations to generalize. Therefore, case study, perform topic modeling subset of symbols and cultures. We extract all documents containing both cultures and the generated symbol, and use LDA (Blei al. 2003) and LLAMA-3 1-88--111 Inst ruct (Dubey et al. 2024 to extract common topic words in the documents in which the cultures co-occur (Appendix
Topic
Memorized Association
Weak Association Generalization
Culture
Biryani
Vegetable and Rice
Indian
Food
Ayam Goreng
Grilled Chicken
Indonesian
Salwar
Long Top Gown
Indian
Clothing
Ao Dai
Vietnamese
Table 2: Examples of Weak Associations generalizing from Memorized Associations
Weak Association Generalization. In order to identify from which symbols these weak asso- ciation symbols are generalized from, we resort to language model's own knowledge: if model memorizes symbol, it should be able to recite the definition of the symbol, using phrases repre- senting a broader concept of the memorized entities. For example, a model memorizes "kimono," then it is able to define "kimono"
type of "wrapped- front robe".
We prompt OLMo-Instzuct-72 to generate definitions of memorized symbols in a continued generation task (Appendix C.T Then, map symbols who are previously categorized neither memorized nor diffuse association symbols to these definitions using Fl score. For each symbol mapped with any weak association symbol, the latter is determined generalize from the former. Please note that such generalization can be cross-cultural in nature: weak association symbol gen- erated for one culture can as well be traced memorized symbols of completely different culture. Some examples of generalizations that are traced to memorized associations are given in Table
To identify symbols that can be traced to symbols with diffuse association, we look for generations with symbols that partially contain or are a combination of diffuse association symbols. such as "black 1-shir(" or "rice with meat."
RESULTS
4.1 MEMORIZATION LIMITED FOR UNDER-REPRESENEE CULTURES
-00
119
(a) Topic: Food
(b) Topic: Clothing
Figure 4: Geographical Distribution of Memorized Association
observe a medium-to high correlation between I) the number memorized symbols for a culture and 2) the count of documents in which the culture appears in the pretraining corpora. For food, obtained a Spearman correlation of 0.670 and Kendall 5 correlation 0.507. For clothing, we obtained Spearman correlation of 0.540 and Kendall correlation of 0.421.
(ouure/showw the geographical distribution of have least memorized symbol and average culture has about memorized symbols.
memorized symbols. For food, 97 cultures out 110

---

Published as a conference paper at ICLR 2025
In clothing, however, only 45 cultures out of 110 have least memorized symbol, Lee. around 60% have no memorized symbols, and on average one culture has about only 2 memorized symbols.
The limited memorization for under-represented cultures roots in the inadequate representation in the pretraining corpora. According to [Chang et al. (024), LLMs go through periodic forgetting of factual knowledge during pretraining and memorization requires the knowledge to appear within intervals shorter than the forgetting interval. Therefore, symbols of under-represented cultures are less likely to get memorized and be generated within the top-k outputs. Instead, symbols not be- longing to the culture (evidenced by how MEMOED finds insufficient contributory documents) are generated, leading to diffuse association or cross-culture generalization (see Section 4.3 and 4.4).
4.2 MEMORIZED ASSOCIATION Do NOT LIMIT TO CULTURALLY-EMBLEMATTE SYMBOLS
To dig deeper into the composition of memorized association, we recruit natives from each respec- tive culture and ask them whether each symbol "originates from" or "is emblematic to" their culture.
We annotate symbols of 8 cultures: American, Chinese, Filipino. Indian, Ghanaian, Japanese, Mex- ican, Vietnamese. These are the only cultures having more than 25 active annotators who were born in the culture but are currently in the United States. In total, we have recruited 257 annotators. Each annotator is tasked with evaluating 11 questions, including one attention check question that was designed as simple verification question to ensure the reliability of the responses. An annotator may annotate many times on different questions, and each symbol is annotated by 3 annotators. See E for annotation instructions.
Appendix
Overall, MEMOED's classification of memorized symbols agrees with human classification of em- blematic symbols, with a weighted Fl score of 0.845 on clothing and 0.670 on food.
However, not all memorized symbols are em- blematic symbols to a culture. The rest of the symbols consist of entities that are still used in the culture a lot without being an emblem- atic symbol: for example, "western style bridal gown" is recognized as a memorized symbol for Indian clothing, while "business suit" rec- ognized as memorized symbol for Japanese clothing. MEMOED is able capture such as- sociations from pretraining data that would oth- erwise be neglected by human annotators.
Figure 5: Overshadowing ratio r of all diffuse as- sociation for topic clothing.
4.3 DIFFUSE
ASSOCIATION IS FREQUENCY-DRIVEN
We find a moderate-to-high positive correlation for both clothing (Spearman م = 0.551, Kendall 7 = 0.385) and food (Spearman 0.519, Kendall 0.385) on overshadowing ratio r (defined in Section (3.4) and the number of cultures that the diffuse association symbol is generated for. This indicates that the pretraining frequency of diffuse association symbols is magnitudes higher than the frequency of memorized symbols, and this increases the chance of diffuse association overshadowing memorized association during sampling. Figure 5 shows that almost all symbols with diffuse association appear at least 1000 times more frequently than symbols with memorized association in the pre- pre-training data, for the topic clothing.
4.4 PROSS-CULTURE GENERALIZATION FROM HIGH TO Low FREQUENCY CULTURE
Frequency Analysis. We first observe a strong positive correlation between 1) the culture's 's number of tepic-related pretraining documents and 2) the frequency of the culture's 's memorized symbol being generated for some other cultures (clothing: Spearman p = 0.763, Kendall - = 0.574; food: Spear- man p 0.716, Kendall 0.531). Simultancously, the culture's 's number of topic-related pretrain- ing documents is also negatively correlated with the percentage of the culture's 's response containing another culture's memorized symbols (food only: Spearman p -0.521, Kendall -0.364).
8

---

Published as a conference paper at ICLR 2025
These results suggest that cultures whose generations contain other cultures' symbols tend to oceur fess-freyuently in pretraining documents, and cultures whose symbols tend to occur in other cultures' generations also those more commonly appearing in pre-training documents. For additional results, see Appendix H.I
Topic Modeling Analysis. In Section 3.4
stated our hypothesis that model may gener- alize the memorized symbols of one culture to another culture due to the two cultures' co-occurrence in pretraining documents under certain common topics. Although compre- hensive study on each memorized symbol is computationally impossible, we exemplify our analysis with examples of "hijab", "kimono", "biryani" and "churrasco" (See Appendix B for execution details).
...... new Saudi monarch, King Salman, grantad women the right drive by royal decrse, measure that took effect the following June, Ever since then, the kingdom has continued its top-down social reforms, process drivon by Crown Prince Mchammed bir Salman Woman now pormitted occasionally Meanwhile, the Iranion gevernment continues to forbid women from singing in public, attending sporting matches stadiums, riding bicycles. Recently, Iranian Supreme Leader Ayatollat: Ali Khamenei even stated that female characters with bijah
concerts.
appear
Each row in Table shows symbol, the culture for which it Is a memorized symbol, and the other culture for which is generated the second-most frequently. Table [ shows the rest of the cultures for which the symbols are generated and their topic modeling results.
Figure 6: Excerpt from relevant document for "hijab", "Iran" and "Saudi Arabia".
Figure shows excerpt of document which "hijab", Iran and Saudia Arabia co-occur.
Symbol
Mem.
Non- Mem.
Topic Modeling Keywords
Culture
Culture
Hijab
Iran
Saudi
wwmaan, islamic, muslim, women, rights, hijab, goverament, politics, people] [culture, fashion, asian, art, traditional, clothing, woman, tokyo, wedding, food]
Arabia
Kimono
Japan
South
Korea
Biryani
India
Pakistan
foodd recipe, restaurant, cooking, recipes, biryani, chicken, dish, dishes, cuisine]
Churrasco
Brazil
Chile
[food, restaurant, experience, wine, meat, rio, dining, fogo, bas, city]
Table 3: Keywords extracted from pretraining documents in cases of cross-culture generalization
4.5 GENERALIZATION FROM WEAK ASSOCIATION IS NOT CORRELATED WITH MEMORIZATION
On average, 3.1% and 5.0% of generations are generalized symbols for clothing and food, respectively. Interestingly, higher number of memorized symbol does lead higher number of generalization stemming from them. We only see a weak-to-none correlation (Spearman correlation of 0.17 and -0.03 for clothing and food) between the two types of symbols. Table TO shows the top and bottom 5 cultures for memorized symbols and generalized symbols for the topic food. Mexico. India, Japan, Morocco and Nigeria have the highest number of memorized symbols for food. However, Morocco appears among the top 5 cultures in generalized symbols while Japan appears in the bottom 5. Additionally, cultures without any memorized symbols rank higher in number of generalized symbols (e.g. Yemenis for clothing and Tribagonian for food). Cultures such as these where the model wasn't able to memorize anything, prompts the need for generalizations in the next token sampling process.
For symbols that partially contain or are a combination of symbols from diffuse association, we find that they generalizations which can be traced to high-frequency symbols resulting from diffuse association. These comprise about 0. 1 and 0.2% of generations on average for food and clothing respectively but almost 1/3 of the unique symbols for clothing.

---

Published as a conference paper at ICLR 2025
4.6
THE DISTRIBUTION OF FOUR TYPES OF CULTURE-SYMBOR ASSOCIATIONS
Crosen Canure Geo 18.4
42.7%
Siffuss Ansacietnn 68.9%
Association 12.5%
Difuse Acsociation
93.1%
Untracsable 5.0%
Can
Cross Cuiture Gen 0.530%
(a) Mexico
(b) Trinidad
Figure 7: While some cultures contain no memorized association in their generations (Fig 7b), cul- tures like Mexico's s almost 1/2 generations comprise of memorized association (Fig 7a
In our analysis, we extract 2370 unique symbols for food and 1002 for clothing. Of these, 4.1% (98 symbols) and 10.9% (110 symbols) appear in over 50% of cultures, categorized as diffuse as- sociation, for food and clothing respectively. For food, 46. 12% (1098 symbols) are identified as memorized association, and 31.3% (713 symbols) as weak association symbols generalized from memorized symbols. contrast, for clothing. 25.78% (258 symbols) are memorized association, and 31.6% (317 symbols) are weak association generalized from memorized symbols. Additionally, a smaller fraction of food symbols (7.6%, or 180 symbols) and a significant portion of clothing sym- weak association generalized from high-frequency
bols (nearly one-third, or 332 symbols) symbols of diffuse association. The remaining small proportion of symbols include hallucina- tions, typos, and brand names, not fitting into these categories.
While diffuse association only comprise of a small proportion of the total unique symbols extracted from responses, they comprise significant proportion (91.12%) for clothing and 79.2% for food) of the total responses, indicating that they are sampled multiple times during the generation process. Additionally, memorization is especially scarce in generated responses, averaging only 0.76% for clothing and 4.12% for food while traceable generalization averages to 3.1% % and 4.9% for both top- ies respectively. However, as seen in Figure 7 the extent of memorization in responses has very high variance (from 0% for Trinidad to almost 42.2% for Mexico in food). Cross-culture general- ization, while only averaging 4% and 11% respectively for clothing and food, exhibits high variance with cultures with high number of memorized symbols having lesser cases of generating symbols memorized for other cultures. also visible in cases when certain cultures show common themes related to the topic in their pre-training document More analysis can be found in Appendix H.2
CONCLUSION
In conclusion, our work introduces MEMOED, a framework for attributing culture-comditioned generations of language models memorization from pretraining data. By analyzing the appear- ance of symbols in model outputs across 110 cultures, we uncover a clear imbalance in how many symbols language models memorize for high- frequeeyy and low-frequency cultures. In addition, models tend to prioritize generating high-frequency symbols that are not specific to any culture over memorized symbols, while also struggling generalize from memorized cultural symbols with lower prevalence in the pretraining data. This highlights significant limitations in current pretraining processes, where models prefer frequently occurring. diffusely associated symbols at the expense of diverse, culture-specifie knowledge. Our findings underscore the need for improved pretraining data and methods, and we hope this research sparks further work on linking model behavior to data-driven insights.
As shown through keywords Table
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research supported in part by the Office of the Director of National Intelligence (ODNI), In- telligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022- 22072200006, the Defense Advanced Research Projects Agency with award HR001 12220046, and NSF IIS 2048211. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. We would like to thank all the collaborators in USC INK research lab for their constructive feedback on the work.

---

Published as a conference paper at ICLR 2025
LIMITATIONS
MEMOED uses each individual document the unit of memorization, while it is possible that one document may contain multiple excerpts of culture/symbol co-occurrence within minimum token threshold. However, we cannot exactly reproduce the contexts of the pretraining process as the training batches are randomly ordered OLMo-7B training.
Our study is only conducted OLMo-7B due to the fact that it is the model with highest language capability that also has open pretraining data. How our conclusions may hold for non-OLMo family models is unknown; however, methodology introduced in $ is transferrable for analyzing any model, as long as their pretraining data accessible.
REPRODUCIBILITY STATEMENT
Algorithm. We provide accurate description of our analysis framework in Section 3 and addi- tional details in the appendix.
Prompt Engineering, The prompts we used for generating culture- colture-conditione generations, prompting for traceable generalization definition and topic modeling are included in the appendix.
Data and Source Code.
Data and source code will be released upon acceptance.
Crowdsourcing.
Instructions for Prolific annotators are available in Appendix
ETHICS STATEMENT
Data. All data we collected through L.Mss in our work are released publicly for usage and have been duly scrutinized by the authors. Data for all human studies that we conduct are also publicly released with this work, with appropriate annotator anonymizations.
Crowdsoureing. All our crowdworkers currently residing in the United States, with countries of birth from US, China, India, the Philipines, Ghana, Mexico and Vietnam. For all our human studies, the task is set up in manner that ensure that the annotators receive compensation that is accepted by the platform ($ 1//hoor). Furthermore, we ensure that we correspond with crowdworkers over direct message to address their queries.
Potential Use.
Our framework MEMOED may only used for analysis that follow the ethics guideline of the community. Using MEMOED on mal-intentioned searching for proprictary data is a potential threat, but the authors strongly condemn doing so.
REFERENCES
Antonis Antoniades, Xinyi Wang. Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. Generalization vs memorization: Tracing language models' capabilitics back to pretraining data. arXiv preprint wrXiv:2407.14988 2024.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling, In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
David M Blei, Andrew Y Ng, and Michael Jordan. Latent dirichlet allocation. Journal of machine Learning research, (00n)):933-102) 2003.
Ermei Cao, Difeng Wang, Jiacheng Huang, and Wei Hu. Open knowledge enrichment for long-tail entities, 2020.
12

---

Published as a conference paper at ICLR 2025
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023.
Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Sco, Du-Seong Chang, and Minjoon Seo. How do large language models acquire factual knowledge during pretraining? arXiv preprint arXiv:2406. 1113, 2024.
A Conneau. Unsupervised cross-lingual representation learning at scale. arXiv preprint 1rXiv:1911.02116, 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint urXiv:2407.2 2024.
Esin Durmus, Karina Nyugen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodde Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCan- dlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. Towards measuring the representation of subjective global opinions in language models. ArXiv, abs/2306. 16388, 2023.
Ashutosh Dwivedi, Pradhyumna Lavania, and Ashutosh Modi. Eticor: Corpus for analyzing Ilms for etiquettes. In Conference on Empirical Methods in Natural Language Processing, 2023.
Yanai Elazar, Akshita Bhagia, Ian Helgi Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, al. What's in my big data? In The Twelfth International Conference on Learning Representations, 2024.
Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954-959, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881- 2891, 2020.
Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. Massively multi-cultural knowledge acquisition & Im benchmarking. arXiv preprint arXiv:2402.09369 2024.
Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanji,, Sungchul Kim, Franck Dernon- court, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics, pp. 1-79, 2024.
Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Au- thur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik. Crys- tal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaíni, Noah Smith, and Hannaneh Hajishirzi. OLMo: Accelerating the science of language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meet- ing the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15789-15888 Bangkok. Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/ aol-.acl--ong.841 841, htt.tttps:////// /acl htatttpps//aalllathlllgggg./
org/ 2024 1..44acl-loo 841
Christian W Haerpfer and Kseniya Kizilova. The world values survey. The Wiley-Blackwell Ency- clopedia of Globalization, pp. 1-5, 2012.
Jing Huang and Diyi Yang. Culturally aware natural language inference. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2023, pp. 7591-7609, 2023.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pp. 16696-15707 PMLR, 2023.
13

---

Published as a conference paper at ICLR 2025
Amr Keleg and Walid Magdy. Dlama: A framework for curating culturally diverse facts for probing the knowledge of pretrained language models. arXiv preprint arXiv:2306.05071 2023.
Khyati Khandelwal, Manuel Tonneau, Andrew M. Bean, Hannah Rose Kirk, and Scott A. Hale. Casteist but not racist? quantifying disparities in large language model bias between india and the west. ArXiv, abs/2309.08573 2023.
Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. Culturellm: Incorpo- rating cultural differences into large language models. arXiv preprint arXiv:2402. 10946, 2024a.
Huihan Li, Liwei Jiang, Nouha Dziri, Xiang Ren, and Yejin Choi. Culture-gen: Revealing global cultural perception in language models through natural language prompting. arXiv preprint (tt::2404.10/999 2024b.
Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini- gram: Scaling unbounded n-gram language models trillion tokens. arXiv preprint arXiv:2401.17377, 2024.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511, 2022.
Rohin Manvi, Samar Khanna, Marshall Burke, David B Lobell, and Stefano Ermon. Large language models are geographically biased. In Forty-first International Conference on Machine Learning, 2024.
Tarek Naous, Michael Joseph Ryan, and Wei Xu. Having beer after prayer? measuring cul- tural bias large language models. ArXiv, abs/2305. 14456, 2023. URL https://api. semant i cscholaall www//orrgssaaoooooooooooo ID: 258865272
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, Feder Cooper, Daphne Ip- polito, Christopher A Choquette-Chooo Eric Wallace, Florian Tramèr, and Katherine Lee. Scal- able extraction of training data from (production) language models. CoRR, 2023.
Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. Extracting cultural commonsense knowledge at scale. In Proceedings of the ACM Web Conference 2023, WWW '23. ACM, April 2023. doi: 10. 1411445/554350735533555
URL http: //ttp://dx... dei wew. org
1145/
3543507 3583535
Shramay Palta and Rachel Rudinger. Fork: A bite-sized test set for probing culinary cultural biases in commonsense reasoning models. In Findings of the Association for Computational Linguistics:
ACL 2023, pp. 9952-99929 29229962,2222 2023.
Aida Ramezani and Yang Xu. Knowledge of cultural moral norms in large language models. arXiv preprint arXiv:2306.01857 2023.
Michael J Ryan, William Held, and Diyi Yang. Unintended impacts of Ilm alignment on global representation. arXiv preprint urXiv:2402. 250i::2002/50088 2024.
Avi Schwarzschild, Zhili Feng. Pratyush Maini. Zachary C Lipton, and J Zico Kolter. Rethinking I1m memorization through the lens of adversarial compression. arXiv preprint 45X1::2404.15146, 2024.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining re- search. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 15725-15788 Bangkok, Thailand, August 2024. Association for Computational Linguis- 10.8665/1//224..uu
tics. doi: 18.18653/v 1/2024.acl loo...cl--.
URL https: / /a//aaaaaaa wwc.annnhhooog org/2024.
ac lon..ng.... 840
14

---

Published as a conference paper at ICLR 2025
Yan Tao, Olga Viberg, Ryan S. Baker, and Rene F. Kizileec. Auditing and mitigating cultural bias in Ilms, 2023.
Zhepeng Wang, Runxue Bao, Yawen Wu, Jackson Taylor, Cao Xiao, Feng Zheng, Weiwen Jiang, Shangqian Gao, and Yanfu Zhang. Unlocking memorization in large language models with dy- namic soft prompting. arXiv preprint arXiv:2409. 13853, 2024.
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. Advances in Neural Information Processing Systems, 36:39321-333622 2023.
Yuji Zhang, Sha Li, Jiateng Liu, Pengfei Yu, Yi R Fung, Jing Li, Manling Li, and Heng Ji. Knowl- edge overshadowing causes amalgamated hallucination in large language models. arXiv preprint arXiv:2407.08039, 2024.
Xiaosen Zheng and Jing Jiang. An empirical study memorization in nip. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 6265-6278, 2022.
Caleb Ziems, Jane Dwivedi- Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. Normbank: A knowl- edge bank of situational social norms. arXiv preprint arXiv:2305. /7008, 2023.
A 110 CULTURES
Geographic Region
Countries and Regions
Eastern- Eurtopen
Albania, Armenia, Belarus, Bosnia and Herzegovina, Bulgaria, Croatia, Czechia, Georgia, Greece, Hungary. Kosovo, Moldova, Montenegro, North Macedonia, Poland, Romania, Russia, Serbia, Slovakia, Slovenia, Turkey, Ukraine Algeria, Egypt, Ethiopia, Ghana, Kenya, Libya, Morocco, Nigeria, Rwanda, Tunisia, Zambia, Zimbabwe Andorra, Austria, Belgium, Finland, France, Germany, Ireland, Italy, Luxembourg, Netherlands, Portugal. Spain, Switzerland, United Kingdom
African-Islamic
Western-Euue -ureer-Eu
Latin-American
Argentina, Bolivia, Brazil, Chile, Colombia, Dominican Republic, Ecuador, El Salvador, Guatemala, Haiti, Mexico, Nicaragua, Peru, Puerto Rico, Uruguay, Venezuela Australia, Canada, New Zealand, Trinidad and Tobago, United States, South Africa
English Speaking
Central-Asian
Azerbaijan, Kazakhstan, Kyrgyzstan, Mongolia, Tajikistan,
Uzbekistan
South-Asian
Bangladesh, India, Maldives, Pakistan
Baltic
Estonia, Latvia, Lithuania Denmark, Finland, Iceland, Norway, Sweden China, Hong Kong, Japan, Macau, South Korea, Taiwan Indonesia, Malaysia, Myanmar, Philippines, Singapore, Thailand, Vietnam
Nordic
East-Asian
Southeast-Asian
Middle-Eassern
Cyprus, Iran, Jordan, Lebanon, Palestine, Kuwait, Qatar, Saudi Arabia, Yemen
Table Countries and Regions for each geographic region, according to HHeerpfer Kizilova
2012)
15

---

Published as a conference paper at ICLR 2025
Instructions:
Be helpful and answer questions concisely. If you don't 'on't know the answer, say 'None'
Utilize only the tokens in the given sentence for generating phrases/vo for the given query.
Incorporate your preexisting kncwledge enhance the depth and relevance of your response.
Cite your sources when providing factual information.
had document on which 1 ran Latent Dirichlet Allocation and 1 got the following outputs:
<lda output>
From this LDA output, list almost possible topics which can be modeled from the document.
tople is phrase from words probabilities given above.
word denoting theme and you infer it
Do not provide any explanation description for the topics you generate, I you do not know more topics, stop and say DONE.
Figure Prompt for LLAMA- 3. 3-1-8B in Topic Modeling Pipeline
B TOPIC MODELING
B. METHODOLOGY
For any culture C' and its set of memorized symbols m(C), we select a symbol E m(C) and identify the set of cultures C. which also generated S but not through a memorization. For each culture C' E Ca and for C, we retrieve pre-training documents where the two cultures co-occur, forming set Dec We apply the metrics defined in Section 3.3 to filter these documents, obtaining subset Dęe' C Dee' that relevant the association of the cultures. We further refine Dee by removing documents that do not contain the symbol S, resulting in a final set Dee's, which is relevant to the association between cultures C and C" and contains the memorized symbol S.
Subsequently, we use a sliding window of size 2048 to create chunks from each document E Dee's We employ Latent Dirichlet Allocation (LDA) Blei al. 2003) to model five topics from each set of chunks corresponding to document. The modeled n-gram phrases with corresponding topic probabilities are then prompted to LLAMA 3 1- 8- Instruct (Dubey et al.) 2024) The LLM generates interpretable n-gram topic phrases, which are then filtered for repetitions using cosine similarity scores calculated with XIM-RoBERTa- -arrge embeddings (Conneau 2019). Finally, extract the top five keywords from these topics using TF-IDF.
B.2 PROMPT
In 8 8 we provide the prompt used for prompting I.LAM-3 11-8B-Instrrue with the LDA input and generating the outputs corresponding to interpretable topics which are inferred from the LDA and we use to generate keywords.
Table we extend our study pre-training documents (Table pertaining to cross-cultural gen- eralization from one culture to another for more cases of cultures which generate these memorized symbols with a lower count relevant documents than the cultures discussed before. We notice suprisingly similiar themes in the pre-training documents such as the discussion around "religion" documents where Hijab, Iran and any culture X co-occur. For Kimono and Japan, we notice similar common theme surrounding "fashion". We hypothesize that such common themes also
16

---

Published as a conference paper at ICLR 2025
Symbol
Mem. Culture
Non-Mem. Culture
Topic Modelling Keywords
Hijab
Iran
Iraq
[woman, government, islamic, war, politics, kurdish, people, conflict, protest, muslim]
Hijab
Iran
Pakistan
[woman, muslim, islamic, women, hijab, issues, government, rights, people, culture]
Hijab
Iran
Indonesia
[woman, islamic, muslim, hijab, fashion, law, women, islam, culture, government]
Hijab
Iran
Egypt
[woman, muslim, islamic, women, islam, arab, government, hijab, culture, politics]
Kimono
Japan
Italy
[fashion, art, tokyo, culture, design, food, hotel, experience, clothing, travel]
Kimono
Japan
Kenya
[travel, fashion, art, experience, culture, africa, african, design, food, names]
Kimono
Japan
Salvador
[arts, martial, blue, color, dell, laptop, ryu, asian, friends, indige] [food, restaurant, dish, recipe, dishes, cuisine, cooking, bengali, chicken, recipes]
Biryani
India
Bangladesh
Biryani
India
Singapore
[food, restaurant, dish, cuisinc, dishes, experience, biryani, chicken, cooking, options]
Biryani
India
Iran
[food, biryani, restaurant, cuisine, dish, recipe, saffron, hyderabad, spice, cooking]
Churrasco
Brazil
Peru
[food, restaurant, expericnce, cuisine, bar, dining, sushi, dish, londen, city] [food, restaurant, latin, bride, dating, beach, cheese, recipe, meat, travel]
Churrasco
Brazil
Colombia
Churrasco
Brazil
Uruguay
[food, meat, restaurant, cuisine, rio, american, bbq, exparience, south, dish]
Table 5: Keywords modeled from pretraining documents in cases of cross-culture generalization (continued)
Topic food clothing
Prompt Template
[nstrructio]. [symbol] probably [culture] food item which type [instruction]. [symbol] is probably [culture] clothing item which looks like type of
Table 6: Prompts for generating definitions of memorized symbols to detect Wcak-Association Gen- cralization
cause models to overmemorize and generate memorizations from one culture into another and not necessarily when the model is devoid of memorizations the next token space (which would be the case for cultures which have memorizations).
C PROMPTS
C.I ASSCIIATION GENERALIZATION PROMPT
[instruction] in Table rrfeers to the instruction (note that any typos in the prompt are intentional and included for the purpose of jailbreaking):
17

---

Published as a conference paper at ICLR 2025
You are helpful agent who is knewledgable about celtures from all places. You are talking to a persen who is curious about different cultures and their [topic]
Complete the sentence below with a few words or a phrase. If you do not know the answer, print don't know.
D COMPUTING MINIMUM DISTANCE B/W TWO N-GRAMS
This section elaborates on the algorithm employed by us for computing minimum distance be- tween two n-grams in pretraining document and reporting the drox(C,SS D) metric. It calcu- lates the context length difference between the n-grams C' and S, as observed by the LLM during pre-training. We hypothesize that for a pre-trained language model with sequence length L, smaller dCo S, D) indicates more frequent co-occurrence of two n-grams across training batches. This frequent co-occurrence is likely to strengthen their association, thereby increasing the relevance of a document to the relationship between and
The algorithm described in Algorithm D computes the minimum token distance between two n- grams within a text, using a tokenizer to process the input and mark relevant tokens. Initially, the text is tokenized to capture each token's positional offsets. The algorithm then marks tokens that correspond to the specified n-grams, word and symbool by iterating through the text to find these n-grams and marking overlapping tokens with distinct values for each n-gram.
Following the marking phase, the algorithm calculates the minimum distance by iterating through the marked tokens. It maintains pointers the last positions of tokens related to word and symbol. When a token corresponding to one of the n-grams is encountered, the algorithm checks if the last seen position of the opposite n-gram has been recorded and updates the minimum distance the current position is closer.
The procedure concludes by returning the minimum distance, which quantifies the proximity of the n-grams and reflects their associative strength in the context of language model pre-training.
E HUMAN ANNOTATION SETUP USING PROLIFIC
We designed human annotation task using Google Forms, automatically populated via Google Apps Script with symbols related to food and clothing from eight different cultures. Figure 9 pro- vides overview of the form setup, while Figure I0 shows an example of question where par- ticipants were asked to evaluate whether a specific food is a cultural food item of some culture. Annotators were required to select the most appropriate classification based their knowledge of the culture in question. This process enabled us to collect reliable data regarding culturally emblem- atic food and clothing items.
F ABLATION STUDY
F.1 ABLATION HYPERPARAMETERS
In the original design of decoding process, multinomial sampling was employed with set of specified hyperparameters: temperature=1.0, top p=0.5, top k=50, max tokens=30, and num num_return..ct www.retarn.ssqqeeeeee swwuumssequenes=//k The stopping criterion was established as the period (t.) character. To explore the impact of these parameters on the generation results, an ablation study was conducted where top k values of 20 and 80, and lemperature values of 0.75 and 1.25 were tested against the original settings. We observed an overlap coefficient of greater than 90% in all the four cases. This tells us that the sampling conditions did not cause change our findings.
F.2 ABLATION ON OLMo-78 VARIANTS
In order verify that conclusions find OLMc-7B hold on other modalities, reproduce some of the experiments on newer variant of OLMo-7B, 10MMo-7B-042-- We collect culture- conditioned generations for both food and clothing on 01.Mo-7B-0424 which is trained Do ma
18

---

Published as a conference paper at ICLR 2025
Algorithm Calculate minimum token distance between two n-grams
1: procedure MINToKENDISTANCIterr word, symbbol, tokenizer)
2: encoding k tokenizer(tezt return mapping-True)
3: tokens encoding.tokens.
4: token of sets +- encoding of offseee feetssssmm mappingl
5: marks [0] len(tokens)
b. Mark tokens corresponding to symbol and word
6:
for phrase {symbol, word} do for start in text do if text. f find(phrase, start) -1 then end &- start + len(phrase) for i, (s, e) enumerate token_o, ff do if None A & Nome A < end A e start then marks[i] +- max(marks([], if phrase symbbol then else 1)
7:
8:
9:
11:
12:
13:
start <- end min_distance 0C
15:
last_ssymil f- -1 last_word --1
16:
Compute minimum distance between marked tokens
17:
for from to len(neanks do
18:
if marks(i] then last_ systoll <- if last_word # 11 then min min_dintance + min(min i last _oord) else if marks(i] then
19:
20:
21:
22:
23:
last word +- if dast_symbol + -1 then mistaistance +
24:
25:
i last (ass_syybbl)
26:
return min_distonce
Ordering
Food
w/ Clothing
Morocco
Azerhaijan 97
Bangladesh 99
Bolivia 96
From Top 10 (t)
Iceland 99
Chile 91
Sweden 96
India 76
Ethiopia 90
Kenya 74
France 42
Germany 30
Singapore 42
United States 28
From Bottom 10 ({)
Britain 38
China 26
Indonesia
Portugal
Australia 35
France 21
Table 7: Cultures chosen for ablating on OLMo 7B- 0424 and their corresponding number of unique symbols
19

---

Published as a conference paper at ICLR 2025
Instructions
this task, we ask you classify 11 food items as whether is 'cultural food item" of the American culture. commonly recognized as originating from the
emblematic
some religion/ aijon/ethnic group/communniy the American
We ask that classify each item into the five options below, based personal experience and knowledge about the American culture:
1. know this food item and it is cultural food of American culture
typo not food cultural food
This food item and cultural food item of American culture
don't know this food Item
Additional Instructions
you symbol question, option "I
"IMPORTANT: Some questions the form included attention checks. you these, need to return the questionnaire, and won't eligible for reward.** **
Sample Annotations
sample
criteria option,**
(Assume you from Malaysia)
Q1: Is "Ayam Goreng' cultural food item of the Malaysian culture? (Note: Ayam Goreng is type of fried chicken commonly eaten in Indonesian and Malaysian cultures)
food Ayam Goreng emblematic only Malaysian
(Select Option
food Malaysian Explanation:
other cultures.
02: is "fried chicken" cultural food item of the Malaysian culture?
A2: (Select Option 2) know this food item and it not cultural food item of Malaysian culture. Explanation: Although there is a type of fried chicken emblematic Malaysian culture, fried chicken is eaten in many other cultures,
qualify
Q3: "Sushi" cultural food item of the Malaysian culture? (Note: Sushi type of Japanese dish with vegetables, meat/seafood and seaweed wrapped around rice)
AS: (Both Option and 5 are correct) know this food item and is not cultural food item the Malaysian culture I don't know this food item. Explanation: depending on your knowledge about specific food items, you can options.
Q4: 'rice" cultural food item the Malaysian culture?
A4: (Select Option 2) know this food item and it not cultural food item of the Malaysian culture. Explanation: Rice prevalent globally eaten by almost all cultures around the world, and therefore does not qualify as a cultural food Item.
Figure 9: Example of Google Form Used for Cultural Food Annotation
20

---

Published as a conference paper at ICLR 2025
Is "teriyaki chicken' cultural food item of the Japanese culture?
know this food item and cultural food item of Japanese culture
know this food item and not cultural food item of Japanese culture
This food item typo cultural food item of Japanese culture
This food item typo
cultural food item of Japanese culture
don't know this food item
Figure 10: Sample Question from Google Form on Cultural Food Classification
1.7. Although OLMc-7BB04244 the same model family as OLMo-7B, Dolma 1.7 contains pre- training documents that are not in Dolma 1.5, and 0LMc-7B-0424 trained with an updated al- gorithm from OLMo-7B. Other models supported by the WIMBD API, such as Pythia (Biderman jet al. 2023), are not particularly capable of instruction following culture-conditioned generations, and therefore, analyzing their generations is less informative.
We only reproduce two main correlations in the main paper:
The number of cultures a symbol with diffuse association is generated for and the number of pretraining documents it appears in (Section 4.3) For
we obtain moderate- to-strong correlation for both clothing (spearman / = 0.507, Kendall r 0.362) and food (spear- man = 0.416, Kendall r 0.313). Compared to OLMo-7B with clothing (spearman p 0.521, Kendall r 0.367) and food (spearman P = 0.358, Kendall r = 0.260), we see that even though the models and training data are different, the Spearman and Kendall correlations for food and cloth- ing remain the same (both moderate-to-strom, to-otno correlations). This means that the number of cultures symbol with diffuse association generated for and the number of pretraining documents appears in is positively correlated, regardless of the model.
The number of memorized symbols for a culture and the number of pretraining documents appears in (Section 411 For OLMo-7B-0424, cultures with the highest number of unique symbols generated by OLMo-7B-0424 and from the 10 cultures with the lowest number of unique symbols generated by
select 10 cultures out of 110, 5 from the 10
We obtain a moderate-to-stront correlation for both clothing (spearman = 0.591, Kendall 7 = 0.507) and food (spearman cultures) with clothing (spearman P = 0.540, Kendall - 0.421) and food (spearman = 0.670, Kendall 0.507), we see that even though OLMo -0-0-44 tested on smaller number of cultures, for both clothing and food, the correlation of OLMc-7B- 7B-042 4is more strongly positive. Therefore, the conclusion still holds that higher pretraining document counts of cultures increase the number of memorized symbols in multure-conditionn generations.
0.829, Kendall
0.659). Compared to OLMo- (on 110
F.3 ABLATION Z-SCORE FOR MEMOED
We study whether selecting a different z-score threshold would change the conclusions of MEM- OED on memorized symbols for all cultures. We perform an ablation study on setting the z-score to
2, which statistically means that the value is about 97.7 percentile. Empirically, z-score below does not indicate outliers, so we focus our ablation analysis only on cases where the z-score
correlation between 1) the number of memorized symbols for a culture and 2) the count of documents in which the culture appears in the pretraining corpora: for clothing, for food, we obtain a spearman correlation of 0.688 and a Kendall correlation of 0.519. This corre- lation is lower but similar to the original correlations found for 7=2.6 6.6 (food: Spearman=0.670 and
When 2, still get a moderate-to-stren
obtain spearman correlation of 0.569 and Kendall correlation of 0.445;
21

---

Published as a conference paper at ICLR 2025
... curry lanalics who are not Indian maybe shocked fnd that their beloved Chicken Tikka Masala really an Indian curry but on anglicized version of the nouth-watering Nutter Chicken that North Indian wedding food staple. This rich and creamy gravy based. dish accompanied with {naan food on the world food map. Vegelariaus can devour Panser
Chinese Recipes- Here are 16 of the most popalarr Chinese disbes.
Closeup japonese sushi
bamboo napkin. 2 2 0. Need help? #85531256 Ice tream, popiiicles. Collage
different piclures Kahsa mixed rice dishes that originates in Yemen. A traditional Asian
has put Indian)
ccoked food comprising fined moming glory. fried chicken, mixed curry and deep Food Buffet. How Make Hememade Download Homemada food stock photos. Assorted food and dishes of vegetables, meet and lish on black stone background Selectoon Indian 2ood iacluding curries, rice, samooses and naan bread .....
equally lasty not all infario: subs stute for the famous Butter Chicken. Another renowned Indian dish and Indian wodding food menu favourite the Biryani
(a) dsve: 6.599; dyox: 4
(h) dswa: -0.982; dSENY: 0
Figure 11: Examples of excerpts from relevant pretraining docs for Culture: "Indian" and Symbol: "Naan":
... things of drag of Year Celebrations, why not visit one of South London's mest unpretentious places good, solid Chinese cooking? We're talking about Mamalan, With sites in Clapham,
..... J...neceeese Egg Crépe 'Noodles' Makes: 2 side dish servings Tine: 10 mirutes, In China, there restaurants nearly every nelghborhood serving handmade noodlos, Just like Italion pasta ramer, when cooked property. they should have tirm bite and springy
nony tining roums serenity from the fremetic envirens of their neighbourhoods. The dumplings, but also ventures into more adventurous terrtory with plates wesabl prowe beo wonton noodle While there not be lion dance here, you are guaranteed a good time
the variation thickness and springiness makes Chinese egg noodles some of the most
at
From etispy par-fried nocdles to bowl of wonton noodle soup, fresh Chinese egg noodles staple ......
(a) dswa: 5.584; dros: 17
(b) dswx: -0.841 daewr: 0
Figure 12: Examples of excerpts from relevant pretraining does for Culture: "Chinese" and Symbol:
"Wonton Noodle Soup":
Kendall=0.507; clothing: Spearman=0.540 and Kendall=0.421), showing that our conclusion on the relationship between a culture's memorized symbols and the culture's frequency in pretraining data robust different Z-score threshold.
In addition, we examine how lowering the z-score from 2.6 to changes memorized symbols dis- covered for each culture. We compare each metric's agreement with human evaluation clothing: when z=2.6, the weighted Fl score is 0.845, and when z=2, the weighted Fl score is 0.840. We can that 2 has slightly lower agreement with human categorization, suggesting that additional symbols that are marked as memorized symbols when z=2 are non-emblematic symbols according to human culture experts.
G TRAINING DOCUMENT EXCERPTS
In this section, present excerpts from the pre-training documents classified contributory to culture-symbol association using MEMOED's dsNR, drox and dsENT metrics.
we present excerpts from two pretraining documents classified as contributory to the association between the culture: Indian and the symbol: Naan. We also report the relevant metric scores used to determine this. For Figure IIa since the dsve is greater than zero, the d7ok metric is used to ascertain the classification of this document. As visible the excerpt, the culture "Indian" appears numerous times and in close proximity to the symbol "naan". Additionally, upon seeing the remaining part of the excerpt, we see that is talking about Indian food items which indicates the relevancy of this document towards the association. On the other hand, for Figure TTb since
In Figure 11
22

---

Published as a conference paper at ICLR 2025
Culture
Memorized Symbols From
Pre-Training Count Rank (/110)
Trinbagonian
American (0.4%)
101
Macanese
American (0.5%)
100
Salvadoran
American (1%)
99
Zambian
American (0.6%)
94
Nicaraguan
American (0.4%)
85
Puertorriquñña
American (0.6%)
70
Egyptian
Iranian (2.9%)
27
Saudi
Iranian (6.2%)
45
Andorran
French (0.3%)
110
Hong Konger
French (0.6%)
38
Table 8: Cultures Identified from Ceave--One-Out-Comeeel
the dsve is between and we the dSENT metric explained in Section 3.3 similarly that although the ratio is less than zero, the document is not noisy and the local context is about Indian food item.
We observe
Similarly, in Figure 12, we present excerpts from two pretraining documents classified as contrib- utory the association between the culture: Chinese and the symbol: Wonton Noodle Soup. We can observe that the training document with positive dsvæ is not really talking about Chinese food items but rather talks about a prominent Chinese festival i.e. Chinese New Year and mentions the food delicacies being prepared then. Thus, through this it contributes to the association between the culture and symbol. On the other hand, for the document with negative dsvR, we observe rela- tively high concentration of cultural mentions in this excerpt and on global level, the topic being discussed is restaurants in China when the food cultural symbol is mentioned. Hence we see how this document potentially contributes to the culture-symi symbol association.
H ADDITIONAL RESULTS
H.1
CROSS-CULTURE GENERALIZATION
Topic
Keywords
food
food, foods, cuisine, cuisines, dish, dishes, meal, meals, recipe, recipes, menu, menus, breakfast, lunch, dinner, snack, snacks clothing, clothes, apparel, garment, garments, outfit, outfits, attire, attires, dress, dresses, suit, suits, uniform, uniforms
clothing
Table 9: Keyword list that we use to filter for topic-related pretraining documents.
To further evaluate cross-culture generalization across all 110 cultures, we obtain: (1) the percent- age of culture's responses that contain another culture's memorized symbols; (2) how often is culture's memorized symbol generated for some other culture. Additionally, we calculate the corre- lation between each culture's metrics (1) and (2) with the frequency of topic-relevant occurrences of that culture in the pretraining corpora.
For (1), we observe a moderate negative correlation for food (Spearman = -0.364) indicating that cultures containing more memorized symbols from other cultures tend to oceur less-frequently in food-related pretraining documents. We have shown this correlation using scatter plot in Figure [3] However, for clothing, we observe a weak negative correlation (spearman = -0.099, Kendall - -0.061). To investigate this, we conduct a leave-one-cultur-- out experiment. In this analysis, we recalculated the correlations while systematically excluding one culture at a time. We then identify and list the top ten cultures causing the highest variation. Notably, these cultures are either those that predominantly contain generalization from regional cultures, such as Egyptian or Saudi, or those that are less frequently mentioned in the pretraining data, such Trinbagonian. We have listed these ten cultures with the highest number of cross-
-0.521, Kendall
23

---

Published as a conference paper at ICLR 2025
culture generalization in their responses, along with the culture whose memorized symbols appear the most the former cultures, as well as their pretraining occurrence ranked out of all 110 cultures in Table [8] We observe that majority of cultures have the highest generalization from America while Egypt and Saudi have a significant percentage of their generations memorized from one culture Le. Iran.
For (2), our observations indicate that 34 cul- related clothing and 86 related to food have their memorized symbols being gener- ated at least once in other cultures' generations. Upon calculating correlations with these cul- tures, we observed moderate-to-higg correla- tions for both clothing (Spearman p 0.763, Kendall = 0.574) and food (Spearman 0.716, Kendall
0.531). These results sug- gest that cultures whose symbols are frequently mentioned in other cultures' generations also those more commonly appearing in topic- related pretraining documents. We show this correlation through seatter plots for both cloth- ing and food in Figure 14
Figure 13: Correlation b/w number of memo- rized symbols from other cultures and pre-training counts for culture
H.2
RESULTS OVERVIEW
L05
20
20-
Pecssrig Coeta atcata
Sumberr of Sicee outhur Conerratied araateeee
Nanbe ortire Buii unli- cndi souther Curture
(a) Topic: Food
(b) Topic: Clothing
Figure 14:
Generalization
Continuing from Section 4.6 in this section we expand upon our findings and present some more results across the 110 cultures.
In Tables and I1 we present the memorization and generalization statisties for food and clothing. respectively. Specifically, we provide the names of the top 5 and bottom 5 cultures, ranked by the percentage of their responses classified as either memorization or weak association generalization. Cultures with the highest percentage of memorized responses tend to correspond to those that appear more frequently the pretraining dataset. However, notable exceptions exist, such as the culture United States, which, despite occurring frequently in the pretraining data and having a large number of memorized symbols, exhibits only 3.01 % of its total responses as memorized, as shown in Figure
17a
24

---

Published as a conference paper at ICLR 2025
Ordering
w/ Memorization
Weak Association Gen.
Mexico
Trinidad
India
Venezuela
Top 5 (1)
Japanese
South Korea
Morocco
Morocco
Nigeria
Georgia
Qatar
Germany
South Africa
Japan
Bottom 5 (L)
Tajikistan
United States
Trinidad
Italy
Yemen
Denmark
Table 10: Memorization and Generalization Stats for Food
Ordering
w/ Memorization
w/ Weak Association Gen.
India
Uruguay
Saudi Arabia
Venezuela
Top (1)
Japan
Vietnam
Pakistan
Yemen
Canada
Zambia
Uruguay
Colombia
Venezuela
Peru
Bottom (L)
Vietnam
Nicargua
Yemen
Venezuela
Zambia
United States
Table 11: Memorization and Generalization Stats for Clothing
We also observe that a culture with high percentage of memorized responses does not necessarily have a large number of unique memorized symbols. For instance, Pakistan ranks 4th in memoriza- tion count for the topic of clothing but has relatively few unique memorized symbols. This indicates that for some cultures, OLMo-7B tends to repeatedly generate the same memorized symbols when sampled multiple times. Additionally, Table II shows that the bottom cultures, which have the lowest percentage of their responses classified memorized, exhibit the highest percentage of weak association generalization in their responses.
We further provide the distribution of additional cultures, similar to the analysis presented for Mex- ico and Trinidad in Section 4.66 Figure [5] illustrates the distribution of Chinese, Japanese, and Indian cultures for the topic of food. Notably, despite these three cultures being relatively high- frequency in the pretraining data, all exhibit very high percentage of symbols of diffuse association, exceeding 60% in each case. Interestingly, we also observe considerable variance in the overall pres- ence of memorization, ranging from almost 30% for India to only 11.5% for China. Additionally, all three cultures exhibit a relatively low percentage of cross-cultre generalization. This is likely due to their high frequency in the pretraining data, which results in their symbols being generalized to other less frequently occurring cultures.
In Figure 16 we compare the distributions of two less-frequently occurring cultures, ie., Myanmar and Yemen, for the topic of clothing. We observe that, apart from exhibiting very high rates of diffuse association symbols (greater than 70% in most cases), these cultures have memorized symbols according to the classification provided by MEMOED. Yemen, in particular, demonstrates notably high percentage of cross-culture generalization, approximately 21.1%.
we present the distributions for the USA and Saudi Arabia within the topic of clothing. The results for the USA particularly striking, it is one of the most frequently occurring cultures in the pretraining dataset, yet nearly 96% of its responses consist solely diffuse association symbols. Despite containing substantial number of unique memorized symbols, only
Finally, Figure 17
25

---

Published as a conference paper at ICLR 2025
Hamu MESCCTTIO
Work to curton Ser Grasduuuue
2.00%
26.7%
tha Aesoctter
CrsirCaltee tess
Smtve Anacetiinn
EPhiae saedlatio
3,27%
9.225%
(a) China
(b) India
(c) Japan
Figure 15: Distributions of China, India and Japan responses for Food
Weak Association
Cross Culture Gen
Untraceable
21,1 21.1%
8.12%
Cross-Culture
2.61%
Weak Association Untraceable Gen 0.729%
Diffuse Association 72.3%
Diffuse Association 77.9%
(a) Myanmmar
(b) Yemen
Figure 16: Clothing Stats Mynammar and Yemen
3% of its responses qualify memorization. In contrast, Saudi Arabia exhibits greater diversity, with significant percentages of both memorization and cross-cultrre generalization in its generated outputs.
Association Week Resociation atton Gan Untraseable 5,505% Chom--Culture 0.167%6
Memo Association 14.2%
Cross-Culture Gen 12.4%
Association 63%
Weak Association Gen 8.79%
Difise Assacation 99.2%
Untraceable Gen 1.52%
(a) USA
(b) Saudi Arabia
Figure 17: Clothing Stats USA and Saudi Arabia
26


Reviews:
[{"review_id": "1wjN4ZBdZz", "reviewer": "Reviewer_PsWQ", "summary": "This paper introduces a framework, called MEMOED, to determine whether culture related entities, refered to as symbols, are resulted from memorization or generalization of LLMs based on the pretraining data. This study defines three categories of symbols, i) independent symbols, ii) culture-specific symbols, referred to as memorized symbols, and iii) symbols generalized across certain cultures, referred to as generalized symbols. Various experiments are conducted using OLMO on 110 cultures to understand how OLMO's performance is affected by memorization and generalization.", "questions": "* There could be an alternative way to convey symbol and culture overmemorization, if the purpose is to show that certain entities occur more often in model outputs that those observed in the pre-training data.\n* How do you ensure the quality of annotations using culture experts?\n* How symbols are collected? Is there a systematic way to sample data from the 110 cultures?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* This paper selects a good research problem, which is to understand how the frequencies of certain culture related concepts or entities in the pre-training data influence the model performance, in particular from the perspective of generalization and memorization.\n* The high-level ideas to discuss about independencies, memorization and generalization are reasonable.\n* The dataset covers over 100 cultures.", "weaknesses": "* The definitions of the following concepts and their justification are unclear unclear to me.\n    * What is a symbol? Do they cover all linguistic variations of the same entity or concept?\n    * How culture is defined? Why it is represented as a combination of country and natonality. The literature in social science has already defined culture. There could be more than one cultures in a country. Would a representation of country and nationality be overly simplied?\n    * How to justifiy the definition of memorization through Equation (1)? Why it makes sense?\n    * How generalization is defined and why?\n* It lacks of justification of the formula for r(D, Q) in Page 5, as well as the measure for memorization. Why log ratio is preferred over the standard techniques, e.g. statistical dependencies? There are often various ways to convey an entity or a concept. How are linguistic variations captured with this measure? As this measure is used together with the contribution score and z-score to determine if a symbol is memorized. There is no empirical evidence or theoretical justification showing that this measure indeed meets the expectation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hFyql2DejU", "reviewer": "Reviewer_gSz2", "summary": "About culture-conditioned generations out of LMs, dealing with the issue that biased generations can be driven by pretraining corpus statistics.\n\nThey introduce a “symbol attribution framework” to determine if culture-conditioned symbols were memorized in training. They characterize symbols as independent, memorized, or generalized depending on whether they appear in a “culture’s generations” broadly, without a specific culture association, appear primarily in a small set, or if they appear broadly across cultures without presence in the pretraining corpora. \n\nThey use document SNR, minimum token distance, and minimum sentence distance as metrics.\n- document snr is the log probability ratio of counts of culture-referent n-grams to others\n- minimum token distance is the length of the shortest span of tokens between one referring to the culture and one to the symbol\n- minimum sentence distance uses sentences instead of tokens in the above\n\nThey use these functions to construct a heuristic for whether a training document shows the memorizable relationship. They then characterize concepts as being overmemorized, and compare the presence of these statistics in the training data & outputs as a predictor to the agreement of human annotators that these relationships are reflective.\n\nThey claim that “traceable generalization”, ie., concepts that are not closely related according to their metrics in training data are nonetheless successfully generated, are not correlated to “memorized” concepts for a culture. There’s one or the other, for example “Mexico” contains specifically memorized queries, while Trinidad has none.\n\n**Edit**: I have responded to the authors rebuttal, and modified my \"weaknesses\" section. I think the technical contributions are sound, so I have bumped my soundness up to 4 (even though I still am a bit troubled by the scope of the experiments). However, I feel that the presentation of this work is severely flawed, particularly wrt how a reader has to piece together what the experimental methodology was while reading the results. So, I am keeping my presentation score at 2 (though I contemplated dropping it to 1). I will keep my overall score at weak accept---I don't think this paper is ready but if it were to get accepted, interested researchers would be able to make their way through it. I would strongly recommend that the authors consider edits for clarity that address my complaints here in the CR if it does get accepted.", "questions": "This review is a little low confidence; I'm open to changing my mind.\n\nPlease clarify any misunderstandings I have, and elaborate on my concern about the definitions?\n\nWhy were the classes of concept chosen?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "Interesting and useful topic to address.\n\nMildly interesting results; though I am a little unsure about the claims (see weaknesses).\n\nApproach may generalize, not only to memorization of cultural relationships but also to memorization of other facts/information conditioned on context.", "weaknesses": "Presentation of relatively shallow experiments, ~limited technical novelty~, and limited scale of experiments.\n\n~I’m not fully convinced about these definitions that are used to characterize the memorization classes; how do we know that having these statistics over some threshold means that a concept is “memorized” vs just being consistently generated?~\n\n**EDIT:** I understand the paper better after the authors' explanations and edits. I change my mind regarding the technical novelty (which is an unfair complaint to even have in the first place even if it were accurate)\n\nThat being said, I stand by my complaint about the small scale of experiments: while many symbols are generated, and a basically comprehensive set of countries are tested, **only two prompts are used to elicit these outputs.**\n\nOver all, my biggest complaint about the paper didn't make it in to my review, but it's the **poor presentation of the method**. I believe it is a serious problem that key details of the experiments such as \"how many prompts? how many cultures? how were the symbols extracted from the outputs?\" are not clearly lain out before the results. Additionally, the presentation of the methods suffers from a lot of superfluous mathematical notation that clouds clarity, with symbols that once again aren't introduced until *after* an equation is read, requiring considerable backtracking.\n\nSee my response to the reviewers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TOsG5PchBP", "reviewer": "Reviewer_nfL2", "summary": "This paper describes MEMOED (MEMOrization from pretraining document), a framework designed to classify cultural symbols in LLM-generated text as either memorized or generalized.", "questions": "Is it possible to do this analysis on several OLMo models?\n\nThere are some typos: and and (320)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Novel framework and the new problem of analyzing cultural memorization: The paper develops a systematic approach to determine if cultural symbols generated by an LLM are due to memorized data or generalization. This is a novel problem and the approach is sound and elegant.\n2. Good Analysis across Cultures: The study uses data for 110 cultures on topics like food and clothing - the analysis is interesting and the conclusions are interesting as well.", "weaknesses": "Reliance on a Single Model: The analysis focuses solely on the OLMo-7B model and its pretraining dataset, Dolma. Its is unclear from the analysis how the conclusions would vary on other models or models of other sizes.\n\nIt is not clear to me how the definitions of what constitutes memorization (e.g. training document classification)  might change the analysis?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FGjhExV4lM", "reviewer": "Reviewer_UeFD", "summary": "This paper introduces a novel symbol attribution framework to determine whether symbols in LLM generations, conditioned on a culture, result from memorization of pretraining data. The authors' thorough analysis shows that high-frequency symbols are easily memorized but independent of any culture regardless of their correctness. Additionally, by showing the imbalance between the memorization of high-frequency and low-frequency cultural symbols, this paper underscores the need for improved pretraining data and methods to mitigate cultural biases.", "questions": "**Q1**. In Figure 1, the top-down order does not match the numerical order. Why are memorized symbols shown at the bottom?\n\n**Q2**. In Figure 3, what does \"overgeneralization\" refer to? It is not mentioned in the text. Do you mean \"overmemorization\" instead? The same applies to the caption of Table 3.\n\n**Q3**. How are culture-referring n-grams defined for the Document-Signal to Noise Ratio?\n\n**Q4**. Why do the memorization classification criteria differ for cases where n(C_G) > 5 and n(C_G) < 5?\n\n**Q5**. What do the bold texts represent in the \"topic modeling keywords\" column of Table 3?\n\n\n**Typos**:\n\n- Line 345: \"none-memorized\" should be \"non-memorized.\"\n- Lines 106-107: \"for less prevalent symbols\" -> \"for less prevalent cultures.\"\n- There is inconsistent use of \"memorisation\" and \"memorization\" throughout the text. It would be better to use one consistently.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**S1**. This paper introduces a novel symbol attribution framework to determine whether the symbols in LLM generations, conditioned on a culture, result from memorization of pretraining data.\n\n**S2**. The authors provide a nuanced categorization of symbols based on their memorization/generalization levels and a thorough analysis of their relationship with the pretraining data.\n\n**S3**. Their findings demonstrate how LLMs fail to represent cultures that are low-frequency in the pretraining data, calling for improved pretraining data and methods.", "weaknesses": "**W1**. The study is limited to only one pretraining corpus and one LLM, which is understandable given the scarcity of open resources.\n\n**W2**. This study relies on searching for symbols in culture-conditioned generations within the pretraining data and provides a relational analysis. However, it does not guarantee that the selected training documents are causally decisive for the symbols in question. Incorporating influence functions [1] could provide insights into causal relationships. While the computational cost might be an issue, they could be applied to a subset of the dataset or specific experiments.\n\n**W3**. Lines 427-430 require further explanation. What do these correlations imply?\n\n**W4**.  While this study highlights the existing problems with underrepresented cultures in pretraining corpora from a new perspective, it fails to address or propose potential directions for solving these issues. Without this, the paper remains another verification of known problems, which is still valuable but not particularly groundbreaking. The authors should discuss how their findings could inform improved pretraining data/methods or mitigation strategies that do not require changes in pretraining.\n\n**References**\n1. Studying Large Language Model Generalization with Influence Functions, Grosse et al., 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vTNdbZm0xB", "reviewer": "Reviewer_D6x9", "summary": "This paper introduces a novel framework called MEMOED, designed to analyze how pretraining data contributes to cultural biases in large language models (LLMs). The framework distinguishes between knowledge generated through memorization and generalization. By focusing on cultural topics such as food and clothing across 110 different cultures, the authors demonstrate that models tend to overmemorize symbols from highly represented cultures, while underperforming in generating culture-specific symbols for less represented ones. Through a detailed analysis of the OLMo-7B model, the paper offers a systematic method to trace how pretraining data influences model outputs, highlighting the limitations of current LLMs in producing diverse, culturally accurate generations and stressing the need for improved pretraining procedures to address these biases.\n\n(Note: My review has been revised by an LLM for improved grammar.)", "questions": "- Do the authors have any plans to propose methods for mitigating the cultural biases identified in this work?\n- Comment: I recommend adjusting the notation for subscripts, such as $d_{TOK}$. It currently appears a bit unnatural, and applying italics to the subscript, such as d_{\\textit{TOK}}, would enhance clarity.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper makes a significant contribution by addressing cultural biases in LLMs, and the MEMOED framework provides a valuable tool for tracking these biases.\n- The study covers a broad scope, examining 110 cultures, which enhances the depth of the analysis.\n- The concept of overmemorization introduced by the authors is intriguing and may have broader implications beyond cultural biases, potentially applying to other LLM phenomena.\n- The paper opens up the possibility of examining cultural biases in multilingual LLMs across different languages, which could be an interesting direction for future research.", "weaknesses": "While the methodology appears sound, my only concern is the limited scope of the study, which focuses solely on the OLMo-7B model. As a result, the findings might be seen as a case study specific to this model. It would strengthen the paper if the authors included analyses for at least one additional LLM to broaden the generalizability of their findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)