Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysis (2024)

Abstract: Training LLMs in low resources languages usually utilizes machine translation (MT) data augmentation from English language. However, translation brings a number of challenges: there are large costs attached to translating and curating huge amounts of content with high-end machine translation solutions; the translated content carries over cultural biases; and if the translation is not faithful and accurate, the quality of the data degrades causing issues in the trained model. In this work, we investigate the role of translation and synthetic data in training language models. We translate TinyStories, a dataset of 2.2M short stories for 3-4 year old children, from English to Arabic using the open NLLB3B MT model. We train a number of story generation models of size 1M-33M parameters using this data. We identify a number of quality and task-specific issues in the resulting models. To rectify these issues, we further pre-train the models with a small dataset of synthesized high-quality stories generated by a capable LLM in Arabic, representing 1% of the original training data. We show, using GPT-4 as a judge and dictionary learning analysis from mechanistic interpretability, that the suggested approach is a practical means to resolve some of the translation pitfalls. We illustrate the improvement through case studies of linguistic and cultural bias issues.

heading: 1 Introduction
text: Large Language Models (LLMs) have transformed the landscape of natural language processing (NLP) significantly. With the realization of unprecedented capabilities, the availability of under-
lying large training corpora and well-established language model training pipelines has shifted the focus in NLP research from defining linguistic inductive biases to the collection and curation of extensive text datasets (Soldaini et al., 2024; Penedo et al., 2024; Computer, 2023; Mehta et al., 2024). The recent trend showed an increase in focus on data curation and augmentation compared to innovation in model architecture or training paradigms (Brown et al., 2020; Touvron et al.,
73
2023). However, emergent capabilities in LLMs are noticeable for models of very large sizes such as 10B parameters or above and most of smaller LLMs of 200M-3B parameters have shown limited abilities in reasoning, fact recall and coherent long generation (Schaeffer et al., 2024). While these models are typically trained with the same data used to train their larger and more capable siblings, the drops in capabilities of these small LLM are usually attributed to their reduced learnability or scale.
However, newer small models have been recently shown to compete with 10x bigger models in challenging tasks. Examples include MiniMA (Zhang et al., 2023) and the Phi family (Abdin et al., 2024; Li et al., 2023). These small language models (SLMs) offer faster training and lower deployment cost at the expense of being more task-specific and less general-use, which is not an issue for application-oriented language models. The key ingredient for the success of these smaller models seems to be the use of sophisticated and aggressive data curation, and highquality synthetic data generated by bigger LLMs.
This new trend, however, does not map equally to languages that are not as privileged with huge amounts of high quality content (or content in quantities that allow aggressive filtering), nor the availability of strong models that can be employed to generate diverse synthetic data in substantial quantities in a cost-effective manner, as is the situation with Arabic and many other low-resourced languages (Thompson et al., 2024).
A commonly-adopted workaround to the issue of data shortage is to turn to machine translation (MT) to benefit from the available content in English, which is evident in the data mixtures of the more capable Arabic models, e.g. Jais (Sengupta et al., 2023). However, the use of machine translation does not come without pitfalls. Specifically, 1) cultural biases that are stored within a language corpus get imported when translation is used, leading to misaligned models, e.g. (Holmström et al., 2023) and 2) based on the quality of the translation, certain linguistic intricacies of languages might not be respected (Zhang and Toral, 2019), leading to degradation in the quality of the
data and thus degradation in the capability of the final model with regard to the quality of the target language.
In this work, we study this phenomenon using the recently-released TinyStories (Eldan and Li, 2023), a synthetic dataset introduced to explore emergent properties in small language models. TinyStories comprises 2.2M short stories in English of about 200 words generated by GPT-3.5 and GPT-4. The relatively small models trained on TinyStories have shown interesting capabilities in generating coherent and creative short stories with correct grammar. We translate TinyStories to Arabic using the open-source translation model NLLB-3B to simulate medium-quality translation. We train models with different sizes using the Arabic-translated TinyStories, and benchmark the trained models against several other Arabic LLMs using GPT-4 as a judge following previous works (Zheng et al., 2024) in the task of story generation using three metrics: grammar correctness, consistency with the provided context, and creativity. We also identify some cultural and linguistic issues that arise from the translation. Then, we synthesize a high quality small dataset of 20K stories to explore the efficacy of continual pre-training in recovering from the issues brought along by low-quality translated data.
The contributions of this work are as follow:
• We investigate the degradation of language models when using translated data of medium quality in training. We identify linguistic issues and cultural biases, which we address by further pre-training the models with a limited amount of high-quality synthetic data.
• We create a dataset with an Arabic version of TinyStories through medium quality translation, plus a small high-quality synthetic data which we will release as open-source to facilitate studying translation issues.
• We provide a comparative analysis before and after refinement using dictionary learning methods from mechanistic interpretability to assess the effects of continual pretraining with high-quality data.
The rest of the paper is organized as follows: First, we describe how we prepare training data. Then we discuss the limitations of translated data. In Section 3.1, we show how continual pre-training with small amount of high-quality improve the models.

heading: 2 Story Generation SLMs

heading: 2.1 TinyStories dataset
text: First we give a brief description of the original TinyStories dataset (Eldan and Li, 2023). It consists of 2.2M short stories for 3-4 year old kids generated by GPT-3.5 and GPT-4. A set of 1500 basic words consisting of nouns, verbs and adjectives are initially collected that can be easily understood by a 3-year-old child. For each story, a verb, a noun and an adjective are randomly selected from the list. GPT models are prompted to generate a story including all the selected words, and that respects certain storytelling features, like a certain plot or a way of ending as could be seen in Table 6 in the Appendix.

heading: 2.2 Translated TinyStories
text: We translated the dataset using an open source machine translation model that belongs to the ‘No Language Left Behind’ project (Costa-jussà et al., 2022) which is intended primarily for 200 low-resource languages, offering a family of five models with sizes ranging from 600M parameters to 54B parameters. NLLB has good Arabic translation performance compared to other solutions (Tiedemann et al., 2023; Kudugunta et al., 2024). We chose NLLB-3B, a model that can be deployed locally with reasonable hardware requirements, e.g, a single GPU with 16GB memory. The 3B size was chosen with practicality in mind, and to simulate the use of medium quality translation when preparing data on a large scale for language model training.

heading: 2.3 Evaluation Method
text: We followed a similar evaluation approach to the previous work on TinyStories (Eldan and Li, 2023). We translate the test set of 44 manuallypicked story completion prompts from English
to Arabic using GPT-4. An example story completion prompt from the test set could be seen in Appendix A.2. The choice of GPT-4 as a translator was to obtain a high-quality test set, and to ensure that the test data is out-of-distribution to the training data from the translation perspective.
During evaluation, GPT-4 is also used as a judge to assess the story completion of the tested models. The LLM judge is asked to pay extra attention to the initial sentence that is cut short, and to assess three main qualities: correctness of grammar, creativity of the story, and consistency to the prompt and the story details. Each of the metrics is given an integer score between 0 and 10. For each model, two completions with temperature 1 are generated for each test prompt, then average scores are reported.

heading: 2.4 Model Training
text: We use the translated dataset to train a number of small language models. We consider the same model size and configuration as in the original TinyStories family1. The only difference was in replacing GPT-NeoX architecture with Llama-2. This choice was based on the success and wider adoption of Llama to train LLMs. The main contrast between the two architectures is that LLama2 uses gated MLP layers, grouped-query attention and an RMS normalization, whereas GPT-NeoX uses parallel attention MLP and Layer-Norm normalization (Zhang and Sennrich, 2019; Liu et al., 2021; Ainslie et al., 2023; Black et al., 2022). In the chosen model family, the number of heads is fixed, whereas the number of layers and residual stream (latent space) are varied. In 1M, 3M and 8M, the number of layers is kept constant at 8, while the dimension of residual stream is increased. In the remaining models, the increase in the number of layers is compensated by a reduction in the residual stream dimension. The goal is to maintain the model size in the range of 20M30M parameters. Table 1 provides a summary of the architecture of trained models.
For SLM training, the choice of tokenizer is of utmost importance. Since the embedding layer
1https://huggingface.co/roneneldan
can consume a large proportion of trainable parameters, it is crucial to carefully choose the vocabulary size and tokens. Existing tokenizers suited for Arabic language models usually include English or are over-sized for our purpose of training SLMs. We decide to train a byte-pair encoding (BPE) tokenizer on the translated training data. We experimented with different vocabulary sizes (8k, 16k, 32k and 85k). The last two choices are motivated by the Llama-2 and Jais tokenizers. We trained a reference model (33M) with the different tokenizers to evaluate the role of vocabulary size on performance. As shown in Table 2, we selected a vocabulary size of 32k given it performed the highest.
For model training, we used the English data and published benchmarks as a means to calibrate the training process, see Appendix B.1. We use AdamW optimizer with constant learning rate of 5 . 10−5 as in TinyStories paper with 5% steps for warm-up (Eldan and Li, 2023).
The training and validation loss of the training can be seen in Appendix B.2. The shape of the loss curves indicate that the training settings are adequate. Models with lower loss have better
benchmarking performance as we will discuss later. The convergence loss for models trained on Arabic data are higher than the models trained on English data. One justification could be that the noise introduced by translation makes the task of next token prediction more difficult than the original task.
Table 3 shows the GPT-4 scores on the three benchmarking dimensions (grammar, creativity and consistency) for the trained models on translated TinyStories. The best performance is achieved between the 28M and 33M models, and we will use the latter for the rest of this paper. An example completion of the 33M-ar model could be found in Appendix A.2.
We also compare the results of our trained SLMs with some state-of-the-art Arabic or multilingual LLMs of sizes 200M-13B (Radford et al., 2019; Antoun et al., 2021; Koubaa et al., 2024; Workshop et al., 2023; Kamal Eddine et al., 2022; Elmadany et al., 2023; Sengupta et al., 2023) in Table 4.

heading: 3 Navigating Pitfalls of Translated Data
text: There are issues faced when we train models with medium-quality translation: 1) Cultural biases: translated data carries over the culture of the source data into the target language, which might not be desirable. We choose in this work one example from this category to focus on, which is that the translated stories come with English person names, leading to an Arabic model that can only generate stories with such names; 2)
Grammatical and style issues: languages differ in how they express the same sentence, and weaker translations might fail to correct for nuanced style issues. We choose a single example related to how the speaker could follow the saying in English but not in Arabic. For example, the direct Arabic translation of the English sentence “‘I’m happy!’, said Tim.” to
. Õæ K ÈĀ ,"! YJ ª A K @\
is not appropriate.

heading: 3.1 Model refinement with continual pre-training
text: In order to tackle some of the MT issues, we test whether the trained models can be improved with a second stage of training utilizing a small amount of high-quality data that does not have the aforementioned issues. This practical and cost-effective approach is somewhat similar to curriculum learning where LLMs are exposed to different levels of complexity at different stages of training (Zhou et al., 2020; Chang et al., 2021).

heading: 3.1.1 Synthetic Generation
text: We synthesized an additional small amount of high-quality stories using an open LLM capable in Arabic. We chose Command R+ (104B) for the task as it is the more capable model in Cohere’s family of multilingual models which scores highly on Huggingface’s Open Arabic LLM Leaderboard (Elfilali et al., 2024). This new high quality content corresponds to 1% of the
original noisy training data. Instead of deploying Command R+ locally (high compute requirement of 3× A100-80GB GPUs for inference), we generate the 20K stories via paid API calls for a cost of $100. We populated the prompts with samples from a set of random Arabic verbs, nouns and adjectives suited for 3-year-old stories, in addition to a selection story features as used in the original dataset. More details of the process can be found in Appendix A.2.

heading: 3.1.2 Continual pre-training
text: We further pre-train the 33M-ar SLM on the highquality synthetically-generated data from the optimizer state saved after a single full epoch of training on the translated data. The additional data represent about 1% of the original training data. Therefore, the additional training time is small. The loss curves for the continual pre-training could be seen in Appendix B.3.The benchmarking results after refinement can be seen in Table 5. Continual pre-training shows performance improvement over the three metrics. Other refinement strategy such as multi-epochs, mixing with translation data in pre-training and then further pre-training with high-quality data will be explored in future work.

heading: 3.2 Interpretability using Sparse Auto-Encoders
text: In order to investigate whether the refinement process has managed to deal with the identified issues, we turn towards mechanistic interpretability. The traditional neuron-based methods have been shown recently not to be sufficient as neu-
rons in language models appear to be polysemantic (Bricken et al., 2023), i.e., a single neuron might activate on multiple unrelated concepts, and traditional interpretability approaches cannot disentangle the real cause of a certain phenomenon. Recently, Dictionary Learning (Elhage et al., 2022a) has been developed as an alternative tool to interpret language models. The use of Sparse Auto-Encoders (SAE) (Bricken et al., 2023; Rajamanoharan et al., 2024; Makelov et al., 2024) to project neuron activations to a sparse and large dimensional feature space has led to a highly-interpretable representation that captures knowledge in generative language models not seen at neuron levels.
In order to understand the effect of our continual pre-training, we train a Sparse Auto-Encoder for both base and continually-trained models on the output of the last MLP layer. For these analyses we choose 2L-33M-ar model2 (see Table 3).

heading: 3.2.1 Training SAEs
text: The architecture of our SAE is one hidden layer MLP trained as an autoencoder, with input weights as an encoder and output weights as the decoder. We chose an expansion factor of 16. The training data for SAEs is prepared as follow: For each context in the training data, the MLP activation vectors are collected after the SILU non-linearity for each token in the context. Activation vectors are sampled from 128 tokens within each context. These sampled vectors are then shuffled together, ensuring that the samples in a given batch originate from a variety of different contexts. Adam optimizer is used to minimize an objective function consisting of two components: the mean squared error loss and an L1 regularization term. The L1 regularization penalty is applied to the activations of the hidden layer within the autoencoder architecture. For a given feature the logit weight is defined as the product of a feature direction and the unembed WUWdec[feature]. The logit weight measures
2This model has previously been used for SAE analysis on English TinyStories: https://huggingface.co/colle ctions/lovish/tinystories-sae-regularization-c omparison-65edb87a85cb3a3386c4b062
the direct effect of the features on the likelihood of next-token prediction. Examples of logit weight distribution can be found in (Johnny Lin, 2024)3.

heading: 3.2.2 Token Set Enrichment Analysis
text: The Token Set Enrichment Analysis (TSEA) introduced in (Joseph Bloom, 2024) is borrowed from Gene Set Enrichment Analysis in bioinformatics (Subramanian et al., 2005) to statistically quantify if a set of features is strongly associated with a hypothesis. TSEA examines the relationship between the distribution of logit weights and predefined sets of tokens. The latter could represent various semantic or linguistic categories such as set of token of boy or girl names. TSAE would in that case identify which features inhibit or promote these sets of names. The steps in TSEA are: 1. Generate a library of token sets for the hypothesis under investigation. 2. Compute the enrichment scores for all features across the sets. The enrichment scores are running sum statistics that expose which features promote or suppress tokens in the hypothesis sets. 3. Identify elevated points in the enrichment scores. 4. Inspect features with high enrichment scores to validate the hypothesis.

heading: 3.2.3 Analyzing MT issues with TSEA
text: In this work, we investigate two hypotheses as case studies of the effect of continual pre-training with high-quality data: 1. The cultural bias of carrying English names to translated data. 2. The English language dialog follows the template “[Quoted text],” said [Person], while in Arabic, the template is Said [Person]: “[Quoted text]”.
For cultural bias, we defined two token sets of 600 common English and Arabic first names. TSEA help identifying which set of names is higher represented in the model. Figure 2 shows the scatter plot of enrichment scores for English vs. Arabic names. Features above the diagonal activate stronger on English than Arabic names, and vice versa. We identified and marked the features farther from the diagonal with at least 2
3https://www.neuronpedia.org/gpt2-small/8-r es-jb/6649
point difference between the English and Arabic name enrichment scores, indication strong bias. The plot shows a higher concentration of features representing English names, while the similar plot after continual pre-training in Figure 3 shows only one feature has the gap. The overall distribution is corrected away from English names, as also could be seen in generated samples. This illustrates the benefit of continual pre-training in correcting cultural bias.
The second case study is on the linguistic issue related to inappropriate dialog tagging in Arabic, which we noticed to be frequent in the translated TinyStories due to the quality of NLLB-3B. We defined the token set with English and Arabic persons names ending a sentence, and performed TSEA analysis before and after continual pretraining. Figure 5 shows the Manhattan plots of enrichment scores of both case studies. The first panel on the left shows the scores for dialog tagging for the model trained on translation data (a) and after continual pre-training (b). We carefully inspected the dashboards of top-3 features (2322, 3353, 14589) and (50, 1455, 9578). We concluded that the top-features in the updated model activate on correct dialog tag phrases whereas the base model shows clearly the issue.
Feature #3144 is a good example to illustrate both case studies. Figure 5 shows the feature dashboard4. The histogram on the top right is of sampled nonzero activations. The list of token of positive and negative logits underneath are the lowest and highest logit difference tokens of that feature, i.e., tokens whose probabilities of being sampled decrease or increase the most when the feature activates. Notice that the highest ranked tokens are of Arabic names, meaning that the model favors to produce an Arabic name after the verb ú«Y K (which means "called") that activates the feature highly as seen in the top activation panel to the left. The color highlights activation level of the
4A guide to read the dashboard could be found in https: //transformer-circuits.pub/2023/monosemantic-f eatures/index.html#setup-interface and more details can be found in (Bricken et al., 2023; Bloom, 2024)
token. A blue underline represents a lower loss (better prediction of that token), whereas a red underline represents a higher loss. The top activation examples illustrate correct examples of proper names ending a sentence.

heading: 4 Related Work

heading: 4.1 Continual pre-training
text: Continual pre-training has been previously used for domain adaption in LMs (Gupta et al., 2023; Ke et al., 2023). For example, general-purpose
LLMs are continually pre-trained for domain specific applications such as in financial applications (Xie et al., 2023) or new languages such as in AceGPT and SeaLLMs (Huang et al., 2023; Nguyen et al., 2023).

heading: 4.2 Mechanistic Intepretability
text: Despite the progress in training capable language models, underlying working mechanisms are still not well understood. This presents risks and challenges in terms of model safety and robustness, as the model may produce inconsistent, irrelevant,
or harmful outputs when faced with unfamiliar or adversarial inputs. The ability to quantify and understand working mechanisms in language models at different phases of model development is an important research topic. The field of mechanistic interpretability aims at providing principled tools and methods to reverse engineer working mechanisms in language models (Elhage et al., 2021, 2022b).

heading: 5 Conclusion
text: In this work, we investigate the role of machine translated data in training language models for story generation in Arabic. We identify linguistic and cultural bias issues in the translated data. We propose to address these issues by further pre-training the models with a small amount of high-quality synthetic data. We investigate the effects of this intervention via Dictionary Learning tools. The trained Sparse Auto-Encoders shows a shift in the learned features towards corrected linguistic properties and reduced cultural bias.

heading: 6 Limitations

heading: 6.1 Limitations of GPT-Eval
text: The use of LLM-as-judge approach to evaluate open-ended generation comes with some limitations. The complexity of the task of evaluating consistency, grammar and creativity is sometimes challenging for GPT-4. We noticed its limitations in complex Arabic evaluation compared to English. A reassessment of GPT-4 in Arabic with respect to newer models such as Llama-3 and Command R+ would be helpful to decide on the best LLM judge.

heading: 6.2 Generalization of the Results to Larger Datasets
text: While TinyStories provides a testbed for exploring different facets of language models, the extension of this work to larger models faces challenges: as the model size grows, there is a need for a large dataset which shifts the burden on synthesizing diverse and high-quality dataset. The work done in models such as Phi-2 and Phi-3 are
promising direction to address this challenge (Abdin et al., 2024; Li et al., 2023).

heading: 6.3 Extension to Other Domains
text: Developing small generative language models that exhibits emergent properties is challenging for tasks beyond creative writing. For example, applications related to generic question answering would require large amount of data and larger models. Further research is needed to identify interesting applications and define the requirements in terms of data volume and quality to train highly capable small language models for other applications.

heading: 6.4 Instruction Fine-Tuning
text: This work is limited to the training of base models. An instruction fine-tuning dataset of English TinyStories is available and similar work in the paper could be applied for the analysis of finetuned models. We will investigate this in future work.

heading: A Story Samples
text: A.1 Example NLLB-3B translation Here is an example of a story and its translated version using NLLB-3B.
A.1.1 Original story in English One day, a little boy named Tim went to the park.He saw a big tree and wanted to climb it.Tim was frightened, but he took a deep breath and started to climb. As he went up, he heard a small voice.The small voice said, "Please help me!"Tim looked around and saw a little bird.The bird had a hurt wing and could not fly.The bird was frightened too.Tim wanted to help the bird, so he thought of a plan.Tim took a big leaf from the tree and folded it.He made a soft bed for the bird. He put the bird on the leaf and climbed
down.The bird was so happy and said, "Thank you, Tim!"The little boy and the bird became good friends.
A.1.2 Translated story using NLLB-3B úÍ@ Õæ
K ú«YK Q ª ú æ. I. ë X ÐAK B@ áÓ ÐñK ú
̄
é JºËð , Õæ K ¬A g . Aê ®Ê K à @ X@P @ð èQ J.» èQm. ø @P . é ®K YmÌ'@ A Kñ ©ÖÞ , Yª AÓY J« . Ê Ë @ ú ̄ @YK. ð A ®J Ô« A ® K Y g @
Õæ K Q ¢ " ú GY«A ½Ê ̄ áÓ\ : @Q ª Ë@ Hñ Ë@ ÈĀ . @Q ª
©¢ J ÕËð A gðQm.× hA Jk. Q KA¢ÊË àA¿ . @ Q ª @ Q £ ø @Pð éËñk
@ YË ,Q KA¢Ë@ èY«A Ó Õæ K X@P @ . A @ A ® KA g Q ¢Ë@ àA¿ . à@Q ¢Ë@ © J . AîE ñ£ð èQj. Ë@ áÓ èQ J.» é̄Pð Õæ K Y g @ . é¢ k ú ̄ Q º ̄ . É ® CË Ê ð é̄PñË@ úÎ« Q KA¢Ë@ © ð . A Ô«A K @ QK Qå Q KA¢ÊË
ú æ. Ë@ iJ. @ "! Õæ K , ½Ë @Qº \ : ÈĀð @Yg. YJ ª Q KA¢Ë@ àA¿ . áK YJ k. ZĀY @ Q KA¢Ë@ð Q ª Ë@
A.2 Synthetic generation using Command R+
We prompt Command R+ to synthesize a small set of high-quality Arabic stories as describe in 3.1.1. The prompt we used is shown in Figure A.1, and the story generation features in Table 6.
Here is an example of a generated Arabic story:
AÖ ß @X Éj. ª J Ó àA¿ YÔg @ . YÔg @ éÖÞ @ Q ª É ®£ ¼A Jë àA¿
AÒ J K. , ÐAK B@ Yg @ ú ̄ . Ò Ë@ Im ' h. PA mÌ'@ ú ̄ I. ªÊË @ I. m ' ð Q¢ÖÏ @ I. m ' ÕË YÔg @ . è Am. ̄ Q¢Ö ß ZAÒ Ë@ H @YK. ,I. ªÊK YÔg @ àA¿ ú ̄ YÔg @ Qº ̄ .I. ªÊË @ áÓ éª JÖß ð é . CÓ ÉÊJ. K àA¿ é K B @YK. @ ú ̄ I. ªÊË @ ©J ¢ ú æk é ® JK. éJ ÖÞ © J à @ PQ̄ð Ég
: AêË A ð éÓ @ úÍ@ YÔg
@ I. ë X. ZAÒ Ë@ Q¢Ö ß AÓY J« ú æk h. PA mÌ'@
à @ YK P @ ? éJ ÖÞ © J ú ̄ ú æK Y«A à @ áºÖß Éë , ú× @\ : IËĀð éÓ @ IÒ K. @" . ú æj. « QK Q¢ÖÏ @ áºËð h. PA mÌ'@ ú ̄ I. ªË @ éÖß Y ®Ë@ é Ô̄ B@ ªK. Ð@Y j J @ A J JºÖß . YÔg @ AK YJ » A JËAK. \ @PñJ. àñº K à @ I. m. ' áºË . éª K @P éJ ÖÞ © J Ë éÊK ñ£ A «ð é JºË @YK. @ @PñJ. áºK ÕË YÔg @" . A J̄ð Y g AK Zú æ Ë @ © J à B èYªË Ym. '. éÓ @ ©Ó YÔg @ ÉÔ« . éÊJ Ôg. éJ ÖÞ X@P @ à B ̄ @ð úÍ@ éÊK ñ¢Ë@ A ªË@ @ñËñkð AªÓ èñ ® Ëð AÒ ®Ë@ @ñª¢̄ . ÐAK @
I KA¿ð éJ Ò Ë@ © J áÓ @ñî D K @ , @Q g @ . éJ Ò ÊË J. ®Ó

heading: H. Qj. J Ë h. PA
text: mÌ'@ úÍ@ h. Q kð @Yg. @YJ ª àA¿ YÔg @! éª K @P
. ©̄ñ JÓ Q « Zú æ HYg , Aêj J ̄ AÓY J« , áºË . èYK Ym.Ì'@ é J ÖÞ úÎ« èQ¢J Ë@ ©¢ J ÕË YÔg @ð èñ ®K. I. î E H @YK. hAK QË @ YÔg @ . Z AÒ Ë@ ú
̄ I ® J k@ð èYK áÓ éJ Ò Ë@ HPA£ . éJ Ò Ë@ Ym. '. ÉÔ« ú æË @ éÊJ Òm.Ì'@ é J ÖÞ Y ® ̄ é K B @Yg. à Qkð ú¾K.
AK ú ¾J. K B\ : éJ @ñ K à @ IËðAgð èY JªË HZAg. Ð B@. Aêª J Ë áºË . Aª K @P A J Iª J ð Ym. '. IÊÔ« ½ K B ½K. èPñ m ̄ A K @ . YÔg @ ÉÒª Kð áK PñJ. àñº K à @ I. m. ' , AÓ A J YK Q K AÓY J« ,Q»
Y K Zú æ É¿ A JÊª ̄ @ X @ ú æk , àAJ k B@ ªK. ú
̄ . é ®J ®j JË Ym. '. áÓ Z Qk. @ Yë . YK Q K AÓ úÎ« É m ' B Ȳ ,iJ m É¾ . YÔg A.ÊÓ B@ HAJ. J k ©Ó ÉÓAª J K J » ÕÎª J K à @ I. m. ' ð
èAJ mÌ'@ Q . Ë@ð XAm.Ì'@ ÉÒªË@ à @ ÕÎª K . ÐñJ Ë @ ½Ë X AÒêÓ A PX ÕÎª K ú ̄ . A J KQ¢J á« ék. PA g ZAJ @ ¼A Jë à @ A @ áºËð , àAÒêÓ É m ' B Ȳ , èY . AÓ A J A Jj. Jk@ @ X @ ú æk , àAJ k B@ ªK. ÕÎª J K à @ áºËð , éËðAjÖÏ @ á« ̄ñ J K à @ ú æªK B @ Yë . éJ Ê«
. ñÒ JË @ ú ̄ QÒ J ð ÕÎ̄ A J K J »
A.3 Completion example
Here is an example of a prompt from the test set (top panel of Figure A.2) and model 33M-ar completion (bottom panel).

heading: B Training Details
text: B.1 Process calibration using original TinyStories
We calibrate the training process by first training the chosen architectures on the original TinyStories dataset and comparing the benchmarks with
the published results. Table 7 shows GPT-4 scores for some trained models, and Figure A.3 shows the loss curve for the training.
B.2 Pre-training loss curves
Figures A.4 and A.5 show the training and validation loss, respectively, for the pre-training stage using translated TinyStories.
B.3 Continual pre-training loss curves
Figures A.6 and A.7 show the training and validation loss, respectively, for the continual pretraining stage using the Arabic synthetic story dataset.
B.4 Feature dashboard examples
Figure A.8 is a dashboard summarizing the properties of the selected feature #14589. The top right histogram gives the activation distribution for the feature. The panel underneath shows the highest and lowest ranked tokens based on the logit difference when the feature is ablated. The text panels show training samples for different feature activation intervals. Feature #14589 captures the verb ’say’ in Arabic in different tense
forms following a quote. This construction is not correct in Arabic as the use of the quote tagging is given before the quote. This reversed structure has been inherited from English due to the lowquality translation of the dataset. This feature helped us reveal some of the limitations of our model trained with low-quality translation data.

heading: Future Work
text: We will investigate this in future work