Persona-aware Generative Model for Code-mixed Language
Authors: Ayan Sengupta
Md Shad Akhtar
Tanmoy Chakraborty
Venue: Accepted by TMLR ()

Abstract: Code-mixing and script-mixing are prevalent across online social networks and multilingual societies. However, a user's preference toward code-mixing depends on the socioeconomic status, demographics of the user, and the local context, which existing generative models tend to ignore while generating code-mixed texts. In this work, we make a pioneering attempt to develop a persona-aware generative model to generate texts resembling real-life code-mixed texts of individuals. We propose PARADOX, a persona-aware generative model for code-mixed text generation, which is a novel Transformer-based encoder-decoder model that encodes an utterance conditioned on a user's persona and generates code-mixed texts without monolingual reference data. We propose an alignment module that re-calibrates the generated sequence to resemble real-life code-mixed texts. PARADOX generates code-mixed texts that are semantically more meaningful and linguistically more valid. To evaluate the personification capabilities of PARADOX, we propose four new metrics -- CM BLEU, CM Rouge-1, CM Rouge-L and CM KS. On average, PARADOX achieves $1.6$% better CM BLEU, $57$% better perplexity and $32$% better semantic coherence than the non-persona-based counterparts.

Published in Transactions on Machine Learning Research (10/2024)
Persona-aware Generative Model for Code-mixed Language
Ayan Sengupta
wyw..sennguplallee...d....
Department of Electrical Engineering Indian Institute of Technology Delhi
Md. Shad Akhtar
shad akhtar Chiit
Department of Computer Science & Engineering Indraprastha Institute of Information Technology Delhi
Tanmoy Chakraborty"
lanckak@rild.com in
Department of Electrical Engineering Yardi School of Artificial Intelligence Indian Institute of Technology Delhi
Reviewed on OpenReview: https: https:://penrevveu.. net/ forum? id= f2P49IsVIh
Abstract
Code-mixing and script-mixing are prevalent across online social networks and multilingual societies. However, a user's preference toward code-mixing depends on the socioeconomic status, demographics of the user, and the local context, which existing generative models lend to ignore while generating code-mixed lexts. In this work, make pioneering al- tempt to develop a persona-aware generative model to generate texts resembling real-life code-mixed texts of individuals. We propose PARADOX, a persona-aware generative model for code-mixed text generation, which is a novel Transformer-bant that encodes an utterance conditioned on a user's persona and generates code-mixed texts without monolingual reference data. We propose an alignment module that re-calibrates the generated sequence resemble real-life code-mixed texts. PARADOX generales code-mixed texts that semantically more meaningful and linguistically more valid. To evaluate the personification capabilities of PARADOX, we propose four new metrics CM BLEU, CM Rouge-1, CM Rouge-L and CM KS. On average, PARADOX achieves 1.6% better CM BLEU, 57% better perplexity and 32% better semantic coherence than the non-persona-based coun- terparts. The source code available at: http:e//gitaub.cco///ttt//////62
mncoder-decoder model
1 Introduction
Code-mixing (aka code-switching) appears when two or more languages are used interchangeably in single utterance. It is common in multilingual societies like India, where more than 24% of the population speaks in more than one language (Sengupta et al.) (022I.. Code-mixing is even more prevalent on social media. Informal usage of code-mixee languages on social media platforms like Twitter, Facebook, YouTube, and other online social networks gives rise to script-märing, in which user
a single script (e-g., Roman) multiple scripts (e.g., Devanagari for Hindi and Roman for English) within the same text (Srivastaval let al. (020). Recent literature has made significant eflorts to understand syntactic structure and semantics from code-mixed texts (Singh et al.) 2018a bl Sengupta et al. 2022b). Similar attempts have been made for pragmatic tasks humour, sarcasm and hate detection in the code-mixed regime (Sengupta et al. 2022a Bansal et al. 2020
Cooreepooning Author

---

Published in Transactions on Machine Learning Research (10/2024)
Text generation models need to understand the syntax and semantics of texts and preserve semantic coher- ence during generation. Previous studies utilized recurrent neural networks with generative models (Zhang) et al.] 2017|, as well as self-attention--asser pre-trained language models (Zhang et al. 2020) for gener- ating monolingual texts. However, such an effort is limited in case code-mixing. Previously, linguistic theories (Pratapa et al. 2018 Gupta et al.) 2020), transfer learning (Gupla et al. 2020). and autoencod- ing (Samanta et al., 2019) based approaches have been used to generate code-mixed texts from parallel corpora or reference data. However, none of these methods incorporates user information while generating code-mixed texts. Unlike traditional languages, code-mixing is derived language whose adoption depends different socioeconomic, demographic, and linguistic factors (Rudra et al.. 2016 Parshad et al. 2016). Figure | demonstrates the code-mixing behaviour among Indian users on Twitter and YouTube in terms of adoption and pallerns in code-mixing. We visualize the mean and standard deviation of the Code-Mixing Index (CMI) Das & Gambäck] 2014) and the length tweets/comments posted by different users. The distributions show how different users conceive and prefer code-mixing.
User persona plays vital role in generation models, particularly in personalized generation settings, such conversational agents and recommendation engines. Several studies have contributed towards persona- based dialogue generation (Zheng et al., 2019 Wang et al. 2021), personalized story generation (Chandu et al. 20191, and other sub-tasks in text generation. Being a conversational language, personification of code-mixing could be deemed appropriate for conversational systems such as recommender engines, mental health counselling bots, and event booking applications.
This motivates us to develop PARADOX, novel persona-aware generative model for code-mixed text generation. It aims to generate personalized code- mixed texts by leveraging users' historical utterances.
10-1.
Prebabiity
20.9.
It uses a Transformer-basees encoder-decoder archi- tecture to learn the semantics of code-mixed genera- tion. The model utilizes a novel persona encoder lo encode a user persona from their behavioural prefer- ences. Instead of projecting the user's persona onto static space, PARADOX projects it onto proba- bilistic latent space and captures the contextual per- sona based on their historical persona. Additionally, PARADOX uses an aligument module to re-align de- coder outputs to generate coherent lexts.
10-1
0.30
201
Avarage
Irobabiity (ag scale)
40
a0-0
10-4.
Stenderd desteliin of
Standard deniiel inn twat length
Twitter
YouTube
Figure 1: User-specific distribution of Code-Mixing In- dex (CMI) and text lengths across different platforms.
We evaluate PARADOX against the vanilla Transformer in terms of both the quality and coherence of gener- CMI is calculated as the fraction of minority language words in text.. For instance, the CMI of the text. "I don't want your nautanki" ("I don't want your gim- rics CM BLEU, CM Rouge-1, CM Rouge-L, and mick") is ! = 0.2, the fraction of Hindi (minority lan- guage in this example) words in the text. Texts skewed
ated texts. To quantify the extent of personification in the code-mixed generation, we propose four met-
CM KS (here CM stands for code-mixing). On aver- age, PARADOX achieves 1.6% better CM BLEU than toward monolingualism, i.e., having an uncqual pro- the non-persona counterpart. We also conduct de- portion of words between different languages, tend to tailed human evaluation, concluding that PARADOX- have lower CMI than multilingual texts. generated code-mixed texts are 32% more semanti-
cally coherent than that of the vanilla Transformer model. PARADOX can imitate user's linguistic preference Transformer model. Our empirical analyses also highlight the effec-
4% better than the non-persona-basee tiveness of PARADOX over pre-trained large language models. On average, PARADOX achieves 4.2% better CM BLEU, 11% better CM Rouge-1, and 9.6% better CM Rouge-L than the pre-trained Llama (Touvron et al. 2023 and GPT-4 (Achiam et al. 2023) models.
Contributions. The major contributions of this paper are summarized below:
We make a pioneering effort in utilizing user persona in code-mixod text generation. Compared to existing approaches, PARADOX does not require parallel corpora or reference data for code-mixed text generation.
2

---

Published in Transactions on Machine Learning Research (10/2024)
We propose a probabilistic persona encoder module that learns the latent persona of user from historical contexts. PARADOX captures the user persona implicitly and does not require explicit persona features such as user demographic information to encode user behaviours.
We design an alignment module to automatically induce alignments between different subwords. Empirical results show that it improves the coherence of generated lexts.
We propose three metrics influenced by supervised machine translation CM BLEU, CM Rougo-1, and CM Rouge-L for evaluating the personification of code-mixed generation models. We also propose CM KS as a distance measure to evaluate code-mixing generation models.
Finally, we collect two large-scale longitudinal datasets from Twitter and YouTube, primarily monolingual Hindi and Hindi- Hindi-English code-mixed texts. The datasets will be valuable for code-mixing research.
2 Related Works
Rule-based and Linguistic Approaches. Codo-mixed text generation has garnered much interest in recent. times. Pratapa el al. (2018) explored equivalence construint (EC) theory lo generale Yeanish-Englisl code-mixed texts from monolingual corpora. Their linguistic theory-based approach showed superiority over recurrent neural networks in complex code-mixed text generation. Rizvi et al. (2021) developed GCM, a toolkit that utilizes different linguistic theories to generate code-mixed texts. Motivated by embedding ma- trix theory. SSrivstava & Singh (2021) proposed rule-based methods to generate Hindi-English code-mixed texts. Santy et al.] (2021) utilized parse tree structures within the monolingual texts for generating code- mixed lexts. Alternative approaches use generative models generalive adversarial networks (Goodlellow et al. 2020), or variational autoencoder (VAE) Kingma & Welling 2014). Towards this, Garg et al. (2018) explored recurrent neural networks with SoqGAN pro-training for generating Mandarin-English code-mixed data. [Samanta al. (2019) developed VAE-based method to generate realistic and coherent Hindi- Sindi-EEnglihh code-mixed texts. Other classes of code-mixed text generation models explore alignment within parallel corpora for code-mixed generation. Notably, WWnata et al. (2019) Tan & Joty (2021) explored word align- ments and candidate selection from parallel corpora for generating synthetic code-mixed texts. Amin et al. (003)) explored word alignments for generating Marathi-Englisl code-mixed text generation. On a similar attempt, Dowlagar & Mamidi (0221) explored gated convolutional encoder-decoder models to identify the compositional structure and translate English texts to Hinglish.
Pre-trained Models. With the inception of self-attention (Vaswani et al. 2017). several attempts have been made to develop large pre-trained models showing exceptional performances in semantic and gener- alive lasks. Among these methods, multilingual models, such as XLM (Conneau & Lample 2019). XLM- RoBERTa (Comneau et al. 2020), and mBART (Liu et al., 2020) have shown noticeable performance even on low-resource languages. Recently, MuRIL (Khanuja et al. 2021 was proposed, supersoding the perfor- mances of multilinnullBBEETT (Devlin et al.
2019) on different syntactic and semantic tasks on a diverse set of low-resource languages. Gupta et al. (2020) devised semi-supervised approach to transfer knowl- edge from XLM to generate synthetic SlnniiEEgglin code-mixed texts. Gautam et al. (2021) explored a pre-trained mBART model for generating Hindi-English code-mixed texts. Jawahar al [2021] | explored multilingual text-to-text models with curriculum learning for generating Hindi-English code-mixed texts. They pre-trained an encoder-decoder model on synthetic code-mixed texts, which improved the generation quality on the gold code-mixod datasct. In a recent study, Yong et al. (2023) explored multilingual large language models (LLMs) for generating code-mixed texts in a zero-shot setting. They explored InstructGPT GPT (Ouyang et al., 2022), BLOOMZ (Muennighoff et al. 2022) and Flan-T5-XXL. (Chhnn et al., 2022) for generating code-mixed texts in Indonesian, Malay, Chinese, Tagalog, Vietnamese, Tamil, and Singlish. They further emphasized the importance of better prompt templates and language pairing for generating more coherent and natural code-mixed texts.
Personification of Code-mixing Languages. Language, as a mode of communication, is often person- alized for different users (King & Cook 2020). The personification of language arises naturally, depending on the context,, socio-cconomic and demographic background of the user and their audience (Reichelt et al., 2014). Particularly in multilingual socicties, the complex dynamics between different languages play cru- cial role in the personification of language. [Sngupta et al. (2024) highlighted the personalization aspects of
3

---

Published in Transactions on Machine Learning Research (10/2024)
Addition
Dot-product
Matrix multiplication
Linear
Scaler Multiplization e- N(0.1)
Nomendature
Add & Norm
Output Probabilities
Contextual
Encoder
Persona
FeedForward
Softmax
Encoder
Add & Norm
Add & Norm
Add
Nx
FeedForward
M.H. FAME
Softmax
μ
Add & Norm
Add & Norm
Alignment Matrix
Linear
Linear
M.H. FAME
Masked M.H.
FAME
Linear
Linear
Positional
Positional
Decoder
Token
Encoding
Persona Embedding
Encoding
Alignment
Input Embedding
Output Embedding
Input Tokens
User ID
Output Tokens (Shifted right)
Figure 2: PARADOX: Transformer encoder-decoder architecture with persona encoder (multi-headed (M..) fused attention (FAME)).
Hindi-English code-mixed language, highlighting the importance of different sociological aspects behind the evolution of personalized code-mixed languages.
Major Limitations of Existing Studies. Despite their popularity in several generative applications, personalization remains neglected in the code-mixed generation. The existing code-mixed generation models utilize parallel corpora to understand the switching patterns and generate code-mixed texts synthetically, These limitations motivate us to develop a code-mixed language generation model that can automatically learn the language's semantics and capture the linguistic preferences of users while generating texts. Our proposed method, PARADOX, improves the quality of generated texts and preserves the real-life phenomenon of code-mixing among users.
3 Proposed Methodology
Here, we explain PARADOX, and the personalized code-mixing (CM, henceforth) generation process utiliz- ing user persona. Before describing the proposed generative model, first elaborate on the definition of personalization of code-mixed language.
3.1 Personalized mixe-mixing
As highlighted in Section 2 the study of the personalization of a language is crucial to develop conversational generative models. Typically, a personalized generative model captures the user's preferential behaviors dur- ing generation. Instead, our work delves into the personalization of user's linguistic preferences in terms of mixing multiple languages in an utterance. We define the personalization of code mixing as a behavioral language that captures user's writing style given the historical trends, social context, targeted demograph- ies, and topical relevance. The proposed method, PARADOX, aims to capture these behavioral and contextual
4

---

Published in Transactions on Machine Learning Research (10/2024)
characteristics of user historical utterances and generate texts that are aligned with their linguistic prefer- ences.
3.2 PARADOX Architecture
PARADDX, shown in Figure 2 consists of four components (a) an encoder, (b) contextual persona encoder, (c) a decoder, and (d) an alignment module. PARADOX utilizes persona encoder to implicitly encode user's persona based on his/her historical utterances. It further projects the persona onto probabilistic latent space to capture the user's contextual persona. Finally, PARADOX employs an aligmment. module to re-calibrate the output sequences that help our model understand the language of vode-mixing and enable the model generate coherent texts.
3.2.1 The Encoder with FAME
The encoder is used to jointly learn the semantics of a code-mixed text and a user's global persona based on the provious comments/ ceeiant//twwtttee /twects. A comment/tweet X, by user u first tokenized using byte-pair encod- ings (Senrrich et al. 2016) into (2), F2, F2,, The initial contextual embedding token æs is conditioned with the user persona as
Emb((a..)) Emb., + PE, + Emb,
where Emb,, is the initial token embedding, PE; is the positional encoding at position i, and Emb, is the user's global persona embedding captured through an embedding layer. Emb, is computed with unique identifier for each user. This unique user ID is important to encode the temporal evolution of user persona. Tn order calculate the global persona for cold-start users (users present the development set but not during model training), we use learnable [UNK] token embedding Emb,. We each encoder consists of multi- fused attention (FAME), followed by residual, layer-normalization. and pointwise feed-forward layers. (Snnuptaa et al. (2021) introduced FAME by combining scaled dot-product attention (Vaswani et al. 2017) and outer-product attention (Le et al. 2020 and showed to be effective in capluring both semantics and morphology of code-mixed texts.
stacked encoder, in which
3.2.2 Contextual Persona Encoder
We project the static persona embedding onto probabilistic latent space for generating the contextual persona embedding for each user in given context. We hypothesize that each user has a static (global) persona and a contextual (local) persona. The motivation behind projecting the persona embedding to a latent space is to capture the contextual perturbations in the user persona. For instance, Table highlights a user who predominantly uses monolingual English for raising political opinions (CMI 0.0), however, at occasions switches between English and Hindi to create stronger narrative.
User II)
Generated Text
CMI
CM: This school student is passionate supporter of clean politics.
0.0
Eng: this school student is a passionate supporter of clean polities.
CM: once again vendetta politics will fail. Satyamov jayate.
0.21
158
Eng: once again vendetta politics will fail. Truth will triumph.
CM: This so called party with difference has links with brastacharis.
0.12
Eng: This so called party with difference has links with corrupts.
Table 1: Examples of user linguistic behavioral changes in different contexts. Usage of Hindi words are highlighted with blue. The user predominantly uses monolingual English for expressing political opinions. However, at different instances the user uses Hindi interchangeably for referring to contextual popular nar- ralives for strengthening their expression.
Formally, we generate contextual persona embodding
Emb,, go(z) Fmmb)) N(uus o
(2)
5

---

Published in Transactions on Machine Learning Research (10/2024)
Towards this, we define two linear projection matrices to learn the distribution location and scale parameters 88
Pa = Emby.W Wi and o = Emb., Wo.
Following the reparameterization trick Kingma & Welling 2014 we define the final generated persona encoding as:
Emb, Mu + Eu au
where cu is the random noise, independently drawn from 1). We refer to this method randomized persona encoder. In another ablation, we only the linear projection Emb, shu Emb,, Wa for representing the contextual persona embedding, which we call linear persona encoder. We obtain the hidden representation of token 2; conditioned on the contextual persona encoding as
h(m.,))) = h(xr,)) + Embh
(4)
where ((x,o) is the final hidden representation obtained from the final layer of the encoder.
3.2.3 The Decoder
We adopt the Transformer decoder conditioned on the contextual user persona. Similar to the original Transformer decoder, we use stacked decoder initialized with the encoded output sequence. Drawing the motivation from autoregressive generative language models like GPT2 (Radford et al. 2019), the decoder's objective is to predict the next loken, conditioned on all the previous tokens. The input to the decoder is the encoded input sequence added with positional encoding. Each decoder block consists of masked multi-headed FAME, a residual connection, and a normalization layer, We also deploy multi-headed FAME to attend to each decoder token with the encoded input tokens h(r,a)) For each decoder input position j, we generate a hidden representation h (fdee) RRVV, representing the output token (j+ + 1)th position; [V] is the vocabulary size of the decoder.
3.2.4 The Alignment Module
The final layer of PARADOX is an alignment module that learns the latent alignment matrix and re-aligns the outputs generated by the decoder. The primary objective behind using alignments in generative models is explicitly learning the global semantic similarity between different tokens. We use two projection matrices, we and W to project the decoder token embedding matrix Eme( into two different subspaces. The alignment matrix is defined as,
QQK"") vd
A softmax
(5)
where Q Emb(dde). . WQ, K = Embldee). W and d is the hidden size of the decoder. This operation resembles the scaled dot-product attention mechanism (Vaswani et al. 2017). However, as opposed to atten- tion, we compute the global context by considering the original embedding space of all tokens. Finally, the re-aligned hidden representation is derived as,
h(gww
(dee)
= 11 (((ee))
+ 1
(dec) (G.a)
(6)
This hidden representation finally fed to a softmax layer to convert the outputs into probabilities. It is important to notice that. the alignment matrix A not intended to capture the semantic similarity ween contextual tokens but rather learn their similaritics at a global level. By doing so, the alignment module can capture the associations between all the tokens and recalibrate their probabilities during decoding, Moreover, the dot- product is computed over the embedding matrix, the autoregressive rule of text generation is not violated here.
For the sake of simplicity, we denote the combination of text encoder and persona encoder as 'encoder' and the combination of Transformer decoder and the alignment module as 'decoder' throughout this paper. The generative model is trained w.r.t. the decoder reconstruction cross-entropy loss. The contextual persona encoder gives rise to a variational KL-divergence loss. We use variational hyperparameter À assign its weightage in the final computed loss.
6

---

Published in Transactions on Machine Learning Research (10/2024)
Algorithm 1: Code-Mixed Text Generation with PARADOX
Require: Trained model M (enc, dec), user id u, historical utterance Zu, prompt word {w1}, decoder
vocabulary V
Require: mar length
L [[CLS], W1);
ti 00;; 0;
= m:
while v + [SEP] and i < mox length do M(s,,u) enc(Fa. u);
P,+1 dec( I, h(um))); 1 arg maxy Pi-1i Lt LU L end Return
3.3 Training Curricula
To learn the model parameters, primarily minimize the reconstruction loss (y1, 22, ... Vm) between defined as,
output sequence
C (19.0)
Σ Wy. Log(Poun (MGr,)) Y (o:) Xim, u))
The output sequence initialized with yo [CLS] token. The contextual persona encoder module arises Kullback Leibler divergence loss between the variational distribution and true posterior distribution, which can be derived (Kingma & Welling 2014 to
d Σ (1+2 log(ot) (pt)² - (o.)²)
(a
(0)
(8)
2
k=1
During training, minimize the task-spocific loss
Li (2,1)) + A L2
(9)
for each text and user id pair (x, u) ~ on training data. The persona encoding weight À a hyperparameter we before running the experiments.
3.4 Code-Mixed Generation
We adopt an autoregressive generation technique to generate new code-mixed texts for different users. To encode the user's historical persona, we use the user's last utterance (comment/tweet) in the encoder. As PARADOX is trained autoregressively on all historical utterances for different users, can capture the entire conversational history of user from the last utterance without passing the entire history during generation. A typical text generation model starts decoding with placeholder [CLS] token. Instead, we pass additional seed word input to the decoder for more guided generation. We formally report the text generation process in Algorithm j
4 Experimental Setup
This section elaborates on the experimental setup we adopt to evaluate our model and baselines on person- alized code-mixed generation.

---

Published in Transactions on Machine Learning Research (10/2024)
4.1 Datasets
To the best of our knowledge, existing longitudinal dataset is available for Hindi-Engline code-mixed. A longitudinal dataset is required to study the temporal evolution of language. Although some datasets in the literature consist of Hindi-English code-mixed texts collected from various online social networks, none of them contain user-specific information, making them unsuitable for our study. To overcome this, we collected code-mixed texts from the two most popular mediums where Indians are engaged Twitter and YouTube. From Twitter, we collected over 0.8 million in tweets starting from the year 2011 till date, from which we filtered only tweets originating from Mumbai and Delhi metropolitan regions, two cities with the largest Hindi population. We used Twitter API for academic research with full archival access] Further, for relevance, restricted ourselves to tweets related to 'Cricket', 'Bollywood', 'Polities', and 'Government Starting at 2014, Twitter automatically tags the language of tweet. We selected tweets with only non-empty language tags. This gives us a total of 226, 480 tweets from 19, 782 users.
From YouTuub, we chose two channels NishaMadhulike 23 (a popular chef based out of India with more than 12.7 million followers), and T-Serie (a popular Hindi music record channel started in 1983 having more than 200 million followers). We selected 42 videos from the NishaMadhulika channel and 69 from the T-Series that were first posted in 2011. We scraped all comments corresponding to these videos, accounting for 144, 822 comments from 99, 998 users.
For both datasets, we use a pre-trained language model open-sourced with Huggingfacd( that was fine-tuned on Hindi-Englisi parts-of-speech (PoS) and language identification (LID) tasks. Using this model, we label each loken each lext. with the corresponding language (Hindi or English) and their associated PoS. We tag a text as code-mixed only when the text contains least one Hindi verb written in either Devanagari or Roman script. We select users who have at least three utterances in their entire timeline. Finally, we are left with 18, 126 tweets (from 2, 241 users) and 9,7 YouTube comments (from 1, 1,9 users)
We remove all the HTML tags, URLs, emoticons, user mentions (starting with @@), and hashtags (starting with #)) For simplicity, we remove all numeric values from texts, as well. Finally, we convert all texts to lowercase. We highlight the key statistics of the datasets in Table 2 We use a 75-25 split for training and validation with stratified sampling. Therefore, we can ensure at least one training and validation sample for each user. We choose the first word for each validation sample as the seed word for generating the code- mixed text. Finally, the generated code-mixed text is evaluated against the original validation text using the proposed personalization evaluation metrics.
4.2 Evaluation Metrics
We adopt intrinsic and extrinsic evaluation metrics to evaluate our model in terms of semantic under- standing of code-mixing language and the ability personify codc-mixing for different users.
Datasel
#Texts
#Users
Mean text length
Mean CMI
Twitter
18126
2241
21.77
0.41
YouTube
8957
1349
28.89
0.36
Table 2: Dataset statistics, with mean text CMI and the average text lengths, demonstrating the extent of guage model, compared against ground truth. We cal- code-mixing.
For the intrinsic evaluation, we use perplexity, a metric that measures the predictive power of a lan-
culate perplexity as closs with loss being the cross-entropy reconstruction loss on the validation data. A lower perplexity score indicates better reconstructibility and ability to learn the semantics of a generative model.
Unlike Gupta et al. 2020)) ; we do not have any labelled gold data for evaluating our generative model. Therefore, traditional supervised evaluation metrics BLEU (Papineni et al., 2002). Rouge [Li 2004 can not be used directly to evaluate the personification aspects of code-mixed generation models. Similarly, other extrinsic evaluation measures such as Multilingual index (M Index) [Barnett et al. 2000). Burstiness and
http://www.oottbbb.o///////aaaaa
http:://www.yuutube.cooo hwt.p/yyotttbbb.ccmoaaaaaagll
8

---

Published in Transactions on Machine Learning Research (10/2024)
Span Entropy Guzmán et al. 2017 can not be used, as these metrics are predominantly used to evaluate the ability to capture corpus- level switching patterns of generative models. To overoome the limitations of the existing evaluation metrics, we propose four metrics for benchmarking generated code-mixed texts against the historical utterances by different users. These proposed metrics calculate the similarity between the linguistic patterns of modellgenerated texts and the user's historical utterances. We devise CM BLEU by calculating the BLEU score between the candidate and reference language sequences. For example- consider a candidate code-mixed text "mujke park janaa hai" ("I want to go to the park") and a reference text "mujhe rice aur curry khana hai" ("I want to eat rice and curry"). ). Using the LID model, can extract the corresponding language sequences (Hi, Hi, Hi, Hi] and (Hi, En, Hi, En, Hi, Hi, Hi, Hi, Hi, Hi, Hi from the candidate and reference texts, respectively (here, Hi and En stand for Hindi and English, respectively). Therefore, considering only the unigram and bigram overlaps between the candidate and the reference, we calculate the CM BLEU score of 0.606, If we use a different reference text "I don't want your nautanki" (Translation "I don't want your gimmick") with the corresponding language sequence (En, En, En, En, Hi), the CM BLEU reduces to 0.218. The proposed metric could calculate the similarity between the switching patterns demonstrated in the candidate text and the historical references by calculating the overlap between the language sequences. Similarly, we compute CM Rouge-1 and CM Rouge-L by computing Rouge-1 and Ronge-L scores between the candidate and reference language sequences.
Additionally, we leverage the user-level historical CMT to evaluate the linguistic patterns of the generated texts. If a user historically prefers monolingualism over multilingualism, we want the generative model to learn the pattern and generate texts with a lower CMI value for the user. Towards this, we propose CM KS, a metric that computes the Kolmogoroo-Smirnox distance between the generated and original CMT distribution of users. We highlight the relationships between those metrics by calculating the Pearson correlation between these measures, reported in Figure 6 of Appendix A.I Strong negative correlations between perplexily and CM BLEU, CM Rouge-1, and CM Rouge-L indicate that understanding semantics is essential to personify and replicate the switching patterns. Therefore, by learning semantics well, the generative models can learn the code-mixing patterns for different users and generate texts that imitate users' linguistic patterns. On the other hand, the correlations between CM KS and other metrics are weak, indicating that the linguistic preferences of users have no apparent linear relationships with switching patterns. These four metrics capture different aspects of the personalized usage of code-mixing for different users, as introduced in Section 3
4.3 Baseline Methods
We consider several codo-mixed gencration models for comparative evaluation.
D VACS (Samanta et al., 2019) is a VAE-based encoder-decoder model, primarily developed for generating HliniiEEgglin synthetic code mixed texts.
D GCM Rizvi et al. 2021 toolkit uses several linguistic theories and heuristics to generate code-mixed
texts.
D> CM-XLM (Gupta et al. 2020 is a generative model that utilizes pre-trained multilingual language model
XLM to generate Hindi-Engee English code-mixed texts from parallel corpora.
These code-mixed generation models consider monolingual reference data for generating code-mixed texts and generate code-mixed texts at corpus level. Therefore, compare these baselines only in intrinsic evaluation. We also utilize several dilfatttention-basee encoder-decoder and pre-trained language models to evaluate the personification aspects of generative models.
2017 is an encoder-decoder architecture utilizing self-attention mechanism that has shown superior performances in generation tasks like machine translation.
D Transformer /Vaswani et al.
>> MURIL (Khanuja et al. 2021 is encoder-based language model pre-trained on 17 Indian languages with a masked language modeling objective.
Can be calculated and validated using http://ppw..wwwatttt...o///attta waw.eaaattteeeecrreee.e.... seeu_score.htar

---

Published in Transactions on Machine Learning Research (10/2024)
(Muennighoff et al. 2022) is a family of large language model based multilingua BLOOM (Workshop et al. 2023) that was fine-tuned with multitask prompting. We use the 3B param- eter BLOOMZ model as our baseline.
BLOOMZ
2023) a family of autoregressive large language models trained with reinforce- ment learning with human leedback (RLHF). We adopt the 13B parameter instruction-tuned model as one of our baselines.
& Llama 2 [Touyron et al.,
D GPT-4 (Achiam et al. 2023 is a large multimodal model, accepting image and textual data and generating text outputs through autoregressive generation.
These pre-trained language models are only utilized in the extrinsic evaluation. We provide the hyperparam eter settings for PARADOX and all the baseline methods in Appendix A.2
5 Comparative Analysis
In this section, we report the performances of PARADOX and the non-persona-bassed code-mixed generation models in terms of the intrinsic and extrinsic evaluation measures.
We report the intrinsic evaluation sults in Table g PARADOX achieves 56% better perplexity on the Twitter datasct than the vanilla Transformer. On the YouTube dataset, the margin is even higher (59%). lower validation perplex- ity shows PARADOX's strong ability un- derstand code-mixing semantics and gen- erate texts of different linguistic varia- tions. PARADOX achieves 18% better per- plexity on the Twitter dataset than the best non-transformer baseline VACS. On the YouTube dataset, the margin is signif- icantly higher (64%).
Model
Perplexity +
Twitter
YouTube
GCM*
4331.85
4323.15
CM-XLM*
5413.22
1603.59
VACS
361.05
552.35
Transformer
680.07
473.84
PARADOX
297.43
196.21
(-) Contextual Persona
320.79
295.26
Speaker ID
582.24
371.70
Alignment
377.63
294.83
FAME
864.98
583.09
Table 4 highlights the extrinsic measures
across all the generative models. On the Table 3: Intrinsic evaluation of the competing models based on Twitter dataset, PARADOX achieves 2.4% perplexity (L: lower value indicates better performance). For mod- better CM BLEU than the Transformer cls highlighted with perplexity is calculated with word-level model. Similarly, on the YouTube dataset, generation. Bold indicates the best results among all the mod- PARADOX performs the best among all els. the baselines and outperforms the Trans-
former model with margin of 2.0%. In terms of the Rouge measures, PARADOX performs consistently better than the non-persona counterpart with an average margin of 1.9%. In terms of distance-based measures, PARADOX performs significantly better than the Transformer model on both datasets. Overall, PARADOX achieves 11% lower CM KS distance than Transformer. Lower KS distance indicates the importance of utilizing user persona in generating user-specific code-mixed texts.
Among the pre-trained language models, fine-tuned Llama and GPT-4 are the most competitive. Interest- ingly, even with single example in the prompt (1-shot), CM BLEU increases by 9.8% for Llama Similar performance improvements are also observed with other extrinsic metrics. However, both Transformer and PARADOX perform significantly better than the pre-trained language models in terms of personalized code- mixed text gencration. On average, PARADOX achieves 7.6% better CM BLEU and 12.5% better CM Rouge-1 than Llama 2. PARADOX even achieves 3.9% better CM BLEU than the fine-tuned Llama model. Similar performance improvements are observed with CM Rouge-1 and CM Rouge-L metrics. PARADOX achieves 4.1% better CM Rougo-1 and 3.8% better CM Rougo-L than the GPT-4 model. Even with CM KS, our model outperforms GPT-4 with a wide margin of 14%. Interestingly, with few-shot in-context learning, GPT-4 model demonstrates stronger performance than the existing baselines. On Twitter dataset, 1-shot
10

---

Published in Transactions on Machine Learning Research (10/2024)
Model
CM BLEU 1
CM Rouge-1
CM Rouge-L 1
CM KS
Twitter
YouTube
Twitter
YouTube
Twitter
YouTube
Twitter
YouTube
MuRII.
9.92
9.85
26.93
21.10
23.65
19.63
0.42
0.23
BLOOMZ
14.20
23.87
49.22
56.61
45.93
55.09
0.40
0.30
Llama 2 (zero-shot)
19.97
7.25
48.91
30.86
43.74
28.72
0.56
0.43
Llama 2 (1-shot)
26.69
20.03
55.17
46.08
49.57
43.17
0.50
0.39
Llama 2 (fine-tnnd))
21.97
26.09
55.89
58.24
51.07
55.17
0.40
0.34
GPT-4 (zero-shot)
30.94
30.33
57.46
57.88
50.89
53.69
0.42
0.39
GPT-4 (1-shot)
31.90
30.57
60.67
61.46
54.55
57.34
0.42
0.36
Transformer
22.21
29.36
58.69
61.02
51.10
57.27
0.42
0.37
PARADOX
24.58
31.37
60.60
62.90
53.03
59.16
0.36
0.34
(-) Contextual Persona
24.06
30.67
60.03
62.44
52.52
58.71
0.35
0.34
(-) Speaker ID
20.10
31.01
56.46
62.72
49.71
58.96
0.35
0.34
()) Alignment
24.37
31.08
60.79
62.46
53.
59.15
0.32
0.37
(-) FAME
18.49
28.29
54.01
60.67
47.68
57.10
0.36
0.37
Table Extrinsic evaluation of pre-trained language models, Transformer and PARADOX terms preserving user-level switching patterns (f (resp. 4): higher (resp. lower) value indicates better performance). Bold indicates the best results among all the models.
GPT-4 model achieves higher CM BLEU and CM Rouge scores than PARADOX. This superior performance can be justified with the facts that tweets are generally shorter and more monolingual (lower CMI), there- fore, capturing the corpus-lovel trends are casier for robust LLMs like GPT-4 with even a single example in the prompt. However, it worth noting that 1-shot GPT-4 still underperforms than PARADOX in terms of the personalization metric CM KS. On YouTube dataset however, PARADOX achieves 0.8% higher CM BLEU and 1.5% higher CM Rouge scores than the 1-shot GPT-4 model demonstrating its superiority in understanding and generating personalized code-mixed texts.
Our ablation study in Table B shows the effectiveness fused attention, contextual persona module, and the alignment mod- empir-
CM Rouge-1
CM BLEU
27.5
ule in PARADOX. Additionally, ically highlight the necessity of speaker ID encoding for personalized code-mixed text generation. Adding FAME improves validation perplexity by 66%. Similarly, the conlextual user persona. and align- ment modules improve validation perplex- ity by 21% and 28%, respectively, justify- ing their contributions to modelling low- resource language. Modeling the user per- sona with an additional identifier (speaker ID) improves the validation perplexity by 48%, highlighting the importance of the module for personalized text generation. Ablation results in Table also suggest the importance of different modules of PARADOX for personalized code-mixed text generation. Removing the contextual per- sona encoder, leads lo an average performance drop by 0.52% in regards to CM BLEU and CM Rouge scores. Moreover, an one-sided t-test concludes the statistical significance (p-value < 0.001) of the performance drop across different extrinsic measures and datascts. Figure 3 highlights the distribution of intrinsic and extrinsic metrics for linear and randomized speaker contextualization modules. Not only does having randomized speaker contextual module improve the performance, but it also improves the robustness of the generation
25.0
22.5
20.0
700
CM Rouge-L
57.5
Perplexity
600
55.0
500
52.5
400
50.0
Uneer
Randomized
Randomize zed Speaker Contextual Module
Linear
Speaker Contextual Module
YouTube
Twitter
Figure 3: Performances of PARADOX under lincar and randomized persona encoder.
11

---

Published in Transactions on Machine Learning Research (10/2024)
model. Therefore, argue that random exploration of user persona helps model in capturing the vari- ability in linguistic pat perterns demonstrated by users in different contexts, enabling it to imitate the linguistic patterns during generation.
5.1 Human Evaluation
We perform a human evaluation study to evalu- ate the code-mixed texts generated by PARADOX and the vanilla Transformer. In the previous sec- tion, we discussed our methodology for evaluat- ing the personification capabilities of PARADOX using the intrinsic and proposed extrinsic met- rics. With human evaluation, we aim to assess the generative models in terms of linguistic correctness and contextual relevance. Towards that, we randomly sample 24 examples from each of these models and ask 30 human evaluators to rate these examples based on Semantic coherence and Linguistic quality. Semantic coherence measures the meaningfulness of the code-mixed texts, whereas, with linguistic quality, we measure their structural validity. Both the scores ranged between 1-5, being the lowest, and 5 boing the highest.
Model
Semantic Coherence 1 Linguistic Quality 1
Transformer PARADOX
2.34
2.32
3.08
3.00
Table 5: Human evaluation the models.
Table presents the average semantic coherence and linguistic quality scores, along with Fleiss's Kappa Fleiss 1971 scores among the annotators. We observe that PARADOX displays better semantic coherence (32% better), well as better linguistic quality (29% better) than the Transformer model. We observe lair agreement (Kappa 0.13 for semantic coherence and Kappa 0.14 for linguistic quality) among the annotators for both models.
6 Analysis of Code-Mixed Generation
We further study the quality of code- mixed generation and compare them against other baselines. We analyze the distribution of length and CMT of texts generated by different generative mod- els and report in Figure 4 Trained on the Twitter dataset, PARADOX generates texts with median length of 16, 25% higher than the other generalive baselines. A similar trend can also be observed in the YouTube datasct. Similarly, the me-
Twitter
114
YouTube
IDO
dian value of CMI on lexts generated by PARADOX is 0.27 and 0.16, respectively, for the Twitter and YouTTue datasets, which are significantly lower than the median CMT achieved by other baselines (0.29 and 0.28, respectively). A lower median CMI indicates that the texts generated by PARADOX are more monolingual at a corpus level, acknowledging the population level trend shown in Figure
10
0.00
0.75
1.00
Length
Real GCM
CM-XLM
Transformer PARADOX
VACS
Figure Comparison of different text generation models.
Figure 5(a) shows the distribution of top Hindi verbs and nouns from the Twitter dataset. Being more inclined towards monolingual, PARADOX assigns more probability to these Hindi words, irrespective of the parts of speech. Figure 5 b) shows the distribution of top Hindi verbs and nouns from the Twitter dataset for different ablations of PARADOX. Interestingly, PARADOX with the alignment module can replicate the word distribution observed in the real dataset. On the other hand, without the alignment module, the generative model could hallucinate and unrealistically use common phrases in incorrect contexts. This highlights the effectiveness of the alignment module in recalibrating output tokens and generating semantically meaningful texts. Although the dimension of the alignment matrix [V| [V] with [V] boing the decoder vocabulary size, the learnable
6 Evaluators are proficient with Hindi- Ennglih code-mixed language their age ranges in 21
12

---

Published in Transactions on Machine Learning Research (10/2024)
Probabiing itt
0.02
Probalo lity
0.01
0.01
0.00
nah
kar
salman bueedd
poliucs srk Top Hindi nouns
0.00
the
ko
ka
bWood
politic
modi
Top Hindii verbs
Top Hindi verbs
Top Hindi nouns
Reall
VACS
CM-XLLM
Real
Transformer
PARADOK
PARADOX
Alignment
(b)
Figure 5: Distribution top Hindi verbs and nouns for Tweets, w.o. aligument the ablation of PARADOX without the alignment module.
parameters are of order Odd), where d the hidden dimension. The total number of additional parameters introduced by the alignment module is 0.025% the entire network, which is insignificant compared other modules.
We further analyze few sample texts generated by the generative models. We report examples of texts generated by PARADOX and the Transformer, along with the average semantic coherence and linguistic quality scores annotated by the annotators in Table 7 of Appendix B Transformer usually picks top nouns in the corpus and generates texts around them without considering the syntax of the code-mixed language, resulting incoherent texts in many cases. On the other hand, PARADOX preserves the grammar of code- mixed texts with a more human-like switching pattern. shows that PARADOX maintains the grammar of the base language (Hindi in this case), attributing to highlight the key differences between the texts generated by persona-based PARADOX and non-persona-basee Transformer models regarding text quality. Table 6 further shows the personalized generation by our model. With the same prompt (e.g., 'salman" in the first example), PARADOX can understand different personas of users and can generate texts suited for different users. The high similarity between the historical average of the user CMIs and the generated CMTs indicates the model's ability to understand the linguistic preferences of users.
coherent and reliable generation. The examples
Table S of Appendix B highlights code-mixed texts generated by the Llama 2 model with zero-shot and 1- shot. PARADOX exhibits better capability in mimicking the code-mixing linguistic traits than Llama. For user ID 2226, who has had more monolingual usage in the past (average CMI 0.04), the text generated by Llama is more code-mixed than monolingual. Similarly, for user ID 3, the Llama model reverses the linguistic preference of the user while generating the code-mixed texts. Albeit demonstrating superior performance across various natural language understanding and reasoning tasks with zero and few-shot in-context learning, pre-trained large language models such as Llama fail to understand the linguistic complexities of informal languages such code-mixed Hinglish. It is imperative to notice that the Llama model not only fails to capture the historical linguistic preferences of users but also fails to impersonate the semantic structure of code-mixed lexts. On the other hand, PARADOX demonstrates better code-mixed language understanding capabilities, captures the linguistic preferences the user from their historical utterances and preserves the information for future generations. This highlights the effectiveness personification code-mixed text generation and the importance of building more robust language understanding models for understanding the linguistic nitty-gritty of low-resource languages.
13

---

Published in Transactions on Machine Learning Research (10/2024)
User ID
Generated Text
Historical Avg. CMI
Generated CMI
CM: salman ji please aapse milna hai
0.28
0.33
Eng: Salman, want to meet you, please
CM: salman ka fan ho na.
264
0.50
0.40
Eng: You are Salman's s fan, right?
CM: politics kr raha kya kr rahi h or truth show
762
0.15
0.30
Eng: Is this politics, or truth show
CM: ye fb ho jaye bhaijaan
2226
0.04
0.00
Eng: This has became fb, brother
Table Example of different prompted (prompts highlighted with blue) generation for different users. Different CMI indicates the difference in prompted generation based on the user persona.
7 Conclusion
This paper described a personalized code-mixed generation model for generating human-alike code-mixed texts. We highlighted the need for personalized generation under the pretext of rode-mixing. Toward this, we devised a novel persona-aware encoder-decoder model coupled with novel alignment module for generating more realistic and coherent Hindi-English code-mixed texts, the first attempt toward personalized code- mixed generation. Empirical analyses would benefit. the research community in developing robust and reliable language models for low-resource languages. Although our current work explored PARADOX for Hindi- Englis gencration, the proposed generative framework can be further extended to generate code-mixed texts other personalized code-mixing and derived languages, such Apannsh--Englisl and Chinese- Engiess....lls..... Although our empirical study has shown the effectiveness of only contextual persona, ignoring other explicit factors. Not only does this restrict model in cold-start generation (text generation for new users without any history), but it also fails to consider the co-association among users. In conversational settings, particularly, this can be deemed essential. Another limitation of PARADOX is the inability to determine the temporal evolution of a user's persona driven by external factors. PARADOX captures the user persona and its evolution solely from contextual information. At the same time, user's linguistic preferences can also be driven by other external socio-demographie and economic factors varying over time, which our model currently undermincs.
text generation, currently PARADOX captures
Broader Impact Statement
Our work highlights the need for personalized generation models for conversational languages like code- mixing. We release our curated datasets to encourage research on personalized code-mixed text generation. Persona-aware code-mixed generation models aid in building data-driven solutions in low-resource lan- guages and can be expanded to broader demographics. We do not collect user-specific features determinants that could reveal user-specific sensitive information. We remove all the user IDs, mentions and delimiters from the user texts to maintain user privacy. Moreover, no user-specific attributes are used to train the generative model except the texts written by the users. Although we do not anticipate any immodiate nog- alive impact of our work, over-personallzationn can lead to largeted spamming and negative misuse of user persona. We ask the researchers to be aware of the potential misuse and use the shared artefacts judiciously to prevent unwarranted events.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tgge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprinl ar STf::2333.88774 2023.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Dhiraj Amin, Sharvari Govilkar, Sagar Kulkarni, Yash Shashikant Lalit, Arshi Ajaz Khwaja, Daries Xavier, and Sahil Girijashankar Gupta. Marathi-english code-mixed text generation. ar preprinl arXiv:2309. 16202, 2023.
Srijan Bansal, Vishal Garimella, Ayush Suhane, Jasabanta Patro, and Animesh Mukherjee. Code-switching patterns can be an effective route to improve performance of downstream NLP applications: A case study of humour, sarcasm and hate speech detection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1018-1023, Online, July 2020. Association for Computational Linguis- lics. doi: 10. 18653/v5 / 22l2000000c2mmainnn
URL http:: aclweb. org/antholloy/20200 acl-0aaalmmmaaan 96
Ruthanna Barnett, Eva Codó, Eva Eppler, Montse Forcadell, Penelope Gardner-Chloros, Roeland van Hout, Melissa Moyer, Maria Carme Torras, Maria Teresa Turell, Mark Sebba, Marianne Starren, and Sietse Wensing. The lides coding manual: A document for preparing and analyzing language inter- action data version 1.1- 1.1-july, 1999. International Journal of Bélingualism, 4(2):131-132, 2000. doi: URL https:///di..oo org/ 10. 1177 00.117//33700000000000000
10.117/116700990000000000
Khyathi Chandu, Shrimai Prabhumoye, Ruslan Salakhutdinov, and Alan W Black. "My Way of Telling a Story": Persona based Grounded Story Generation. In Proceedings of the Second Workshop Storytelling,
pp. 11 21, Florence, Italy, 2019. Association for Computational Linguisties. doi: 10. / URL https:/ http:::/wwww..ee www.wwlwwb.....w..wwwwwwwwwww..w org/ wwtg//anthooogy/W11191
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint ar Kiv:22200. 1e:22200.1.4. 11416, 2022.
Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. Advances in neural information processing systems, 32, 2019.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaune Wenzek, Francisco Guzmán, Edouard Grave, Myle Out, Luke Zeiteemoyrr, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pp. 8440-8451, 2020.
Amitava Das and Björn Gambäck. Identifying languages at the word level in code-mixed Indian social media text. Tn Proceedings of the 1 11th International Conference on Natural Language Processing, pp. 378 -37, Goa, India, December 2014. NLP Association of India. URL
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirec- tional Transformers for Language Understanding, May 2019. URL http://arxiv. hrg. abgg//////1/...////////////// 04805. Number: arXiv:1810.0450 arXiv:1810.04800 [cs].
Suman Dowlagar and Radhika Mamidi. Gated convolutional sequence to sequence based learning for english- hingilsh code-switched machine translation. In Proceedings of the Fifth Workshop on Computational Ap- proaches Linguistic. Code-Swiitching, pp. 26-30, 2021.
Joseph L Fleiss. Mcasuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378, 1971.
Saurabh Garg, Tanmay Parekh, and Preethi Jyothi. Code-switched Language Models Using Dual RNNs and Same-Source Pretraining. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3088- Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10. 10.116533v1/D18. 1346. URL
Devansh Gautam, Prashant Kodali, Kshitij Gupta, Anmol Goel, Manish Shrivastava, and Ponnurangam Kumaraguru. Comet: Towards code-mixed translation using parallel monolingual sentences. In Proceedings of the Fifth Workshop on Compulational Approaches to Lânguistic Code-Swiilching, pp. 47 47-55, 55, 2021.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commumications of the A ACM, 63(11): 139 144, 2020.
Deepak Gupta, Asif Ekbal, and Pushpak Bhattacharyya. A Semi-supervised Approach to Generate the Code- Mixed Text using Pre-trained Encoder and Transfer Learning. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pp. 2267 2280, Online, 2020. Association for Computational Lin- guistics. doi: 10. 18633/61 /tt./222.....nnn-eeeee madingg--emnl..22 206
URL https: //www acwweb. orgg/nthologgy/022
Gualberto Guzmán, Joseph Ricard, Jacqueline Serigos, Barbara E. Bullock, and Almeida Jacqueline Toríbio. Metrics for Modeling Code-Switching Across Corpora. Tn Interspeech 2017, pp. 67-71. ISCA, August 2017. doi: 10.21437/ http://tterseech221---1-1 interspeech 2017 guzman17m interspeech. html
URL https://www.iscaaappa hwwp:hch...ggrracieee....... archive/
Ganesh Jawahar, El Moatez Billah Nagoudi, Muhammad Abdul-Mageed, and Laks VS Lakshmanan. Ex- ploring text-to-text transformers for english to hinglish machine translation with synthetic code-mixing, arXiv preprint Xrv:2 105 08807, 2021.
Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha Talukdar. MuRIL: Multilingual Representations for Indian Languages, April 2021. URL
10730 Number: arXiv:2103.1073) a1X:v:2103.10330 [cs].
Milton King and Paul Cook. Evaluating approaches to personalizing language models. In Nicoletta. Calzo- lari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfth Language Resowrces and Evaluation Conference, pp. 2461-2469, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546--444. URL https://aclantool. wrwg:orgg//0000 2rg//200.1rec1111....
Diederik P. Kingma and Max Welling. Auto- EntooEEnnddig Variational Bayes, May 2014. URL http:://taxt Number: arXiv:1312.6114 arXiv:1312.611 [cs, stat].
hog/abss13312 1112. 6114
Hung Le, Truyen Tran, and Svetha Venkatesh. Self-attentive associative memory. In International Conference on Machine Learning, pp. 5682 5691. PMLR, 2020.
Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Lá, Sergey Edunov, Marjan Ghavvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Compulational Linguistics, 8:726 742, 2020.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. arXiv preprini ar Xix:221 1.01 786, 2022.
Long Ouyang, Jeffrey WU, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Newral Information Processing Systems, 35:27730 15-27730-27744, 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311 318, 2002.
Rana D. Parshad, Suman Bhowmick, Vineeta Chand, Nitu Kumari, and Neha Sinha. What is India speaking? Exploring the "Hinglish" invasion. Physica A: Statistical Mechanics and its Applications, 449:375 380, May 2016. ISSN 03784371. doi (0.1016/j..hyss.20106 retrraeree/p1///0077781116000
URL http:::/iinnnngguubbbtt elsevie..r com/
16

---

Published in Transactions on Machine Learning Research (10/2024)
Adithya Pratapa, Gayatri Bhat, Monojit Choudhury, Sunayana Sitaram, Sandipan Dandapat, and Kalika Bali. Language Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic Data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. -553, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10.18653/ (0053/11/P18////
URL http://aclweb. http://wwwbbborrr/aae.ee wrggaanbhologgy/18-111
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Nya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Maria Reichelt, Frauke Kämmerer, Helmut M Niegemann, and Stelfi Zander. Talk to me personally: Per- sonalization of language style in computer-bar based learning. Computers in Human behavior, 35:199 210, 2014.
Mohd Sanad Zaki Rizvi, Anirudh Srinivasan, Tanuja Gamu, Monojit Choudhury, and Sunayana Sitaram. GCM: A Toolkit for Generating Synthetic, Code-mixed Text. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lònguistics: System Demonstrations, pp. 205 211, Online, 2021. Association for Computational Linguistics. doi: 10.18653 2006653/v/22000 http:/222..callooo https://acllanthoo.og ww..orgg///00 20tg/ eacl demos. 24
URL
Koustav Rudra, Shruti Rijhwani, Rafiya Begum, Kalika Bali, Monojit Choudhury, and Niloy Ganguly. Understanding Language Preference for Expression of Opinion and Sentiment: What do Hindi-English Speakers do on Twitter? In Proceedings of the 2016 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 1131-1141, Austin, Texas, 2016. Association for Computational Linguistics. doi: (0.18653/v1/DIE4 1121. URL http: http:///acaebb..oogaaooooooo////
Bidisha Samanta, Sharmila Reddy, Hussain Jagirdar, Niloy Ganguly, and Soumen Chakrabarti. A Deep Generative Model for Code Switched Text. In Proceedings of the Soventy-Eights International Joint Con- ference on Artificial Intelligence, pp. 5175 5181, Macao, China, August 2019. International Joint Confer- ences on Artificial Intelligence Organization. ISBN 978-0-9992441-4411 doi: 10.24963 (0.24963//jeai.2000 1jeai.2019/719. URL https://www. htt.:jaaaiiiprceeeeeeeeeeeeeeeee ort/pproceeddnng////1//99999
Sebastin Santy, Anirudh Srinivasan, and Monojit Choudhury. Bertologicomix: How does codo-mixing interact with multilingual bert? In Proceedings of the Second Workshop on Domain Adaplation for NLP, pp. 111 121, 2021.
Ayan Sengupta, Sourabh Kumar Bhattacharjee, Tanmoy Chakraborty, and Md Shad Akhtar. Hit-a hierar- chically fused deep attention network for robust code-mixed language representation. In Findings of the Association for Compulational Linguislies: ACL-IJCNLP 2021, pp. 4625-400 0555-463, 2021.
Ayan Sengupta, Sourabh Kumar Bhattacharjee, Md. Shad Akhtar, and Tanmoy Chakraborty. Does ag- gression lead to hate? Detecting and reasoning offensive traits in hinglish code-mixed texts. Neu- rocomputing, 488:598-617, June 2022a. ISSN 0925-2311 doi: 10.
URL
Ayan Sengupta, Tharun Suresh, Md Shad Akhtar, and Tanmoy Chakraborty, A Comprehensive Un- derstanding of Code-mixed Language Semantics using Hierarchical Transformer, April 2022b. URL http://arxiv.ott bogg/abs//e 2204. 12753 Number: arXiv:2204.1221 arXiv:2204.12553 [cs].
Ayan Sengupta, Soham Das, Md Shad Akhtar, and Tanmoy Chakraborty. Social, economic, and demographic factors drive the emergence of hinglish code-mixing on social modia. Humanities and Social Sciences Communicaatinss, 11(1):1-12 2024.
Rico Semnrich, Barry Haddow, and Alexandra Birch. Neural Machine Translation of Rare Words with Sub- word Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1715-1725, Berlin, Germany, 2016. Association for Computational Linguis- tics. doi: 10. (0.18653/v1/P16-11 URL http:://acceeb.oor//tttooollo//p
17

---

Published in Transactions on Machine Learning Research (10/2024)
Kushagra Singh, Indira Sen, and Ponnurangam Kumaraguru. Language Identification and Named Entity Recognition in Hinglish Code Mixed Tweets. In Proceedings of ACI, 2018, Student Research Workshop, pp. 52 58, Melbourne, Australia, 2018a. Association for Computational Linguistics. doi: 10.18653/v1/ P18-3008. URL http://aclweb. org/ htt.c//ow...ora.aaaoooooo/////// aat/aaaoooo///--000000
Kushagra Singh, Indira Sen, and Ponmurangam Kumaraguru. A Twitter Corpus for Hindi-Englin Code Mixed POS Tagging. In Proceedings of the Sizth International Workshop on Natural Language Processing for Social Media, pp. 12-17, Melbourne, Australia, 2018b. Association for Computational Linguistics. doi: 11153//11/18-3151 URL httt:://acleeb.orr//anhhoooooood wttp:///aaoooog////------0000
Abhishek Srivastava, Kalika Bali, and Monojit Choudhury. Understanding script-mixing: A case study of hindi-english bilingual twitter users. In Proceedings of the The 4th Workshop on Computational Approaches to Code Switching, pp. 36-44, 2020.
Vivek Srivastava and Mayank Singh. Hinge: A dataset for generation and evaluation of code-mixed hinglish text. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pp. 200-208, 2021.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3645 3650, Florence, Italy, July 2019. Association for Computational Linguistics. doi: (0.18653/v1/ P19- 1355. URL
wwcellogg.org///o.. org/P19-135
Samson Tan and Shafiq Joty. Code-mixing on sesame street: Dawn of the adversarial polyglots. arXiv preprinl Xiv:2100 09593, 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Tukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedamuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosscini, Rui Hou, Hakan Tnan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanic Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Zihao Wang, Ming Jiang, and Junli Wang. A Speakor-aware Parallel Hierarchical Attentive Encoder-Decoder Model for Multi-turn Dialogue Generation, October 2021. URL http://arxiw..orgabbae 06823 Number: arXiv:2110.0682 arXiv:2110.0692 [cs].
Genta Indra Winata, Andrea Madotio, Chien-Sheng Wi, and Pascale Fung, Code-switched language models using neural based synthetic data from parallel sentences. arXiv preprint arXiv:1909.0880 2019.
BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas MuennighofT, Albert Villanova. del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMiilan-Majoor, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Fmezue, Christopher Klamm, Colin Leong, Daniel van Strien,
18

---

Published in Transactions on Machine Learning Research (10/2024)
David Ifcoluwa Adelani, Dragomir Radev, Eduardo González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, lan Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joy- deep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Saško, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilaa Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Tor- rent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Leperoq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut Emre Taşar, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea San- tilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj, Urmish Thakker, Vikas Raunak, Xiangru Tang, Zheng-Xin Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phang, Ofrr Press, Conglong Li, Deepak Narayanan, Hatím Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anasta- sia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurélic Névéol, Charles Lovering, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, Jan-Chhistoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna. Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bam- berger, Zdeněk Kasner, Alice Rucda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdollahi,, Aycha Tammour, Azadeh HajiHosseini, Baharch Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Daniel Mc- Duff, Danish Contractor, David Lansky, Davis David, Douwe Kiela, Duong A. Nguyen, Edward Tan, Emi Baylor, Ezinwanne Ozoani, Fatima Mirza, Frankline Ononiwu, Habib Rezanejad, Hessie Jones, Indrani Bhattacharya, Irene Solaiman, Trina Sedenko, Isar Nejadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akin- lolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Rajani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Viguier, Thanh Lc, Tobi Oyebade, Tricu Lc, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, An- tonio Miranda-
Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourieer, Daniel León Periñán, Daniel Molano, Dian Yu, Enrique Manjava- cas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Tmane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Marc Pàmies, Maria A Castillo, Marianna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cul- lan, Michael Weinberg, Michiel De Wolf, Mina Mihaljeic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee Lang-aroonssin Srishti Kumar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya
19

---

Published in Transactions on Machine Learning Research (10/2024)
Kainuma, Wojeiech Kusa, Yanis Labrak, Yash Shailesh Bajaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. Bloom: A (76b-parameter open-access multilingual language model, 2023.
Zheng Xin Yong, Ruochen Zhang, Jessica Forde, Skyler Wang, Arjun Subramonian, Holy Lovenia, Samuel Cahyawijaya, Genta Winata, Lintang Sutawika, Jan Christian Blaise Cruz, et al. Prompting multilingual large language models to generate code-mixed texts: The case of south cast asian languages. In Proceedings of the 6th Workshop Compalational Approuches to Linguislic Code-Swilching, pp. 43 63, 2023.
Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. Adversarial feature matching for lext generation. In International Conference on Machine Learning, pp. 4006 4015. PMLR, 2017.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William B Dolan. Dialogpt: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 270 278, 2020.
Yinhe Zheng, Rongsheng Zhang, Xiaoxi Mao, and Minlie Huang. A Pre-training Based Personalized Dialogue Generation Model with PPesona-ssaare Data, November 2019. URL/http://arxiv org//bs/111111 04700 Number: arXiv:1911.00 190.040000 arXiv:1911.04700 [cs].
A Experimental Setup
A.1 Evaluation Metrics
We compute Pearson correlation (c.f. Figure 6 between the proposed metrics to understand their rela- tionships. We observe strong positive correlations among the metrics CM BLEU, CM Rouge-1 and CM Rouge-L. Similarly, all of these metrics have a strong negative correlation with perplexity. This highlights the strong linear associations between the evaluation metrics that measure the semantical behavior of the code-switching patterns. On the other hand, the correlations between the CM KS measure and other metrics are meagre, indicating that the linguistic preferences of users have very minimal impact on the switching patterns. In other words, the switching patterns of a user who prefers multilingual could be more erratic than those of a user who prefers monolinguals.
Perplexity
CM BLEU
CM Rouge-1 CM Rouge-L
CM KS
Perplexity
1.0
-0.58
-0.78
-0.62
0.15
0.8
CM BLEU
0.6
-0.58
1
0.9
0.99
0.012
CM Rouge-1
-0.78
0.9
1
0.9
-0.062
CM Rouge-L
-0.2
-0.62
0.99
0.9
0.019
-0.4
CM
0.15
0.012
-0.062
0.019
1
-0.6
Figure 6: Pearson correlation between different evaluation measures the validation dataset.
20

---

Published in Transactions on Machine Learning Research (10/2024)


<<sYS>>
generate Hindi-English based user's previous message word. Make sure that understand the user's linguistic pattern from the previous message and generate text that starts with the seed word.
You helpful


Message: (user_messagee
Seed Word: (seed word) (/INST]
Figure 7: Prompt used with the Llama 2 model for Hindi- English code-mixed text generation.
A.2 Training Details
For all the models across all the experiments, we use a maximum text length of 40. PARADOX consists of six encoder and decoder layers, with hidden sizes of 768 in all the layers. For multi-headed FAME and masked multi-headed FAME blocks, we use total of eight. heads with Dropout probabilities set as 0.1. The total number of parameters is 296M. We use six encoder and six decoder layers in the Transformer model, with eight heads in each multi-headed attention block. For training PARADOX, We set the persona encoding variational weight À 0.5. All the models are trained for 50 epochs with early stopping condition validation loss with the patience of 10. We set validation. We use Adam optimizer with a learning rate of 4e - 4 and 81 = 0.9, B2 0.98 for both PARADOX and Transformer. We fine-tune the MuRIT, and BLOOMZ models on autoregressive language modeling tasks for 10 epochs with learning rates -5 and 3e - 6, respectively. The Llama-2 baseline used in zero-shot and 1-shot settings with the prompt shown Figure 7 We use one Tesla P100 and one Tesla V100 GPU run all our experiments. For PARADOX, each training and validation iteration takes 0.18 and 0.12 seconds, respectively. Strubell et al. (2019) proposed estimation of power usage and carbon emission behind running deep learning experiments. Following those guidelines, estimate total power usage of 23.56 kWh and an equivalent CO2 emission of 22.46 pounds.
size = 4 in all experiments during training and
B Analysis of Made-Mixed Generation
We highlight few examples of texts generated by PARADOX and the vanilla Transformer model in Table 7 along with their semantic coherence and linguistic quality scores annotated by the annotators. As highlighted in Table 5 the semantic coherence of the texts generated by PARADOX is much higher than the non-persona- based counterpart. Moreover, lexts generated by PARADOX are grammatically more valid than those generated by the Transformer model. This analysis highlights the importance of incorporating persona for an enhanced understanding of conversational code-mixed languages.
Table 8 highlights the texts generated by the persona- persona-based PARADOX and the Llama model for two different users. The margin between the historical average CMT and the generated CMI highlights our method's 's superiority in capturing users' linguistic patterns. A large pre-trained language model such as Llama, even with few-shot in-context learning, may not be superior to a persona- persona-based small language model fine-tuned on low-resource code-mixed texts.
21

---

Published in Transactions on Machine Learning Research (10/2024)
Model
Generated Text
Semantic Coherence
Linguistic Quality
CM: are bhai apne!
3.70
3.60
Eng: Hey my brother!
PARADOX
CM: bhai bahut thik ho gaya
4.27
4.27
Eng: brother is very well
CM: rockstar walo se samjha me-
3.07
3.57
dia........
Transformer
Eng: Media understood by rockstars
CM: boy likha hai thanks bhai
3.57
3.53
Eng: Boy written thanks brother
Table 7: Examples of code-mixed texts generated by PARADOX and Transformer with human annotated average Semantic coherence and linguistic quality scores.
User ID
Model
Generated Text
Historical Avg. CMI
Generated CMI
CM: salman please aapse milna hai
PARADOX
0.28
0.33
Eng: Salman, want to meet you, please
CM: Salman ne kaha tha flush me jeutega apka siddhant
Llama zero-shot
0.28
0.22
Eng:
Salman had said flushme jeetega apna prin- ciple
CM: Salman bhai koha, Flush mein jitega upna
Llama 1-shot
0.28
0.11
side, main toh bus nikal raha hoon!
Eng: Salman had said flushme jecloga apna side, am just leaving in bus
CM: ye fb ho Jaye bhaijaan
PARADOX
0..04
0.00
Eng: This has became fb, brother
CM: ye superb jabardast bahut khub, bhadaai ho
2226
Llama zero-shot
0.04
0.37
Eng: This superb, smazing, and congrstulations
CM: ye superb jabardast bahut khub, bhadaai ho
Tlama I-shot
0.04
0.37
Eng: This superb, amazing, and congratulations
Table 8: Example of texts generated by PARADOX and Llama models.
22


Reviews:
[{"review_id": "lYmua51ePU", "reviewer": "Reviewer_qd2s", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method shows effective results, performing better than GPT-4 in zero-shot scenario, Llama2 in zero-shot and one-shot scenarios and finetuned Llama2 in most cases. \n2. The proposed model has real-world usefulness.\n3. The writing of this paper is good.\n\nWeakness:\n1. The author mentioned \"We define the personalization of code mixing as a behavioral language that captures a user’s writing style given the historical trends, social context, targeted demographics, and topical relevance.\" However, in the proposed method, the authors used a randomly initialized user embedding to model the personalization. It is an unknown question that what the user embedding really learned finally. Is there any evaluation we could intuitively know this?\n\n2. The author mentioned \"The motivation behind projecting the persona embedding to a latent space is to capture the contextual perturbations in the user persona\". But in Table 3, after removing the contextual persona, the model performance has very minor difference with the original one. Did the contextual persona really capture the contextual perturbations?\n\n3. As shown in Table3, Llama2 performs significantly better from zero-shot to 1-shot. Then why didn't the author try 1-shot on GPT-4?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes PARADOX, which aims to generate personalized code-mixed texts. PARADOX consists of a encoder modeling both global and local persona, an alignment module to re-calibrate the generated sentence as well as an autoregressive decoder to decode a sequence. It was evaluated on Twitter and Youtube datasets, and showed promising results.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses above and revise the manuscript accordingly.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kP20iPYaCP", "reviewer": "Reviewer_Vres", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is clearly written and the proposed technique is sound.\n\nThis work makes a solid contribution by generating personalized code-mixed texts by leveraging users’ historical utterances. The research direction is timely and the motivation is reasonable.\n\nThe experiments are comprehensive and the results well indicate the effectiveness of the proposed approach.\n\n**Comments for the revised version**\n\nOverall, this paper has been improved after the revision. The logic flows more smoothly with more clear motivation of the incorporating user persona for code-mixed text generation, as well as how the technique is applied and evaluated. My concerns in the last round have been addressed.\n\nHowever, it remains unclear how the alignment module (Sec. 3.2.4) works as a subsequent layer after the decoder (Sec. 3.2.3). Based on the description in Section 3.2.4, the alignment module leverages the token embedding from the decoder as the input. However, the first sentence mentions that \"re-aligns the outputs generated by the decoder\". This may possibly mislead that the alignment module aligns the discrete textual outputs of the decoder, which makes it hard to understand how the approach is implemented. In addition, a justification of choice of why the token embedding is needed. In Equation 6, the hidden states from the decoder are used. It is suggested to clarify how Equation 5 transforms to Equation 6.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a personalized code-mixed generation model for generating human-alike code-mixed texts. The key motivation is to incorporate user information while generating code-mixed texts. To this end, this paper makes a pioneering effort in utilizing user persona by leveraging a Transformer-based encoder-decoder architecture to learn the semantics of code-mixed generation and utilizing a novel persona encoder to encode a user persona from their behavioural preferences. Instead of projecting the user’s persona onto a static space, the proposed approach projects it onto a probabilistic latent space and captures the contextual persona based on their historical persona. Additionally, the approach uses an alignment module to re-align decoder outputs to generate coherent texts. In contrast to existing studies, the proposed approach does not require parallel corpora or reference data for code-mixed text generation.\n\nThis is a resubmission, so I put some texts from the last round of review that are still applicable to this version.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Clarify the relationship between the alignment module and the decoder. Explain how the information flows between those layers.", "broader_impact_concerns": "The authors have incorporated a statement for broader impact. Although the authors claim that they do not anticipate any immediate negative impact, a possible risk of privacy leakage could be discussed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "goIFsf2LMJ", "reviewer": "Reviewer_qdqK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n1) Lots of contributions - new dataset, new metric, new method \n2) Method outperforms all other methods and the ablations shows each component is generally useful \n\nWeaknesses \n1) As far as I understand, there is no part of the model that helps with the problem of code-mixing (i.e. generating text in different languages). This motivation for the model seems specifically designed for conditioning on user information (adding the user embedding and the VAE part). This problem of conditioning on user information is then applied to is code-mixing. I think the framing in the intro implies the main problem is code-mixing (and thus when reading, I expected the motivation for the new method to address code-mixing and not conditioning on user information). \n2) I didn’t follow the motivation for projecting the static persona embedding onto a probabilistic latent space. It says “The motivation behind projecting the persona embedding to a latent space is to capture the contextual perturbations in the user persona. For example, a user can have a generally positive outlook (global) on a particular concept; however, given a specific situation, the extent of the positive outlook (local) might change”.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "They propose a new model to solve the task of persona-aware generation, where text from different languages are used together. The new model consists of an 1) encoder where the embeddings are sampled (like a VAE) and consist of user ID embeddings and a 2) decoder where there is an alignment of tokens at the very last stage. They also 1) construct 2 datasets for this task 2) develop evaluation metrics for this task. Finally, they show their method outperforms other methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Minor\n- I think the intro should be modified (maybe add 2 sentences) to describe the problem of conditioning on user information first and then transitioning to how this can help a lot with code-mixing and then talking about code-mixing. See Weakness1 above. \n- I think a sentence should be added to describe how sampling from a normal distribution captures the dependence of the local persona on the specific situation more than just using the mean embedding would. See weakness 2 above. \n\nNit\n- How come perplexity can be computed (which I thought requires the ground truth) but there is no labelled gold data for eval (sec 4.2)? \n- Regarding new users, if I understand correctly, any new user would have the same user id since there is only 1 UNK token learned? \n- Also, what percentage of users in the validation are actually not seen during training? \n- For the alignment module, is there a specific reason outputs generated by the decoder have to be aligned for persona-aware generative models?\nHow are user ids leveraged in the zero,one, or two shot setting for Llama or GPT-4?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is the second revision of this paper, and the reviewers are now unanimous that it meets TMLR's acceptance criteria. The one reviewer who provided a more borderline assessment was convinced by your response, and nothing further emerged in our discussion. 

Congratulations! 

Looking back at the initial reviews and this second round, I notice several questions about the proposed contextual persona modelling. Readers would likely have the same questions, and the authors can still improve the exposition in the camera-ready version. In particular, I believe that a more detailed discussion of the results of the ablation study would be worthwhile. Here are two more precise suggestions: 
- Would it be worth adding a specific example toward the start of Section 3.2.2? You already have part of it, but perhaps connecting it to the proposed model would clarify any remaining questions. 
- Reviewer qd2s highlighted the seemingly small differences in performance when removing the contextual persona modelling. Adding your response (highlighting statistical significance) and a short discussion about why the performance improvements are significant but modest would help, especially in comparison to Table 2, where the gains seem larger. Similarly, discussing the results of the new baseline (GPT-4-1-shot) in the text would be helpful as it seems to perform similarly to the full Paradox on Twitter (of course, I understand there is an ablated version of Paradox, which works better here). 

Minor: 
- On page 10, it is written, "Our ablation study (c.f. Table 2)" Did you mean, e.g.? 
- In Table 3, you use boldface; perhaps I missed it, but I did not find a description of what this signifies exactly (statistical significance?).
Decision: Accept (poster)