Enhancing Cross-lingual Transfer via Phonemic Transcription Integration
Year: 2023

Abstract:
Previous cross-lingual transfer methods are restricted to orthographic representation learning via textual scripts. This limitation hampers cross-lingual transfer and is biased towards languages sharing similar well-known scripts. To alleviate the gap between languages from different writing scripts, we propose PhoneXL, a framework incorporating phonemic transcriptions as an additional linguistic modality beyond the traditional orthographic transcriptions for cross-lingual transfer. Particularly, we propose unsupervised alignment objectives to capture (1) local one-to-one alignment between the two different modalities, (2) alignment via multi-modality contexts to leverage information from additional modalities, and (3) alignment via multilingual contexts where additional bilingual dictionaries are incorporated. We also release the first phonemic-orthographic alignment dataset on two token-level tasks (Named Entity Recognition and Part-of-Speech Tagging) among the understudied but interconnected Chinese-Japanese-Korean-Vietnamese (CJKV) languages. Our pilot study reveals phonemic transcription provides essential information beyond the orthography to enhance cross-lingual transfer and bridge the gap among CJKV languages, leading to consistent improvements on cross-lingual token-level tasks over orthographic-based multilingual PLMs.1

heading: 1 Introduction
text: Despite recent advances in cross-lingual pretrained language models (PLM) such as mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020), PLMs remain heavily biased towards highresourced languages due to the skewed amount of available pre-training data under parameter capacity constraints. This heavily affects the downstream task performance of less-represented languages during pre-training. In addition, as most
1Our code and datasets are publicly available at https://github.com/nhhoang96/phonemic_xlingual
high-resourced languages share the similar scripts (i.e. mostly Latin scripts), PLMs tend to perform well on languages sharing similar scripts with those languages (Pires et al., 2019; Muller et al., 2021; Fujinuma et al., 2022). As most challenging lowresourced languages do not share similar scripts with common high-resourced languages, this limitation leaves them significantly behind.
To alleviate the challenges on low-resource or zero-resource target languages, recent works attempt to transfer knowledge from high-resourced languages (typically English (EN)) to lowresourced target languages via augmentations from bilingual dictionaries and parallel corpora. However, these approaches are restricted to English source language and results in less significant performance gain on languages that are more distant from English (Yang et al., 2022; Fujinuma et al., 2022). Languages can be considered distant due to the differences in orthography, phonology, morphology and grammatical structures. The fact that performance drop occurs on distant languages
2For the sake of simplicity and clearer comparison of phonemic representation similarity between source and target languages, we omit the tonal IPA characters. Tonal IPA characters are preserved as a part of the phonemic inputs for tokenization and training purposes.
9163
when transferring from English indicates that additional works are needed to exploit connectivity between closely related languages, especially under extreme parameter constraints.
Besides purely relying on the orthographic representation in the format of written scripts, additional modality of languages such as articulatory signals can provide essential information beyond the written scripts to enhance textual representations (Bharadwaj et al., 2016; Chaudhary et al., 2018). Phonemic transcriptions which capture linguistic articulatory signals are beneficial to understanding non-Latin-based languages when integrated with PLMs (Sun et al., 2021). They can also facilitate for knowledge transfer between languages sharing lexical similarities in phonology but possessing different writing scripts. As demonstrated in Table 1, despite differences in orthographic representations, the terms “电子” (ZH) and “Điện tử” (VI) possess significant phonemic similarities when encoded into International Phonetic Alphabet (IPA). Similarly, although “ベトナム” (JA) and “베트 남” (KO) are different, their phonemic representations (“bIt@nAmu” and “bEtunæm” respectively) are almost identical in terms of articulatory features.
Motivated by the inherent lexical similarities in terms of phonology among CJKV languages, we propose a novel cross-lingual transfer framework to integrate and synthesize two specific linguistic modalities (1) textual orthographic input scripts, (2) phonemic transcription, represented in International Phonetic Alphabet (IPA) format. Our unified cross-lingual transfer framework aims to effectively (1) align both orthographic and phonemic transcriptions via multi-modality learning, (2) capture additional alignment between the two modalities via contextual information, (3) enhance crosslingual transfer of the two modalities with additional bilingual dictionary. Our work specifically targets Chinese-Vietnamese-Japanese-Korean languages which are not well-studied in cross-lingual transfer and possess lexical similarities with one another in terms of phonology. Our contributions can be summarized as follows:
• We provide the first pre-processed orthographic-phonemic transcription alignment dataset for token-level tasks (i.e. Part-of-Speech Tagging (POS) and Named Entity Recognition (NER)) among CJKV languages (Chinese-Japanese-KoreanVietnamese).
• We propose a multi-modality learning paradigm with unsupervised alignment objectives to fuse the knowledge obtained from both modalities/ transcriptions to enhance cross-lingual transfer.
• Our proposed framework yields consistent improvements over the orthographic-based multilingual PLMs (mBERT and XLM-R) on both POS and NER tasks.

heading: 2 Related Work
text: Cross-lingual Transfer Recent works in Crosslingual transfer focus on generating multilingual contextualized representation for different languages based on the Pre-trained Language Models (PLM) via bilingual dictionaries (Qin et al., 2021) and/or machine translation approaches (Fang et al., 2021; Yang et al., 2022). Qin et al. (2021) proposes a comprehensive code-switching technique via random selection of languages, sentences, and tokens to enhance multilingual representations, leading to improved performance on target languages on different downstream tasks. On the other hand, other approaches leverage parallel corpora generated by Machine Translation to (1) distill knowledge from source languages to target languages (Fang et al., 2021) or augment source language data with target language knowledge during training (Yang et al., 2022; Zheng et al., 2021). However, current cross-lingual efforts concentrate on single source language (EN) to multiple target languages. Under parameter capacity constraints, cross-lingual transfer has been shown to be biased towards high-resourced languages which share similar scripts and possess larger corpora of unlabeled data during pre-training (Fujinuma et al., 2022). Unlike previous works, we specifically target enhancing performance of low-resourced languages by exploiting inherent linguistic similarities between closely-connected languages (Nguyen and Rohrbaugh, 2019; Zampieri et al., 2020).
Multi-modality Learning Multi-modality learning (Radford et al., 2021; Li et al., 2021, 2022) was initially proposed for the task of Visual-Question Answering (Goyal et al., 2017). The objective is to find alignment between the given images and textual input (i.e. caption). The two aligned modalities are trained to maximize the agreement with ground truth textual-image alignment. Despite its simple objectives, the CLIP (Radford et al., 2021)
pre-training mechanism is considered the state-ofthe-art in multi-modality representation learning. Motivated by multi-modality learning, we integrate multi-modality learning approaches in unifying two modalities of transcriptions (orthographic and phonemic) for better representation enrichment.

heading: 3 Problem Formulation
text: In this work, we study the problem of Cross-lingual Transfer in a bilingual setting where there exists annotated data collection of high-resource language S, namely DtrainS = {(X (S) i , Y (S) i )}Nsi=1, and unlabeled data collection of low-resource target language T, denoted as DtestT = {(X (T ) j )}Ntj=1. Ns, Nt denote the sample size of source language training data and target language inference dataset respectively.
Formally, given an i-th input source language utterance with the length of M orthographic tokens x(S)i = [x (S) i,1 , x (S) i,2 ..., x (S) i,M ] and the corresponding phonemic transcriptions z(S)i = [z
(S) i,1 , z (S) i,2 ..., z (S) i,M ] and token-level labels y (S) i =
[y (S) i,1 , y (S) i,2 ..., y (S) i,M ], the overall training objective is summarized as:
θS = argmin θ
1
Ns
Ns∑
i=1
l(F (x (S) i , z (S) i ; θ), y (S) i )
(1) where F (·) denotes the transformation function that takes an input of both x(S)i , z (S) i to output probability prediction of label y(S)i for individual tokens. θ denotes the parameters of the transformation framework and l(·) is the token-level crossentropy loss.
The overall trained framework is then evaluated in a zero-shot setting on target language T as follows:
p(y(T )|x(T ), z(T )) = argmax k F (x(T ), z(T ); θS)
(2) where k denotes the label space of token-level tasks. In our cross-lingual transfer settings, no labeled target data or parallel data between source and target language is leveraged during training.

heading: 4 Proposed Framework
text: In this section, we introduce our proposed Crosslingual Transfer Framework, namely PhoneXL, with 3 major learning objectives: (1) OrthographicPhonemic Alignment (Lalign), (2) Contextual
Cross-modality Alignment (LMLM ) , (3) Contextual Cross-lingual Alignment (LXMLM ). The overall learning objective is summarized as follows:
L = Ltask+αLalign+βLMLM+γLXMLM (3)
where Ltask denotes the corresponding downstream token-level task and λ, β, γ correspond to weights of the respective losses for balanced loss integration. For Ltask is computed based on the training objective in Equation 1 as we leverage the generic CRF layer on top of the sequence output from PLM to generate the probability distribution of each token over the token-level class labels.

heading: 4.1 Orthographic-Phonemic Alignment
text: Traditional PLMs such as BERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020) encode the pre-tokenized input texts into 3 separate trainable embeddings: (1) token embedding (−→wt), (2) positional embedding(−→wp), (3) segment embedding (−→ws) where −→wt,−→wp,−→ws ∈ RD and D denotes the hidden dimensions of corresponding PLM. In our work, we name the token embedding as orthographic embedding (OE) to distinguish from (1) phonemic embedding, (2) unified embedding from both phonemic and orthographic inputs. The overall representation of individual tokens is computed as a summation of three types of embedding −→w = −→wt +−→wp +−→ws.
With the goal of enhancing textual representations via both orthographic and phonemic transcriptions, we introduce the Phonemic Embedding (PE) to directly capture phonemic transcriptions. Phonemic embedding, namely −−→wPE ∈ RD, encodes the representations of phonemic transcription inputs. Phonemic Embedding is integrated with orthographic embedding, positional and segment embedding to form the token representations 3.
Motivated by previous works (Conneau and Lample, 2019; Chaudhary et al., 2020), we introduce additional Language Embedding (−→wl ∈ RD) to encode the input language types. These signals are beneficial to training objectives in cross-lingual settings with code-switched inputs introduced in Section 4.3.
The final word representation for the PLM Encoder is −→w = −→wt + −→wp + −→ws + −−→wPE + −→wl. We
3To ensure the alignment between length-variable phonemic and orthographic input resulted from tokenization, we meanpool the embedding of sub-tokens of individual inputs to construct representation for token-level tasks.
To encourage the alignment between the orthographic textual input and its corresponding phonemic representation, we leverage cross-modality alignment and propose the computation of the phonemic-orthographic alignment loss:
LOtoP = CrossEntropy(simOtoP , labels) (4) The similarity matrices between phonemic and
orthographic inputs (simOtoP ) is computed as:
simOtoP =
M∑
m
−−−−→wm,PE ||−−−−→wm,PE || ∗ −−→wm,t ||−−→wm,t|| ∗ τ (5)
where τ denotes the learnable soft temperature parameter and || · || is L2-normalization. −−−−→wm,PE ,−−−→wim,t denote OE and PE of the m-th token in a sentence of length M.
Similarly to text-image alignment in crossmodality learning (Radford et al., 2021), the alignment is computed as a bi-directional interaction between orthographic and phonemic transcriptions. Therefore, the overall alignment loss is as follows:
Lalign = (LOtoP + LPtoO)/2 (6)

heading: 4.2 Contextual Cross-modality Alignment
text: The introduced alignment from 4.1 is restricted to
1-1 alignment between IPAs and tokens. However, contexts can significantly affect alignment between IPA and tokens. For instance, the orthography of 行 usually corresponds to [C̄iN]. However, the orthography of 行业 corresponds to [xÁN jÈ]
To overcome the challenges, we propose introducing additional Masked Language Modeling to further align the two modalities. In other words, we randomly mask µ% of input orthographic tokens and train the models to predict the masked tokens via (1) contextual/ non-masked orthographic tokens , (2) all of the phonemic transcriptions (including those of masked tokens). This objective encourages the alignment between phonemic and orthographic inputs via contextual information from both modalities of languages. Specifically, given a masked orthographic input and its corresponding phonemic representation, the model aims at predicting the masked tokens correctly. The loss is summarized as follows:
LMLM = − ∑
j∈C log(P (yj |−→vj ; θ) (7)
where yj ,−→vj denote the j-th location ground truth MLM label and input representation produced by PLM as introduced in 4.1 respectively. C denotes the number of masked tokens in the input training sentence.

heading: 4.3 Cross-lingual Contextual Alignment
text: Up to this point, the model does not leverage any specific cross-lingual signals for knowledge transfer between source and target languages. Therefore, we further introduce the Cross-lingual Contextual Alignment objective via bilingual dictionary. Similarly to Contextual Multi-modal Alignment as introduced in Section 4.2, we leverage MLM objectives to encourage the recognition of source language orthographic tokens given the phonemic inputs and multilingual orthographic contexts. The major difference between XMLM and MLM is that the input to XMLM is the code-switched input utterances which contain a mix of tokens from both source and target languages. Specifically, following (Qin et al., 2021), we conduct random codeswitching of tokens of the source input utterances with ratio of r% where r is a hyperparameter. The code-switched inputs follow the same procedure of MLM as introduced in Section 4.2. The XMLM training objective is summarized as follows:
LXMLM = − ∑
j∈C′ log(P (yj |
−→̃ vj ; θ) (8)
where −→̃ vj is the PLM representation of j-th token for the code-switched input sentences of source language and C ′ is the number of masked tokens based on percentage of code-switched source language inputs. Depending on the selected tokens for code-switching and its corresponding target language tokens, the absolute values of C and C ′ are not necessarily the same since the number of tokens in source samples and code-switched samples might not be identical.

heading: 5 Experiments

heading: 5.1 Datasets & Preprocessing
text: We evaluate our proposed framework on tokenlevel tasks, including Named Entity Recognition (NER) and Part-of-Speech Tagging (POS) among four languages: Chinese (ZH), Vietnamese (VI), Japanese (JA) and Korean (KO). Based on linguistic structural similarities (SVO vs SVO structural orders) and lexical similarities in terms of phonemic representations, we divide the four languages into 2 groups: (JA,KO) and (ZH,VI) where the former in each pair is considered a high-resourced language and the latter is a low-resourced counterpart. During training, only high-resourced languages are leveraged and we conduct zero-shot evaluation on the target low-resourced languages.
To evaluate our proposed framework on tokenlevel tasks, we first construct a new dataset by preprocessing the alignment between orthographic and phonemic transcriptions. Specifically, we leverage NER and POS datasets (namely PANX and UDPOS) from XTREME benchmark datasets (Hu et al., 2020). Given input utterances from the datasets, we generate the corresponding phonemic transcriptions on token-level. As phonemic transcriptions can either Romanized Transcriptions (i.e. Pinyin for ZH, Romaji for JA, Romaja for KO) or IPA Transcriptions, we generate both types of phonemic transcriptions and conduct empirical study on both in Section 6.
Generating Romanized Transcriptions As VI is written in Latin, we preserve the original characters as the corresponding Romanized transcriptions. For ZH, JA, KO, we directly obtain the corresponding Romanized transcriptions via dragonmapper, pykakasi and korean_romanizer respectively.
Generating IPA Transcriptions As PanPhon (Mortensen et al., 2016) does not support some of our targeted languages (JA, KO), we leverage external open-source tools to generate IPA transcriptions for individual languages. Specifically, we use dragonmapper, viphoneme to generate IPA transcriptions for ZH, VI respectively. As there are no direct open-source IPA transcription tools available for JA,KO, we generate IPA transcriptions via a two-stage process. First, we generate Romanized transcrptions for JA, KO via pykakasi and korean_romanizer respectively. Then, as the aforementioned Romanized transcriptions are in Latin-based format, we treat them as Latin char-
acters and input them to PanPhon to generate IPA transcriptions for JA and KO. Details of our constructed datasets are provided in Table 2.

heading: 5.2 Implementation Details
text: For evaluation of both NER and POS tasks, we report the F-1 score for different individual language pairs. We report performance on both development and test set of both source and target languages.
As IPA involves unique characters that are outside the typical orthographic vocabulary (i.e. /N/, /C/, /Ð/, /Z/), we extend PLM vocabulary to account for these special characters. Therefore, both OEs and PEs are resized to account for the newly extended vocabulary. The impact of extended vocabulary will be further discussed in Section 6.
For each language pair (ZH-VI vs JA-KO) and token-level tasks (NER vs POS), we tune hyperparameters of our framework based on the development set of the source language. Specifically, we conduct grid search for λ, β, γ over the space [0.1, 0.01, 0.001, 0.0001]. Mask ratio (µ) and cs_ratio (r) are tuned over the space [0.1, 0.4] inclusive with a step of 0.05. Hyperparameter details for each task and language pair are reported in Table 5.
We train our model with the batch size of 32 and 16 for mBERT and XLM-R respectively. Both
multilingual base versions (L=12, H=12, D=768 where L,H,D denote the number of hidden layers, the number of attention heads per layer and hidden dimension respectively) are used as backbone PLM architectures for our experiments. Both training and inference of our framework are conducted on NVIDIA TITAN RTX and NVIDIA RTX 3090 GPUs. We report our experimental results as the average performance of 3 runs from different random seed initializations with standard deviations. Due to space constraints, we report our empirical studies on test sets in Table 3 and 4. Additional results on development (dev) sets for both datasets are summarized in the Appendix A.
In our study, we leverage publicly available MUSE bilingual dictionary (Conneau et al., 2017). As bilingual dictionary is only available between
EN and target languages, we construct bilingual dictionaries for our language pairs (ZH-VI and JAKO) by leveraging EN as a bridge for semantic alignment between source and target languages.

heading: 5.3 Baseline
text: We compare our method with previously proposed cross-lingual transfer frameworks for token-level tasks. We conduct experiments with both mBERT and XLM-R backbone PLM architecture. We compare our work with both approaches leveraging Machine Translation (MT) and/or Bilingual Dictionary (Dict), including:
• CoSDA-ML (Qin et al., 2021): Multi-level Code-switching augmentation for various cross-lingual NLP tasks, including POS and NER tasks.
• FILTER (Fang et al., 2021): Cross-lingual Transfer via Intermediate Architecture Disentanglement with Knowledge Distillation objectives from source to target languages.
• xTune (Zheng et al., 2021): Two-stage augmentation mechanisms with four exhaustive augmentation methods for cross-lingual transfer.
• XMIXUP (Yang et al., 2022): Cross-lingual transfer via Manifold Mixup Augmentation and Machine Translation Alignment between source and target languages.
As FILTER, xTune and XMIXUP require training parallel corpora, we generate translation of training data from source languages (ZH, JA) to target languages (VI, KO) via third-party MT package4. Ground-truth label sequence of MT data is the same as the original source language data.
As CoSDA-ML and xTune both require bilingual dictionary for cross-lingual transfer, we leverage the reconstructed MUSE bilingual dictionary for our setting as introduced in 5.2.

heading: 6 Result & Discussion
text: Our experimental results for NER and POS tasks are summarized in Table 3 and 4. Based on the empirical study, our proposed PhoneXL framework consistently outperforms by the backbone PLM architecture of mBERT and XLM-R in both evaluated token-level tasks for low-resourced target
4https://pypi.org/project/googletrans/
languages VI and KO. For instance, with ZH-VI pair, we observe the target language’s F1 evaluation metric improvement of 2.34 points and 2.01 points for NER and POS tasks respectively as compared to the fine-tuned backbone mBERT architecture. This improvement implies the phonemic information provides essential information beyond orthographic representation to further bridge the gap between the source and target languages.
In NER task, the larger and state-of-the-art multilingual PLM XLM-R yields worse performance than mBERT in both of our language pair performance on source and target languages. On the other hand, for POS task, XLM-R based architecture only results in marginal performance gains when compared with mBERT. Interestingly, our mBERT-based framework achieves competitive performance with XLM-R backbone architecture on POS task despite leveraging smaller vocabulary size and less pre-training data. We hypothesize this might be due to the fact that XLM-R has been trained with more languages, leading to biases towards certain types of languages during pre-training that might not share common properties with CJKV languages.
Despite the performance gain over XLM-R based architecture observed in Table 3, our PhoneXL framework does not consistently outperform the previous baselines such as FILTER, xTune and XMIXUP in Table 4. However, these baselines require extra parallel corpora obtained from machine translation which might not always be readily available for all languages, especially for low-resourced ones. On the other hand, our proposed method achieves state-of-the-art performance among methods leveraging only bilingual dictionary. In addition, despite its impressive performance, xTune requires two-stage training procedures, four different exhaustive augmentation methods as well as the knowledge of both machine translation and bilingual dictionary. Therefore, it is deemed more time-consuming and resourceintensive than our approach.
Romanized vs IPA Phonemic Transcriptions As observed in Table 6, leveraging Romanized transcriptions from individual languages instead of IPA degrades the downstream POS task performance on both low-resourced target languages (averaged 0.93 and 1.57 points of performance drop on VI and KO respectively from PhoneXL-IPA (full) to PhoneXLRomanized (full)). We hypothesize it might be due
to the lack of phonemic consistency among Romanized transcriptions of different languages. In addition, as certain low-resourced languages might not have their own Romanized transcriptions, IPA phonemic transcriptions provide a more generalized and consistent pathway to generate phonemic transcriptions across different language families.
Impact of Phonemic Embeddings Based on Table 6, we also observe that the introduction of IPA embedding, even without any unsupervised objectives, also provide additional improvements as compared to the backbone orthographic-based mBERT architecture. However, further training with our introduced objectives provide stronger boost in target language performance improvements on the downstream task.
Impact of Vocabulary Extension As observed in Table 6, vocabulary extension is important to learn effective Phonemic Embedding. Vocabulary extension allows the model to differentiate unique IPA characters when encoding tokens, leading to 0.19 and 1.42 points of F1 score improvements on mBERT for VI and KO respectively. It is intuitive since additional special phonemic characters possess different meanings than typical orthographic characters. However, we still observe a significant gap between mBERT with PE and PhoneXL framework. It is due to the lack of alignment between embedding of phonemic and orthographic inputs.
Impact of Unsupervised Objectives As observed in Table 3, each introduced alignment objective between phonemic and orthographic inputs provides additional performance gain over the original backbone mBERT and XLM-R PLMs on both language groups. Additionally, from Table 6, the existence of performance gap between simple introduction of PE (mBERT (w PE, w extensions)) and PhoneXL (i.e. 1.57 points on VI and 2.33 points on KO in POS task) implies that the unsupervised alignment objectives are crucial to bringing about
the benefits of PE.
Impact of Language Embedding Language Embedding is crucial to our framework performance, leading to consistent performance gain on both target languages in POS task. In fact, Language Embedding is especially important to LXMLM as inputs are code-switched sentences which are made up of tokens from different languages. Without language indication from Language Embedding, the model is unable to predict the correct masked tokens in the correct language.

heading: 7 Conclusion & Future Work
text: In our work, we propose PhoneXL, a novel mechanism to integrate phonemic transcription with orthographic transcription to further enhance representation capability of language models in cross-lingual transfer settings. By encouraging the alignment between the two linguistic modalities via direct one-to-one alignment, indirect contextual alignment and additional code-switching via bilingual dictionaries, our proposed PhoneXL yields consistent performance improvements over the backbone orthographic-based PLM architecture in downstream cross-lingual token-level task among the CJKV languages. We also release the first aligned phonemic-orthographic datasets for CJKV languages for two popular token-level tasks (NER and POS). In future work, we plan to train our proposed unsupervised objectives with larger CJKV corpora as pre-training mechanisms to evaluate effectiveness of the representations in multigranularity downstream tasks (i.e. sentence-level classification tasks to Question-Answering tasks). Further extensions towards few-shot learning settings (Nguyen et al., 2020; Xia et al., 2020) where a small number of target language examples can be leveraged to exploit orthographic-phonemic similarity between source and target languages is a promising direction for our future work.
Limitations
Our approach is heavily dependent on the quality of the pre-processed orthographic-phonemic transcription data as it provides the ground-truth for unsupervised alignment objectives. Generating phonemic transcriptions and aligning them correctly with orthographic representations can be costly. Despite our significant efforts, the alignment is still far from perfect optimality.
Secondly, our approach might not be effective in improving performance on randomly chosen language pairs. As our framework aims to exploit phonemic similarities of languages with different orthographic representations, the methods are only effective in cross-lingual transfer between lexically similar languages in terms of phonology such as CJKV languages. Languages that do not fall into this category might observer little to no performance gains with our proposed framework.

heading: A Additional Experiments
text: We provide additional experiments on dev set of PANX and UDPOS datasets of XTREME benchmark datasets in Table 7 and 8. Our observations are mostly consistent between dev and test sets on both evaluated token-level tasks.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitation section
3 A2. Did you discuss any potential risks of your work? Limitation section
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 5
3 B1. Did you cite the creators of artifacts you used? Section 5
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 5 and Limitation
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 5
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 5
C 3 Did you run computational experiments? Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In future work, we plan to train our proposed unsupervised objectives with larger CJKV corpora as pre-training mechanisms to evaluate effectiveness of the representations in multigranularity downstream tasks (i. e. sentence-level classification tasks to Question-Answering tasks). Further extensions towards few-shot learning settings (Nguyen et al. , 2020; Xia et al. , 2020) where a small number of target language examples can be leveraged to exploit orthographic-phonemic similarity between source and target languages is a promising direction for our future work. Limitations
Our approach is heavily dependent on the quality of the pre-processed orthographic-phonemic transcription data as it provides the ground-truth for unsupervised alignment objectives. Generating phonemic transcriptions and aligning them correctly with orthographic representations can be costly. Despite our significant efforts, the alignment is still far from perfect optimality. Secondly, our approach might not be effective in improving performance on randomly chosen language pairs. As our framework aims to exploit phonemic similarities of languages with different orthographic representations, the methods are only effective in cross-lingual transfer between lexically similar languages in terms of phonology such as CJKV languages. Languages that do not fall into this category might observer little to no performance gains with our proposed framework