Package: csify
CSifyGenerate code-switched texts from monolingual texts.If you got here by DOI citation on papers, this might be a snapshot of the repository during the time of writing.
The latest release can be found below.This repository is an implementation of our paper "Generating Code-Switched Text from Monolingual Text with Dependency
Tree," accepted for publication atALTA 2022.In this documentation, we define the notation [X]-[Y] as code switched sentence with X language as the base language and
Y language as the inserted language.
We use ISO 639-1 Code for our naming convention. For example, JA-KO means a Japanese-Korean code switched text generated
from a monolingual Japanese text.SetupThis package is available atPyPI. You can install with pip.pip install csifyThis package only comes withspaCyand contains no machine translator.The CSify ClassThe CSify class generates code-switched text from a monolingual base sentence by translating parts of it
to the language you want to insert via the translate function.You need to bring your own machine translator.
Here is an example code on generating EN-JA code-switched sentence
usingDeepL API.fromcsifyimportCSifyimportdeepl# Initialize DeepL machine translatortranslator=deepl.Translator("<deepl_apikey>")EN_TO_ENJA={"spacy_model":"en_core_web_sm","translate_func":lambdabase_sentence:translator.translate_text(base_sentence,target_lang="JA").text.strip("。"),"space":' '}code_switcher=CSify(**EN_TO_ENJA)print(code_switcher.generate("your last report was more than two weeks ago."))print(code_switcher.generate("our lives are not our own, from womb to tomb, we're bound to others."))outputsyour last report was 二週間以上前 .
私たちの人生は、自分だけのものではないのです、胎内から墓場まで , we 're bound to others .Upon initialization, the CSify class takes three arguments:spacy_model: The spaCy trained pipeline of the base sentence's language (e.g. "en_core_web_sm" for English).
Here is thelist of available pipelines. Note that the pipeline MUST support dependency
parsing. There is no need to download the spaCy pipeline beforehand. The Csify class will do it for you.translate_func : An str -> str function. It takes a text of the base sentence's language as input and outputs the
input's inserted language translation. Wrap the machine translator's translate function to a new function. It is
recommended to truncate all kinds of punctuation of the inserted language in this function as most of the translation
will be done on subsentences, not complete sentences.space : default=' '. Word separator of the base language. Some languages, such as Chinese and Japanese, don't use
space. In that case, space should be an empty string.If you are using DeepL or Google Cloud Translation API,
there are already some pre-built function arguments for CSify class atdemo/deepl_args.pyanddemo/google_translate_args.pyrespectively. For example, to generate EN-ZH with DeepL,
the CSify function arguments look something like thisEN_TO_ENZH={"spacy_model":"en_core_web_sm","translate_func":lambdabase_sentence:translator.translate_text(base_sentence,target_lang="ZH").text,"space":' '}Adding More Language PairsAdding more language pairs equates to adding a function argument combination for the Csify class. Do note that
base sentences can only be from languages that have Spacy trained parser pipeline. You can even bring your own machine
translator. The following code is an example template of using your custom machine translator to create DE-SV
code-switched
sentences.fromcsifyimportCSifyfrommy_awesome_translatorimportgerman_to_swedish_translatormy_translator=german_to_swedish_translator()my_code_switcher_args={"spacy_model":"de_core_news_sm","translate_func":lambdabase_sentence:my_translator.my_translate_function(base_sentence),"space":' '}code_switcher=CSify(**my_code_switcher_args)print(code_switcher.generate("Mein Name ist Sam, obwohl er kurz für Samantha ist."))Setup - Demo:warning: WARNINGWarning: The JESC demo translates around 100,000 characters. Pay attention to your API character limit!Clone this repositorygit clone https://github.com/Selubi/CSify.gitInstall library dependenciespip install -r requirements.txtSetup eitherDeepL APIorGoogle Cloud Translation AIor both as machine translators.
Alternatively, you can bring your own machine translator. Refer toThe CSify ClassandAdding More Language Pairsfor more details.For DeepL, getDeepL API Keyand insert the key
indemo/constants.pydeepl_apikey="<insert deepl API key here>"For Google Cloud Translation AI, follow thissetup guideuntil
"Create a service account key." You should get a JSON file. Save the JSON file and insert the path to it
indemo/constants.py.path_to_google_cloud_JSON_key="<insert path to google cloud JSON key here>":warning: WARNINGIt is recommended to assumeconstants.pyas unchanged in git to prevent API key leakage.git update-index --assume-unchanged demo/constants.pyDeepL is relatively easier to set up but has less supported language than Google Cloud Translation AI.Demo: Generating EN-JA and JA-EN fromJESC CorpusRefer to the below snippet ofdemo/main.py."""This demo function below is defined at ./demo.pyIt downloads and extracts the JESC split corpus, a parallel Japanese-English monolingual corpus.Of the extraction results located at ./data/split, we will take the test data (./data/split/test) that contains2000 lines and generate code-switched data from it.The result will be in 2 files:English sentences and code-switched sentences generated from it will be stored in ./data/CSified/EN-Code-SwitchedJapanese sentences and code-switched sentences generated from it will be stored in ./data/CSified/JA-Code-SwitchedThis demo also features a progress bar that tracks how many sentences it has generated and its speed init/s (sentences per second)."""demo.generate_jesc_cs():warning: WARNINGWarning: this demo translates around 100,000 characters. Pay attention to your API character limit!