Package: g2p
Gᵢ2PᵢGrapheme-to-Phoneme transformations that preserve input and output indices!This library is for handling arbitrary conversions between input and output segments while preserving indices.Table of ContentsGᵢ2PᵢTable of ContentsBackgroundInstallUsageCLIupdateconvertgenerate-mappingStudioMaintainersContributingContributorsHow to CiteLicenseSee also:Gᵢ2Pᵢ documentation7-part series on the Mother Tongues BlogGᵢ2Pᵢ StudioBackgroundThe initial version of this package was developed byPatrick Littelland was developed in order to allow for g2p from community orthographies to IPA and back again inReadAlong-Studio. We decided to then pull out the g2p mechanism fromConvertextractwhich allows transducer relations to be declared in CSV files, and turn it into its own library - here it is! For an in-depth series on the motivation and how to use this tool, have a look at this7-part series on the Mother Tongues Blog, or for a more technical overview, have a look atthis paper.InstallThe best thing to do is install with pippip install g2p.
This command will install the latest release published onPyPI g2p releases.You can also clone the repo and pip install it locally:$gitclonehttps://github.com/roedoejet/g2p.git
$cdg2p
$pipinstall-e.UsageThe easiest way to create a transducer is to use theg2p.make_g2pfunction.To use it, first import the function:from g2p import make_g2pThen, call it with an argument forin_langandout_lang. Both must be strings equal to the name of a particular mapping.>>>transducer=make_g2p('dan','eng-arpabet')>>>transducer('hej').output_string'HH EH Y'There must be a valid path between thein_langandout_langin order for this to work. If you've edited a mapping or added a custom mapping, you must update g2p to include it:g2p updateWriting mapping filesMapping files are written as either CSV or JSON files.CSVCSV files write each new rule as a new line and consist of at least two columns, and up to four. The first column is required and corresponds to the rule's input. The second column is also
required and corresponds to the rule's output. The third column is optional and corresponds to the context before the rule input. The fourth column is also optional and corresponds to the context after the rule input. For example:This mapping describes two rules; a -> b and c -> d.a,b
c,dThis mapping describes two rules; a -> b / c _ d1and a -> ea,b,c,d
a,eTheg2p studioexports its rules to CSV format.JSONJSON files are written as an array of objects where each object corresponds to a new rule. The following two examples illustrate how the examples from the CSV section above would be written in JSON:This mapping describes two rules; a -> b and c -> d.[{"in":"a","out":"b"},{"in":"c","out":"d"}]This mapping describes two rules; a -> b / c _ d1and a -> e[{"in":"a","out":"b","context_before":"c","context_after":"d"},{"in":"a","out":"e"}]CLIupdateIf you edit or add new mappings to theg2p.mappings.langsfolder, you need to updateg2p. You do this by runningg2p updateconvertIf you want to convert a string on the command line, you can useg2p convert <input_text> <in_lang> <out_lang>Ex.g2p convert hej dan eng-arpabetwould produceHH EH YIf you have written your own mapping that is not included in the standardg2plibrary, you can point to its configuration file using the--configflag, as ing2p convert <input_text> <in_lang> <out_lang> --config path/to/config.yml. This will add the mappings defined in your configuration to the existingg2pnetwork, so be careful to avoid namespace errors.generate-mappingIf your language has a mapping to IPA and you want to generate a mapping between that and the English IPA mapping, you can useg2p generate-mapping <in_lang> --ipa.  Remember to rung2p updatebefore so that it has the latest mappings for your language.Ex.g2p generate-mapping dan --ipawill produce a mapping fromdan-ipatoeng-ipa. You must also rung2p updateafterwards to updateg2p. The resulting mapping will be added to the folder ing2p.mappings.langs.generatedNote: if your language goes through an intermediate representation, e.g., lang -> lang-equiv -> lang-ipa, specify both the<in_lang>and<out_lang>of your final IPA mapping tog2p generate-mapping. E.g., to generatecrl-ipa -> eng-ipa, you would rung2p generate-mapping --ipa crl-equiv crl-ipa.g2p workflow diagramThe interactions betweeng2p updateandg2p generate-mappingare not fully intuitive, so this diagram should help understand what's going on:Text DB: this is the textual database of g2p conversion rules created by contributors. It consists of these files:g2p/mappings/langs/*/*.csvg2p/mappings/langs/*/*.jsong2p/mappings/langs/*/*.yamlGen DB: this is the part of the textual database that is generated when running theg2p generate-mappingcommand:g2p/mappings/generated/*Compiled DB: this contains the same info as Text DB + Gen DB, but in a format optimized for fast reading by the machine. This is what any program usingg2preads:g2p convert,readalongs align,convertextract, and alsog2p generate-mapping. It consists of these files:g2p/mappings/langs/langs.pklg2p/mappings/langs/network.pklg2p/mappings/langs/static/languages-network.jsong2p/mappings/langs/static/swagger.jsonSo, when you write a new g2p mapping for a language, saylll, and you want to be able to convert text fromllltoeng-ipaoreng-arpabet, you need to do the following:Write the mapping fromllltolll-ipain g2p/mappings/langs/lll/. You've just updated Text DB.Rung2p updateto regenerate Compiled DB from the current Text DB and Gen DB, i.e., to incorporate your new mapping rules.Rung2p generate-mapping --ipa lllto generate g2p/mappings/langs/generated/lll-ipa_to_eng-ipa.json. This is not based on what you wrote directly, but rather on what's in Generated DB.Rung2p udpateagain.g2p generate-mappingupdates Gen DB only, so what gets written there will only be reflected in Compiled DB when you rung2p updateonce more.Once you have the Compiled DB, it is then possible to use theg2p convertcommand, create time-aligned audiobooks withreadalongs align, or convert files with theconvertextractlibrary.StudioYou can also run theg2p Studiowhich is a web interface for creating custom lookup tables to be used with g2p. To run theg2p Studioeither visithttps://g2p-studio.herokuapp.com/or run it locally usingpython run_studio.py.Alternatively, you can run the app from the command line:g2p runMaintainers@roedoejet.ContributingFeel free to dive in!Open an issueor submit PRs.This repo follows theContributor CovenantCode of Conduct.Have a look atContributing.mdfor help using our standardized formatting conventions and pre-commit hooks.Adding a new mappingIn order to add a new mapping, you have to follow the following steps.Determine your language'sISO 639-3 code.Add a folder with your language's ISO 639-3 code tog2p/mappings/langsAdd a configuration file atg2p/mappings/langs/<yourlangISOcode>/config.yaml. Here is the basic template for a configuration:<<:&sharedlanguage_name:<This is the actual name of the language>mappings:-display_name:This is a description of the mappingin_lang:This is your language's ISO 639-3 codeout_lang:This is the output of the mappingtype:mappingauthors:-<YourNameHere>mapping:<FilenameOfMapping><<:*sharedAdd a mapping file. Look at the other mappings for examples, or visit theg2p studioto practise your mappings.
Mappings are defined in either a CSV or json file. Seewriting mapping filesfor more info.After installing your local version (pip3 install -e .), update withg2p updateAdd some tests ing2p/testspublic/data/<YourIsoCode>.psv. Each line in the file will run a test with the following structure:<in_lang>|<out_lang>|<input_string>|<expected_output>Runpython3 run_tests.py langsto make sure your tests pass.Make sure you havechecked all the boxesand make a [pull request]((https://github.com/roedoejet/g2p/pulls)!Adding a new language for support with ReadAlongsThis repo is used extensively byReadAlongs. In order to make your language supported by ReadAlongs, you must add a mapping from your language's orthography to IPA. So, for example, to add Danish (ISO 639-3:dan), the steps above must be followed. Thein_langfor the mapping must bedanand the out_lang must be suffixed with 'ipa' as indan-ipa. The following is the proper configuration:<<:&sharedlanguage_name:Danishmappings:-display_name:Danish to IPAin_lang:danout_lang:dan-ipatype:mappingauthors:-Aidan Pinemapping:dan_to_ipa.csvabbreviations:dan_abbs.csvrule_ordering:as-writtencase_sensitive:falsenorm_form:'none'<<:*sharedThen, you can generate the mapping betweendan-ipaandeng-ipaby runningg2p generate-mapping --ipa. This will add the mapping tog2p/mappings/langs/generated- do not edit this file, but feel free to have a look. Then, rung2p updateand submit apull request, and tada! Your language is supported by ReadAlongs as well!Footnotes1If this notation is unfamiliar, have a look atphonological rewrite rules↩ContributorsThis project exists thanks to all thepeople who contribute.CitationIf you use this work in a project of yours and write about it, please cite us using the following:Aidan Pine, Patrick Littell, Eric Joanis, David Huggins-Daines, Christopher Cox, Fineen Davis, Eddie Antonio Santos, Shankhalika Srikanth, Delasie Torkornoo, and Sabrina Yu. 2022.Gᵢ2Pᵢ Rule-based, index-preserving grapheme-to-phoneme transformations Rule-based, index-preserving grapheme-to-phoneme transformations. InProceedings of the Fifth Workshop on the Use of Computational Methods in the Study of Endangered Languages, pages 52–60, Dublin, Ireland. Association for Computational Linguistics.Or in BibTeX:@inproceedings{pine-etal-2022-gi22pi,
    title = "{G}$_i$2{P}$_i$ Rule-based, index-preserving grapheme-to-phoneme transformations",
    author = "Pine, Aidan  and
      Littell, Patrick  and
      Joanis, Eric  and
      Huggins-Daines, David  and
      Cox, Christopher  and
      Davis, Fineen  and
      Antonio Santos, Eddie  and
      Srikanth, Shankhalika  and
      Torkornoo, Delasie  and
      Yu, Sabrina",
    booktitle = "Proceedings of the Fifth Workshop on the Use of Computational Methods in the Study of Endangered Languages",
    month = may,
    year = "2022",
    address = "Dublin, Ireland",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.computel-1.7",
    pages = "52--60",
    abstract = "This paper describes the motivation and implementation details for a rule-based, index-preserving grapheme-to-phoneme engine {`}G$_i$2P$_i$' implemented in pure Python and released under the open source MIT license. The engine and interface have been designed to prioritize the developer experience of potential contributors without requiring a high level of programming knowledge. {`}G$_i$2P$_i$' already provides mappings for 30 (mostly Indigenous) languages, and the package is accompanied by a web-based interactive development environment, a RESTful API, and extensive documentation to encourage the addition of more mappings in the future. We also present three downstream applications of {`}G$_i$2P$_i$' and show results of a preliminary evaluation.",
}LicenseMIT. SeeLICENSEfor the Copyright and license statements.