Package: wikipron
WikiPronWikiPron is a command-line tool and Python API for mining multilingual
pronunciation data from Wiktionary, as well as a database of pronunciation
dictionaries mined using this tool.Command-line toolPython APIDataModelsDevelopmentIf you use WikiPron in your research, please cite the following:Jackson L. Lee, Lucas F.E. Ashby, M. Elizabeth Garza, Yeonju Lee-Sikka, Sean
Miller, Alan Wong, Arya D. McCarthy, and Kyle Gorman (2020).Massively
multilingual pronunciation mining with
WikiPron. InProceedings
of the 12th Language Resources and Evaluation Conference, pages 4223-4228.
[bibtex]Command-line toolInstallationpipinstallwikipronUsageQuick StartAfter installation, the terminal commandwikipronwill be available. As a
basic example, the following command scrapes G2P data for French:wikipronfraSpecifying the LanguageThe language is indicated by a three-letterISO
639-3language code,
e.g.,frafor French. For which languages can be scraped,hereis the complete list of languages on Wiktionary that have pronunciation entries.Specifying the DialectOne can optionally specify dialects to target using the--dialectflag. The
dialect name can be found together with the transcription on Wiktionary. For
example, "(UK, US) IPA: /təˈmɑːtəʊ/". To restrict to the union of dialects use
the pipe character '|': e.g.,--dialect='General American | US'.
Transcriptions which lack a dialect specification are selected regardless of the
value of this flag.SegmentationBy default, thesegmentslibrary is used
to segment the transcription into whitespace. The segmentation tends to place
IPA diacritics and modifiers on the "parent" symbol. For instance, [kʰæt] is
renderedkʰ æ t. This can be disabled using the--no-segmentflag.ParenthesesSome of transcriptions contain parentheses to indicate alternative pronunciations.
The parentheses (but not the content) are discarded in the scrape unless the--no-skip-parensflag is used.OutputThe scraped data is organized with each <word, pronunciation> pair on its own
line, where the word and pronunciation are separated by a tab. Note that the
pronunciation is inInternational Phonetic Alphabet
(IPA), segmented
by spaces that correctly handle the combining and modifier diacritics for
modeling purposes, e.g., we havekʰ æ twith the aspirated k instead ofk ʰ æ t.For illustration, here is a snippet of French data scraped by WikiPron:accrémentitielle    a k ʁ e m ɑ̃ t i t j ɛ l
accrescent  a k ʁ ɛ s ɑ̃
accrétion   a k ʁ e s j ɔ̃
accrétions  a k ʁ e s j ɔ̃By default, the scraped data appears in the terminal. To save the data in a TSV
file, please redirect the standard output to a filename of your choice:wikipronfra>fra.tsvAdvanced OptionsThewikipronterminal command has an array of options to configure your
scraping run. For a full list of the options, please runwikipron -h.Python APIThe underlying module can also be used from Python. A standard workflow looks
like:importwikipronconfig=wikipron.Config(key="fra")# French, with default options.forword,proninwikipron.scrape(config):...DataWe also make availablea database of over 3 million word/pronunciation
pairsmined using
WikiPron.ModelsWe host grapheme-to-phoneme models and modeling softwarein a separate
repository.DevelopmentRepositoryThe source code of WikiPron is hosted on GitHub athttps://github.com/CUNY-CL/wikipron,
where development also happens.For the latest changes not yet released throughpipor working on the codebase
yourself, you may obtain the latest source code through GitHub andgit:Create a fork of thewikipronrepo on your GitHub account.Locally, make sure you are in some sort of a virtual environment (venv,
virtualenv, conda, etc).Download and install the library in the "editable" mode together with the
core and dev dependencies within the virtual environment:gitclonehttps://github.com/<your-github-username>/wikipron.gitcdwikipron
pipinstall-Upipsetuptools
pipinstall-rrequirements.txt
pipinstall--no-deps-e.We keep track of notable changes inCHANGELOG.md.ContributionFor questions, bug reports, and feature requests, pleasefile an
issue.If you would like to contribute to thewikiproncodebase, please seeCONTRIBUTING.md.LicenseWikiPron is released under an Apache 2.0 license. Please seeLICENSE.txtfor details.Please note that Wiktionary data in thedata/directory hasits own licensing terms.