Package: lingua-franca
Lingua FrancaMycroft's multilingual text parsing and formatting libraryLingua Franca (noun)a framework that is adopted as the common language between speakers with different native tonguesLingua FrancaFormattingPronounce numbersPronounce datetime objectsPronounce durationsParsingExtract numbersExtract durationsExtract datesContributing to this project0. Sign a Contributor Licensing Agreement1. Setup a local copy of the project2. Writing tests3. Run tests to confirm they fail4. Write code5. Document your code6. Try it in Mycroft7. Commit changes8. Submit a PR9. Waiting for a reviewCreditsFormattingConvert data into spoken equivalentsPronounce numbersspoken versions of numbersfromlingua_franca.formatimportnice_number,pronounce_numberassertnice_number(25/6)=="4 and a sixth"assertnice_number(201)=="201"assertnice_number(3.14159269)=="3 and a seventh"assertpronounce_number(3.14159269)=="three point one four"assertpronounce_number(0)=="zero"assertpronounce_number(10)=="ten"assertpronounce_number(201)=="two hundred and one"assertpronounce_number(102.3)=="one hundred and two point three"assertpronounce_number(4092949192)=="four billion, ninety two million, nine hundred and forty nine thousand, one hundred and ninety two"assertpronounce_number(100034000000299792458,short_scale=True)==\"one hundred quintillion, thirty four quadrillion, "\"two hundred and ninety nine million, seven hundred and ninety "\"two thousand, four hundred and fifty eight"assertpronounce_number(100034000000299792458,short_scale=False)==\"one hundred trillion, thirty four thousand billion, "\"two hundred and ninety nine million, seven hundred and ninety "\"two thousand, four hundred and fifty eight"Pronounce datetime objectsspoken date for datetime.datetime objectsfromlingua_franca.formatimportnice_date,nice_date_time,nice_timeimportdatetimedt=datetime.datetime(2017,1,31,13,22,3)assertnice_date(dt)=="tuesday, january thirty-first, twenty seventeen"assertnice_time(dt)=="one twenty two"assertnice_time(dt,use_ampm=True)=="one twenty two p.m."assertnice_time(dt,speech=False)=="1:22"assertnice_time(dt,speech=False,use_ampm=True)=="1:22 PM"assertnice_time(dt,speech=False,use_24hour=True)=="13:22"assertnice_time(dt,speech=False,use_24hour=True,use_ampm=True)=="13:22"assertnice_time(dt,use_24hour=True,use_ampm=True)=="thirteen twenty two"assertnice_time(dt,use_24hour=True,use_ampm=False)=="thirteen twenty two"assertnice_date_time(dt)=="tuesday, january thirty-first, twenty seventeen at one twenty two"Pronounce durationsspoken number of seconds or datetime.timedelta objectsfromlingua_franca.formatimportnice_durationassertnice_duration(1)=="one second"assertnice_duration(3)=="three seconds"assertnice_duration(1,speech=False)=="0:01"assertnice_duration(61),"one minute one second"assertnice_duration(61,speech=False)=="1:01"assertnice_duration(5000)=="one hour twenty three minutes twenty seconds"assertnice_duration(5000,speech=False),"1:23:20"assertnice_duration(50000)=="thirteen hours fifty three minutes twenty seconds"assertnice_duration(50000,speech=False)=="13:53:20"assertnice_duration(500000)=="five days  eighteen hours fifty three minutes twenty seconds"assertnice_duration(500000,speech=False),"5d 18:53:20"fromdatetimeimporttimedeltaassertnice_duration(timedelta(seconds=500000),speech=False)=="5d 18:53:20"ParsingExtract data from natural language textExtract numbersfromlingua_franca.parseimportextract_number,extract_numbers# extract a numberassertextract_number("nothing")isFalseassertextract_number("two million five hundred thousand tons of spinning ""metal")==2500000assertextract_number("six trillion")==6000000000000.0assertextract_number("six trillion",short_scale=False)==6e+18assertextract_number("1 and 3/4 cups")==1.75assertextract_number("1 cup and a half")==1.5## extracts all numbersassertextract_numbers("nothing")==[]assertextract_numbers("this is a one twenty one  test")==[1.0,21.0]assertextract_numbers("1 dog, seven pigs, macdonald had a farm, ""3 times 5 macarena")==[1,7,3,5]Extract durationsextract datetime.timedelta objects## extract durationsfromlingua_franca.parseimportextract_durationfromdatetimeimporttimedeltaassertextract_duration("nothing")==(None,'nothing')assertextract_duration("Nineteen minutes past the hour")==(timedelta(minutes=19),"past the hour")assertextract_duration("wake me up in three weeks, four hundred ninety seven"" days, and three hundred 91.6 seconds")==(timedelta(weeks=3,days=497,seconds=391.6),"wake me up in , , and")assertextract_duration("The movie is one hour, fifty seven and a half minutes long")==(timedelta(hours=1,minutes=57.5),"the movie is ,  long")Extract datesextract datetime.datetime objects## extract date timesfromdatetimeimportdatetimefromlingua_franca.parseimportextract_datetime,normalizedefextractWithFormat(text):date=datetime(2017,6,27,13,4)# Tue June 27, 2017 @ 1:04pm[extractedDate,leftover]=extract_datetime(text,date)extractedDate=extractedDate.strftime("%Y-%m-%d%H:%M:%S")return[extractedDate,leftover]deftestExtract(text,expected_date,expected_leftover):res=extractWithFormat(normalize(text))assertres[0]==expected_dateassertres[1]==expected_leftovertestExtract("now is the time","2017-06-27 13:04:00","is time")testExtract("in a couple minutes","2017-06-27 13:06:00","")testExtract("What is the day after tomorrow's weather?","2017-06-29 00:00:00","what is weather")testExtract("Remind me at 10:45 pm","2017-06-27 22:45:00","remind me")testExtract("what is the weather on friday morning","2017-06-30 08:00:00","what is weather")testExtract("what is tomorrow's weather","2017-06-28 00:00:00","what is weather")testExtract("remind me to call mom next tuesday","2017-07-04 00:00:00","remind me to call mom")testExtract("remind me to call mom in 3 weeks","2017-07-18 00:00:00","remind me to call mom")testExtract("set an alarm for tonight 9:30","2017-06-27 21:30:00","set alarm")testExtract("on the evening of june 5th 2017 remind me to call my mother","2017-06-05 19:00:00","remind me to call my mother")Getting StartedLoading a languageBefore using Lingua Franca's other functions, you'll need to load one or more languages into memory, using part or all of a
BCP-47 language code:# Load a languagelingua_franca.load_language('en')# Load multiple languages at once## If no default language is set, the first# element will become the defaultlingua_franca.load_languages(['en','es'])# Change the default languagelingua_franca.set_default_language('es')See the documentation for more information about loading and unloading languages.Calling localized functionsMost of Lingua Franca's functions have been localized. You can call a function in any language you've loaded; this is always specified by the function'slangparameter. If you omit that parameter, the function will be called in the current default language.Example:>>>fromlingua_francaimportload_languages,\set_default_lang,parse>>>load_languages(['es','en'])>>>parse.extract_number("uno")1>>>parse.extract_number("one")False>>>parse.extract_number("one",lang='en')1>>>set_default_lang('en')>>>parse.extract_number("uno")False>>>parse.extract_number("one")1In some languages, certain parameters have no effect, either because
those parameters do not apply, or because the localization is not complete.It's important to remember that Lingua Franca is in alpha. Support for a
particular language may be inconsistent, and one language's version of a
complex function might be outdated compared with another.New functionality usually starts in the languages spoken by major
contributors. If your language's functions are lacking, we'd love your help
improving them! (See below, "Contributing.")Contributing to this projectWe welcome all contributions to Lingua Franca. To get started:0. Sign a Contributor Licensing AgreementTo protect yourself, the project, and users of Mycroft technologies, we require a Contributor Licensing Agreement (CLA) before accepting any code contribution. This agreement makes it crystal clear that, along with your code, you are offering a license to use it within the confines of this project. You retain ownership of the code, this is just a license.You will also be added toour list of excellent human beings!Please visithttps://mycroft.ai/clato initiate this one-time signing.1. Setup a local copy of the projectFork the projectto create your own copy.Clone the repository and change into that directorygitclonehttps://github.com/your-username/lingua-franca/cdlingua-francaSetup a lightweight virtual environment (venv) for the project. This creates an isolated environment that can have it's own independent set of installed Python packages.python3-mvenv.venvsource.venv/bin/activateTo exit the venv you can rundeactivateor close the terminal window.Install the package and it's dependenciespipinstallwheel
python-mpipinstall.
pipinstallpytest
pythonsetup.pyinstallTo check that everything is installed correctly, let's run the existing test-suite.pytest2. Have a look at the project's structureThe package's layout is described inproject-structure.md, along with some important notes. It's pretty
intuitive, but uniform file and function names are important to Lingua Franca's operation.3. Writing testsWe utilize a Test Driven Development (TDD) methodology so the first step is always to add tests for whatever you want to add or fix. If it's a bug, we must not have a test that covers that specific case, so we want to add another test. If you are starting on a new language then you can take a look at the tests for other languages to get started.Tests are all located inlingua_franca/test.
Each language should have two test files:test_format_lang.pytest_parse_lang.py4. Run tests to confirm they failGenerally, using TDD, all tests should fail when they are first added. If the test is passing when you haven't yet fixed the bug or added the functionality, something must be wrong with the test or the test runner.pytest5. Write codeNow we can add our new code. There are three main files for each language:common_data_lang.pyCommon data that can be used across formatting and parsing such as dictionaries of number names.format_lang.pyAll formatting functions for this language.parse_lang.pyAll parsing functions for this language.Since we have already written our unit tests, we can run these regularly to see our progress.6. Document your codeDocument code usingGoogle-style docstrings. Our automated documentation tools expect that format. All functions and class methods that are expected to be called externally should include a docstring. (And those that aren't should beprefixed with a single underscore.7. Try it in MycroftLingua Franca is installed by default when you install Mycroft-core, but for development you generally have this repo cloned elsewhere on your computer. You can use your changes in Mycroft by installing it in the Mycroft virtual environment.If you added the Mycroft helper commands during setup you can just use:mycroft-pipinstall/path/to/your/lingua-francaOtherwise you need to activate that venv manually:cd~/mycroft-coresourcevenv-activate.sh
pipinstall/path/to/your/lingua-francaNow, when talking with Mycroft, it will be using your development version of Lingua Franca.8. Commit changesMake commits in logical units, and describe them thoroughly. If addressing documented issue, use the issue identifier at the very beginning of each commit. For instance:gitcommit-m"Issues-123 - Fix 'demain' date extraction in French"9. Submit a PROnce your changes are ready for review,create a pull request.Like commit messages, the PR title and description should properly describe the changes you have made, along with any additional information that reviewers who do not speak your language might need to understand.10. Waiting for a reviewWhile you wait for a review of your contribution, why not take a moment to review some other pull requests? This is a great way to learn and help progress the queue of pull requests, which means your contribution will be seen more quickly!CreditsThough it is now a standalone package, Lingua Franca's codebase was a spinoff from Mycroft-core. In addition to those represented in Lingua Franca's git log, a great many people contributed to this code before the spinoff.Although all are listed in MycroftAI'sList of Excellent People, it seems proper to acknowledge the specific individuals who helped writethispackage, since they are no longer represented ingit log.To the best of the maintainers' knowledge, all of the "lost" contributors are listed inpre-spinoff-credits.md. Names are listed as they appeared ingit log, or as they are known to the Mycroft community.Those who've contributed since the spinoff are, of course, in Lingua Franca'sgit logand the GitHub "Contributors" pane. All contributors are on the List of Excellent People, regardless of when they contributed.If you contributed to the original code, and your name is missing frompre-spinoff-credits.md, please inform a maintainer or file an issue, so we can give credit where credit is due!