Package: quantulum3
quantulum3Python library for information extraction of quantities, measurements
and their units from unstructured text. It is able to disambiguate between similar
looking units based on theirk-nearest neighboursin theirGloVevector representation
and theirWikipediapage.This is the Python 3 compatible fork ofrecastrodiaz'
forkofgrhawks'
forkofthe original by Marco
Lagi.
The compatibility with the newest version of sklearn is based on
the fork ofsohrabtowfighi.User GuideInstallationpipinstallquantulum3To install dependencies for using or training the disambiguation classifier, usepipinstallquantulum3[classifier]The disambiguation classifier is used when the parser find two or more units that are a match for the text.Usage>>>fromquantulum3importparser>>>quants=parser.parse('I want 2 liters of wine')>>>quants[Quantity(2, 'litre')]TheQuantityclass stores the surface of the original text it was
extracted from, as well as the (start, end) positions of the match:>>>quants[0].surfaceu'2 liters'>>>quants[0].span(7, 15)Thevalueattribute provides the parsed numeric value and theunit.nameattribute provides the name of the parsed unit:>>>quants[0].value2.0>>>quants[0].unit.name'litre'An inline parser that embeds the parsed quantities in the text is also
available (especially useful for debugging):>>>printparser.inline_parse('I want 2 liters of wine')I want 2 liters {Quantity(2, "litre")} of wineAs the parser is also able to parse dimensionless numbers,
this library can also be used for simple number extraction.>>>printparser.parse('I want two')[Quantity(2, 'dimensionless')]Units and entitiesAll units (e.g.litre) and the entities they are associated to (e.g.volume) are reconciled against WikiPedia:>>>quants[0].unitUnit(name="litre", entity=Entity("volume"), uri=https://en.wikipedia.org/wiki/Litre)>>>quants[0].unit.entityEntity(name="volume", uri=https://en.wikipedia.org/wiki/Volume)This library includes more than 290 units and 75 entities. It also
parses spelled-out numbers, ranges and uncertainties:>>>parser.parse('I want a gallon of beer')[Quantity(1, 'gallon')]>>>parser.parse('The LHC smashes proton beams at 12.8–13.0 TeV')[Quantity(12.8, "teraelectronvolt"), Quantity(13, "teraelectronvolt")]>>>quant=parser.parse('The LHC smashes proton beams at 12.9±0.1 TeV')>>>quant[0].uncertainty0.1Non-standard units usually don't have a WikiPedia page. The parser will
still try to guess their underlying entity based on their
dimensionality:>>>parser.parse('Sound travels at 0.34 km/s')[0].unitUnit(name="kilometre per second", entity=Entity("speed"), uri=None)DisambiguationIf the parser detects an ambiguity, a classifier based on the WikiPedia
pages of the ambiguous units or entities tries to guess the right one:>>>parser.parse('I spent 20 pounds on this!')[Quantity(20, "pound sterling")]>>>parser.parse('It weighs no more than 20 pounds')[Quantity(20, "pound-mass")]or:>>>text='The average density of the Earth is about 5.5x10-3 kg/cm³'>>>parser.parse(text)[0].unit.entityEntity(name="density", uri=https://en.wikipedia.org/wiki/Density)>>>text='The amount of O₂ is 2.98e-4 kg per liter of atmosphere'>>>parser.parse(text)[0].unit.entityEntity(name="concentration", uri=https://en.wikipedia.org/wiki/Concentration)In addition to that, the classifier is trained on the most similar words to
all of the units surfaces, according to their distance inGloVevector representation.Spoken versionQuantulum classes include methods to convert them to a speakable unit.>>>parser.parse("Gimme 10e9 GW now!")[0].to_spoken()ten billion gigawatts>>>parser.inline_parse_and_expand("Gimme $1e10 now and also 1 TW and 0.5 J!")Gimme ten billion dollars now and also one terawatt and zero point five joules!ManipulationWhile quantities cannot be manipulated within this library, there are
many great options out there:pintnatuquantitiesExtensionTraining the classifierIf you want to train the classifier yourself, you will need the dependencies for the classifier (see installation).Usequantulum3-trainingon the command line, the scriptquantulum3/scripts/train.pyor the methodtrain_classifierinquantulum3.classifierto train the classifier.quantulum3-training--lang<language>--data<path/to/training/file.json>--output<path/to/output/file.joblib>You can pass multiple training files in to the training command. The output is in joblib format.To use your custom model, pass the path to the trained model file to the
parser:parser = Parser.parse(<text>, classifier_path="path/to/model.joblib")Example training files can be found inquantulum3/_lang/<language>/train.If you want to create a new or differentsimilars.json, installpymagnitude.For the extraction of nearest neighbours from a vector word representation file,
usescripts/extract_vere.py. It automatically extracts theknearest neighbours
in vector space of the vector representation for each of the possible surfaces
of the ambiguous units. The resulting neighbours are stored inquantulum3/similars.jsonand automatically included for training.The file provided should be in.magnitudeformat as other formats are first
converted to a.magnitudefile on-the-run. Check outpre-formatted Magnitude formatted word-embeddingsandMagnitudefor more information.Additional unitsIt is possible to add additional entities and units to be parsed by quantulum. These will be added to the default units and entities. See below code for an example invocation:>>>fromquantulum3.loadimportadd_custom_unit,remove_custom_unit>>>add_custom_unit(name="schlurp",surfaces=["slp"],entity="dimensionless")>>>parser.parse("This extremely sharp tool is precise up to 0.5 slp")[Quantity(0.5, "Unit(name="schlurp", entity=Entity("dimensionless"), uri=None)")]The keyword arguments to the functionadd_custom_unitare directly translated
to the properties of the unit to be created.Custom Units and EntitiesIt is possible to load a completely custom set of units and entities. This can be done by passing a list of file paths to the load_custom_units and load_custom_entities functions. Loading custom untis and entities will replace the default units and entities that are normally loaded.The recomended way to load quantities is via a context manager:>>>fromquantulum3importload,parser>>>withload.CustomQuantities(["path/to/units.json"],["path/to/entities.json"]):>>>parser.parse("This extremely sharp tool is precise up to 0.5 slp")[Quantity(0.5, "Unit(name="schlurp", entity=Entity("dimensionless"), uri=None)")]>>># default units and entities are loaded againBut it is also possible to load custom units and entities manually:>>>fromquantulum3importload,parser>>>load.load_custom_units(["path/to/units.json"])>>>load.load_custom_entities(["path/to/entities.json"])>>>parser.parse("This extremely sharp tool is precise up to 0.5 slp")[Quantity(0.5, "Unit(name="schlurp", entity=Entity("dimensionless"), uri=None)")]>>># remove custom units and entities and load default units and entities>>>load.reset_quantities()See the Developer Guide below for more information about the format of units and entities files.Developer GuideAdding Units and EntitiesSeeunits.jsonfor the complete list of units andentities.jsonfor
the complete list of entities. The criteria for adding units have been:the unit has (or is redirected to) a WikiPedia pagethe unit is in common use (e.g. not thepremetric Swedish units of
measurement).It's easy to extend these two files to the units/entities of interest.
Here is an example of an entry inentities.json:"speed":{"dimensions":[{"base":"length","power":1},{"base":"time","power":-1}],"URI":"https://en.wikipedia.org/wiki/Speed"}Thenameof an entity is its key. Names are required to be unique.URIis the name of the wikipedia page of the entity. (i.e.https://en.wikipedia.org/wiki/Speed=>Speed)dimensionsis the dimensionality, a list of dictionaries each
having abase(the name of another entity) and apower(an
integer, can be negative).Here is an example of an entry inunits.json:"metre per second":{"surfaces":["metre per second","meter per second"],"entity":"speed","URI":"Metre_per_second","dimensions":[{"base":"metre","power":1},{"base":"second","power":-1}],"symbols":["mps"]},"year":{"surfaces":["year","annum"],"entity":"time","URI":"Year","dimensions":[],"symbols":["a","y","yr"],"prefixes":["k","M","G","T","P","E"]}Thenameof a unit is its key. Names are required to be unique.URIfollows the same scheme as in theentities.jsonsurfacesis a list of strings that refer to that unit. The library
takes care of plurals, no need to specify them.entityis the name of an entity inentities.jsondimensionsfollows the same schema as inentities.json, but thebaseis the name of another unit, not of another entity.symbolsis a list of possible symbols and abbreviations for that
unit.prefixesis an optional list. It can containMetricandBinary prefixesand
automatically generates according units. If you want to
add specifics (like different surfaces) you need to create an entry for that
prefixes version on its own.All fields are case sensitive.Contributingdevbuild:If you'd like to contribute follow these steps:Clone a fork of this project into your workspaceRunpip install -e .at the root of your development folder.pip install pipenvandpipenv shellInside the project folder runpipenv install --devMake your changesRunscripts/format.shandscripts/build.pyfrom the package root directory.Test your changes withpython3 setup.py test(Optional, will be done automatically after pushing)Create a Pull Request when having commited and pushed your changesLanguage supportThere is a branch for language support, namelylanguage_support.
From inspecting theREADMEfile in the_langsubdirectory and
the functions and values given in the new_lang.en_USsubmodule,
one should be able to create own language submodules.
The new language modules should automatically be invoked and be available,
both through thelang=keyword argument in the parser functions as well
as in the automatic unittests.No changes outside the own language submodule folder (i.e._lang.de_DE) should
be necessary. If there are problems implementing a new language, don't hesitate to open an issue.