Package: marve
# MarveA measurement relation extractor[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)Marve leverages [grobid-quantities](https://github.com/kermitt2/grobid-quantities) and [Stanford CoreNLP](http://stanfordnlp.github.io/CoreNLP) to extract and normalize measurements and their related entities and descriptors from natural language text.```shellsample = "The patient returned to Europe at 28 weeks of gestation."#Simplified output#-----------------value: "28"unit: "weeks"quantified: "gestation"related: ["patient", "Europe"]```Marve employs grobid-quantities to find measurement values, units, and a limited set of "quantified" substances using linear CRF models trained on a labeled corpus of text. CoreNLP is then used to link measurements to related words in the sentence using word dependencies and POS tags from CoreNLP. Common dependency/POS patterns relating measurements to other words/entities are specified in `/marve/dependency_patterns.json` and can be adjusted without modifying code.## InstallationRunning Marve requires grobid-quantities and CoreNLP to be running:Download and unzip [CoreNLP](http://stanfordnlp.github.io/CoreNLP/download.html):```shellcurl -LOk "http://nlp.stanford.edu/software/stanford-corenlp-full-2016-10-31.zip" | unzip <path to CoreNLP>/stanford-corenlp-full-2016-10-31.zip```Run (requires Java 8):```shellcd <path to CoreNLP>/stanford-corenlp-full-2016-10-31 | java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer 9000```Install grobid by following instructions [here](http://grobid.readthedocs.io/en/latest/Install-Grobid/) </br></br>Then follow [grobid-quantities instructions](https://github.com/kermitt2/grobid-quantities) to install, build, train, and runInstall Marve:</br>`pip install marve`## UsageOnce both CoreNLP and grobid-quantities are running, Marve can be used as such:```shell# coding: utf-8from marve import Measurements as m# Strings longer than a paragraph should be split before passing to Marve_test = "The patient returned to Europe at 28 weeks of gestation."coreNLP = "http://localhost:9000"grobid = "http://localhost:8080"patterns = "dependency_patterns.json"write_to = "sample_output.txt"m.extract(_test, coreNLP, grobid, patterns, write_to, show_graph=False, pretty=True, simplify=False)````(This example can be found in `sample.py`)<b>IMPORTANT</b>: Text should be in sentence or paragraph chunks before passing to Marve.<b> Note </b>: The first time Marve is run, a timeout error might be thrown due to longer CoreNLP model loading times. If this happens, run again and CorenLP should run properly.## Dependency and Part-of-Speech PatternsMarve will only return words related to measurements if they meet criteria laid out in the dependency pattern file `/marve/dependency_patterns.json`.Take the phrase `"a spatial resolution of 10m"`. Marve uses a graph to represent each sentence, where edges are the dependencies between words (represented in green ovals below) and nodes are words and their part-of-speech (POS) labels (represented in blue).![example](https://github.jpl.nasa.gov/hundman/marve/blob/master/blob/example.png)There are a handful of general patterns that relate measurement units, values, and other related words or entities in a sentence. For instance, units are generally connected to values via the numerical modifier (`"nummod"`) dependency (see above). Nominal modifiers (`"nmod"`) is then a common dependency linking units to the thing being quantified. Common patterns linking values, units, and related words have been defined in `/marve/dependency_patterns.json`, and the bit of JSON that would match `"m"` to `"resolution"` in our above example is:```shell"nmod": {"enhanced": true,"of":{"measurement_types": ["space_between", "attached"],"pos_in":{"NN": null}}}```Here's how this example matches:</br></br>1. `nmod` is the dependency type between `m` and `resolution`</br></br>2. Since we utilized CoreNLP's enhanced dependency parser, we also see `:of` attached on the end of `nmod`. Since enhanced is set to `true`, the `of` must be attached to the dependency</br></br>3. Because the measurement is `10m` we identify it as being an `attached` `measurement_type`</br></br>4. `"pos_in"` forces the part of speech of the attached node to contain at least one of its keys. In this case, "NN" means the part of speech must be a noun (valid POS tags could be: `NN`, `NNS`, `NNP`, `NNPS`). Since `NN`'s value is null, we are finished and can return resolution as a related word and add it to the output. For some POS tags such as `VB`, we might need to continue traversing edges in the graph, in which case the value could specify a function to be called (e.g. `get_cousin()`)</br>All such dependency patterns listed in the JSON will be evaluated and if there are any matching patters, they will be added as related words for a measurement.## API```shellextract(content, corenlp, grobid, dependency_patterns_file, output_file=None, show_graph=False, pretty=False, simplify=False)Returns extracted measurements from a sentence or paragraph.Parameters: 	content: stringSentence or paragraph to extract measurements from.corenlp: stringCoreNLP server endpoint (e.g. "http://localhost:9200").grobid: stringGrobid server endpoint (e.g. "http://localhost:8080").dependency_patterns_file: stringFilepath to JSON file containing valid dependency/POS patterns forextracting words and entities related to measurements.output_file: string, optionalFile to write extracted measurement output to.show_graph: boolean, optionalIf True, a visualization of the dependency and POS network graphwill be displayed for each sentence parsed.pretty: boolean, optionalIf True, JSON written to file will be indented. If False, one extractiondoc per line will be written to the output filesimplify: boolean, optionalIf True, only the measurement, unit, and related words of the extractedoutput will be returned and written to the output file (see 'Output Options'section for more detail).Returns:		dict: see "Output Options" below```## Output Options#### simple=True```shell{"value": 6, "unit": "year", "quantified": {}, "related": {"period": ["study"]}}```#### simple=False, pretty=False```shell{"type": "value","quantity": {"parsedValue": 6,"rawValue": "six","rawUnit": {"offsetStart": 13,"offsetEnd": 14,"tokenIndices": ["3"],"after": " ","name": "year"},"offsetEnd": 131,"offsetStart": 128,"tokenIndex": 24,"type": "time"},"related": [{"rawName": "period","connector": "","offsetEnd": 21,"relationForm": "compound","offsetStart": 15,"tokenIndex": 5,"descriptors": [{"rawName": "year","tokenIndex": "4"}]}]}```## LicenseMarve is distributed under [Apache 2.0 license](http://www.apache.org/licenses/LICENSE-2.0).Contact: Kyle Hundman (<khundman@gmail.com>)## Acknowledgements* [Chris Mattmann](http://sunset.usc.edu/~mattmann/), JPL* Sonny Koliwad, JPL* Jason Hyon, JPL* [Ian Colwell](https://github.com/iancolwell), JPL