Package: pytakes
pytakesSimple entity extraction module released under the MIT license.OverviewThis module will look for a pre-defined set of terms in a corpus of text, and use a variation of the negex/context
algorithm to determine whether these terms express negation, historical, or various other qualifiers. (The set of
negation terms is also configurable.)RequirementsPython 3.8+See requirements.txt (pip install -r requirements.txt)Various requirements-_.txt files are provided depending on your needs:dev: for running tests, general developmentdb: for connecting to database using pyodbcpsql: connecting to postgres databasesas: if data is stored in SASPrerequisitesGenerate a word list of terms/concepts ('concept dictionary')in pyTAKES, a 'concept' is a set of terms with more or less the same meaning (e.g., ckd, chronic kidney disease)the minimal should be a CSV file with three columns:id - unique int for each linecui - string label for a 'concept' ('concept unique identifier')text - text to look fordictionary builder script is also provided which help generate variations of terms(documented below)A corpus with an id (for tracking, this will be in output) and text field (for processing, extracting concepts)DocoBasicsThe entry point ispython example/run.py config.py.You can see an exampleconfig.pyatexample/simple/example.config.pypytakesmodule must be on your PYTHONPATH, soset/export PYTHONPATH=srcprior to runningInstallClone from git repo:git clone ...pytakes.gitcd pytakes(optional) build virtualenvPYTHON_INSTALL/Scripts/virtualenv .venvpip install virtualenvif not yet availablePip install prerequisitespip install -r requirements.txtRun tests (pytest tests)UseYou will need to have an inputconcepts.csvfile with at least three columns (id,cui,text). There are several
examples in thepytakes/tests/datadirectory.Negation TableThis table implements a modified version of Chapman's ConText (see,
e.g.,http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.145.6566&rep=rep1&type=pdf,
andhttps://code.google.com/archive/p/negex/).This table is loosely based on the csv file
here:https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/negex/lexical_kb.csvNegation works by searching for the following word categories and then using a set of rules to determine which words
should be 'qualified' in different ways. Within the recommendedjsonloutput, you'll find this included under
thequalifierskey. Consider the following example:{..."qualifiers":{"certainty":0,"hypothetical":true,"historical":false,"other_subject":false,"terms":["not","afraid"]},...}All values begin asFalseexceptcertaintywhich starts as 4, meaningdefinite/affirmative. Here,certaintyis
0, meaningnegated. This was done by the term 'not' which appears in the list ofterms.hypotheticalhas been
marked asTruedue to the termafraid(often in a context like 'afraid that he will catch a cold').Certainty:NumberInterpretation0negated1improbable2possible3probable4definiteColumns:negex: negation (or related) term; capitalization and punctuation will be normalized (i.e., removed) so just include
letters; I don't think regexes worktype: four letter abbreviation for negation role with brackets (these will vary based on your text and what you want
to extract)[IMPR]: improbable words (e.g., 'low probability')setscertaintyto 1[NEGN]: negation words (e.g., 'denies')setscertaintyto 0[PSEU]: pseudonegation (e.g., 'not only')[INDI]: indication (e.g., 'rule out')[HIST]: historical (e.g., 'previous')setshistoricaltoTrue[CONJ]: conjunction - interferes with negation scope (e.g., 'though', 'except')[PROB]: probable (e.g., 'appears')setscertaintyto 3[POSS]: possible (e.g., 'possible')setscertaintyto 2[HYPO]: hypothetical (e.g., 'might')setshypotheticaltoTrue[OTHR]: other subject - refers to someone other than the subject (e.g., 'mother')setsother_subjecttoTrue[SUBJ]: subject - when reference of OTHR is still referring to the subject (e.g., 'by patient mother')[PREN]: prenegation <- not sure if this is supposed to be used[AFFM]: affirmed (e.g., 'obvious', 'positive for')[FUTP]: future possibility (e.g., 'risk for')direction0: directionality doesn't make sense (e.g., CONJ)1: term applies negation, etc.backwardin the sentence (e.g., 'not seen')2: term applies negation, etc.forwardin the sentence (e.g., 'dont see')3: term applies negation, etc.forward and/or backwardin the sentence (e.g., 'likely')Concept/Term TableThis is the table containing the terms you want to search for (i.e., the entities you want extracted). I have added a
script to autogenerate these based on some basic configuration files.ColumnTypeDescriptionIDintidentity column; unique integer for each rowCUI	 stringcategory identifier; can be used to "group" different     terms togetherTextstringtermRegexVariationintamount of variation: 0=none; 3=very; 1=default; -1=don't even allow suffixes, exact matches only; see #Rules#parameters below; I suggest you just use "0" or "-1"WordOrderinthow accurate must the given word order be; 2=exactly; 1=fword constraint; 0=no word orderMaxInterveninginthow many intervening words to allow when locating words; 'how many intervening words do I allow?'MaxWordsinthow many words to look ahead to find the next word; is ‘how far do I look ahead after each term?’MaxIntervening and MaxWords should not be used together.To autogenerate this table format, use thepytakes-build-dictionaryscript installed into the Python Scripts
directory. For an example, run this with the--create-sampleoption (optionally specify the output with
the--path C:\somewhereoption. For additional specifications, see the "Dictionary Builder" section below.Document TableThis is the table containing the text you are in interesting in searching in.The text itself must currently be labeled 'note_text'. The option to specify this is currently not implemented. Sorry.The document table must also include a unique id for each note_text (just make an autoincrementing primary key). Specify
this and any other meta information you want to pass along under--meta-labelsoption (ensure that the unique doc_id
is specified first).Example Config FileI prefer to specify the configuration file as a Python file (config.py). Yaml and json are also accepted. See an
example below (copied fromexample.config.py). Please note that theprint(config) at the end is required.config = {
    'corpus': {  # how to get the text data
        'directories': [  # specify path to .txt files
            r'PATH'
        ],
        'connections': [  # specify other connection types
            {
                'name': 'TABLENAME',
                'name_col': 'TEXT ID COLUMN',
                'text_col': 'TEXT COLUMN',
                # specify either driver/server/database OR connection_string
                # connection string examples here: https://docs.sqlalchemy.org/en/13/core/engines.html
                'connection_string': 'SQLALCHEMY-LIKE CONNECTION_STRING',
                # db args: driver/server/database
                'driver': 'DRIVER',  # available listed in pytakes/iolib/sqlai.py, or use connection string
                'server': 'SERVER',
                'database': 'DATABASE',
            }
        ]
    },
    'keywords': [  # path to keyword files, usually stored as CSV
        {
            'path': r'PATH',
            'regex_variation': 0  # set to -1 if you don't want any expansion
        }
    ],
    'negation': {  # select either version or path (not both)
        'path': r'PATH TO NEGATION CSV FILE',
        'version': 1,  # int (version number), built-in/default
        'skip': False,  # bool: if skip is True: don't do negation
    },
    'output': {
        'path': r'PATH TO OUTPUT DIRECTORY',
        'outfile': 'NAME.out.jsonl',  # name of output file (or given default name)
        'hostname': ''
    },
}

print(config)Dictionary BuilderFor a simple example, run (you will first need to install this package, runpython setup.py installin the base
directory):pytakes-build-dictionary --create-sample --path OUTPUT_PATHCOMMAND LINE ARGUMENTSShortLongDescription-p--pathSpecifies parent directory of folders; program will prompt if unable to locate the directory-o--outputSpecify output CSV file; if ".csv" is not included, it will be added-t--tableSpecify output table in specified database (See below)-v--verbositySpecify amount of log output to show: 3-most verbose; 0-least verbose--driverIf -t is specified, driver where table should be created. Defaults to SQL Server--serverIf -t is specified, server where table should be created.--databaseIf -t is specified, database where table should be created.OUTPUT COLUMNSNot all of these output columns are required (most don't do anything). This was originally designed for building a
dictionary using cTAKES.ColumnTypeDescriptionIDintidentity column; unique integer for each rowCUIvarchar(8)category identifier; can be used to "group" different terms togetherFwordvarchar(80)first word of termTextvarchar(8000)termTextLengthintlength of term (all characters including spaces)RegexVariationintamount of variation: 0=none; 3=very; 1=default; see #Rules#parameters belowWordOrderinthow accurate must the given word order be; 2=exactly; 1=fword constraint; 0=no word orderValenceintthis should just be "1"; program is not designed to work with this correctlyRULESRules are the text entries in the cTAKES-like dictionary, however, they can include "categories" in addition to just
text.A category is any string of text surrounded by "[" and "]". The intervening text string is the name of a "
category". The category must have a definition, and each item (synonym) in the definition will be used in the rule.For example, if a rule is[smart_person] is smartand the categorysmart_personis defined by the terms "Albert
Einstein", "Old McDonald", and "Brain", then the resulting output will beAlbert Einsten is smart
Old McDonald is smart
Brain is smartThe rule file consists of a set of rules (as above), and each rule must be on its own line.[smart_person] is smart
smart [smart_person]
[smart_person] not dumb
[not_so_smart] not smartThe rule file must be named "rules" or "rules.some-extension" (e.g., "rules.txt").ParametersRules may also maintain configuration parameters.The configurations are indexed in the following order (bold indicates
the default parameter):RegexVariationDescription0(Default)no variation in regular expression coverage1minimal variation in regular expression coverage2moderate variation in regular expression coverage3high flexibility in regular expression coverageWordOrderDescription0free word order1(Default)enforce first word rule2require precise word orderValenceDescription1(Default)Always use 1These are designated by the double percent ('%%') and follow the rule.[category]%%REGEX_VARIATION,WORD_ORDER,VALENCEFor example:[smart_person] is smart%%1,2   # minimal regex variation; requires precise word order 
[smart_person]%%,2        # same as above: the first parameter is left blank, and the default is used
[smart_person] not dumb           # default for both parameters are used
[not_so_smart] not smart%%2    # default second parameterDefinitions/categories (see below) can also be assigned parameters in exactly the same way. When the parameters
collide/disagree (e.g., the rule asks for free word order, but the definition asks that the first word rule be
enforced), the more conservative will be selected.DEFINITIONSThe definition (also called "category" files) provide a set of words to replace the name of a category in a particular
rule.The definition file must either be within a "cat" directory, or must have the extension ".cat". The program will
choose one or the other--which one is undefined.There are several ways to write the definition for a particular
category. Examples:In the definition file smart_person.cat, each row will be considered a definition for the smart_person category. Also a
not_so_smart.cat should be included.
In the definition file definitions.cat:[smart_person]                       
Albert Einstein                       
Old McDonald                       
Brain                       
[not_so_smart]                       
Humpty DumptyComments.All lines beginning with "#" are ignored, and all characters occurring after a '#' are ignored as comments.# last updated by me, yesterday evening                       
[smart_person]                       
Albert Einstein       # comment here                       
Old McDonald                       
Brain                       
[not_so_smart]                       
# shouldn't there be others?!?                       
Humpty DumptyCUIs.CUIs are usually assigned uniquely for each rule, rather than for a category. A CUI can be included for a given
definition of a category by assigning it with the syntax: C1025459==Albert Einstein
Or, in the entire definition file:[smart_person]                       
C1025459==Albert Einstein                       
C4495545==Old McDonald                       
Brain                       
[not_so_smart]                       
Humpty DumptyIn the above example, Humpty Dumpty and Brain are both assigned a default CUI.Word Variant Notation.Definitions may also be written on a single line, separated by the double pipe (i.e., '||'). If more than three or four
definitions are listed on a single line, the definitions file becomes somewhat unreadable. Thus, it is best practice to
only include word variants on a single line.[smart_person]                       
C1025459==Albert Einstein||Einstein                       
C4495545==Old McDonald||Ol' McDonald||Ole McDonald||Jeff                       
Brain                       
[not_so_smart]                       
Humpty Dumpty||Humpty-Dumpty # this is a common use for word-variant notationParametersFor definitions, see the parameters section under Rules.Example:[smart_person]                       
C1025459==Albert Einstein%%1,2   # all rules involving this definition with have minimal regex variation; requires precise word order                       
C4495545==Old McDonald                       
Brain                       
[not_so_smart]                       
Humpty Dumpty              # will use the defaultsConflict Resolution.Regardless of how the conflict occurs, the more conservative of the rule and all relevant definitions will be chosen.
NB: This process will never choose values that have been left as default (unless the default is specifically requested).