Package: concrete
Concrete-python is the Python interface toConcrete, a
natural language processing data format and set of service protocols
that work across different operating systems and programming languages
viaApache Thrift.  Concrete-python contains generated Python
classes, utility classes and functions, and scripts.  It does not contain the
Thrift schema for Concrete, which can be found in theConcrete GitHub repository.This document provides a quick tutorial of concrete-python installation and
usage.  For more information, including an API reference and development
information, please see theonline documentation.Table of ContentsLicenseRequirementsInstallationBasic usageReading ConcreteWriting Concreteconcrete-inspect.py–id–metadata–sections–entities–mentions–situations–treebankOther optionscreate-comm.pyOther scriptsWorking with UUIDsValidating Concrete CommunicationsLicenseCopyright 2012-2019 Johns Hopkins University HLTCOE. All rights
reserved.  This software is released under the 2-clause BSD license.
Please seeLICENSEfor more information.Requirementsconcrete-python is tested on Python 3.5 and requires the
Thrift Python library, among other Python libraries.  These are
installed automatically bysetup.pyorpip.  The Thrift
compiler isnotrequired.Note: The accelerated protocol offers a (de)serialization speedup
of 10x or more; if you would like to use it, ensure a C++ compiler is
available on your system before installing concrete-python.
(If a compiler is not available, concrete-python will fall back to the
unaccelerated protocol automatically.)  If you are on Linux, a suitable
C++ compiler will be listed asg++orgcc-c++in your package
manager.If you are using macOS Mojave with the Homebrew package manager
(https://brew.sh), you can install the accelerated protocol using
the scriptinstall-mojave-homebrew-accelerated-thrift.sh.InstallationYou can install Concrete using thepippackage manager:pip install concreteor by cloning the repository and runningsetup.py:git clone https://github.com/hltcoe/concrete-python.git
cd concrete-python
python setup.py installBasic usageHere and in the following sections we make use of an example Concrete
Communication file included in the concrete-python source distribution.
TheCommunicationtype represents an article, book, post, Tweet, or
any other kind of document that we might want to store and analyze.
Copy it fromtests/testdata/serif_dog-bites-man.concreteif you
have the concrete-python source distribution or download it
separately here:serif_dog-bites-man.concrete.First we use theconcrete-inspect.pytool (explained in more detail
in the following section) to inspect some of the contents of the
Communication:concrete-inspect.py --text serif_dog-bites-man.concreteThis command prints the text of the Communication to the console.  In
our case the text is a short article formatted in SGML:<DOC id="dog-bites-man" type="other">
<HEADLINE>
Dog Bites Man
</HEADLINE>
<TEXT>
<P>
John Smith, manager of ACMÉ INC, was bit by a dog on March 10th, 2013.
</P>
<P>
He died!
</P>
<P>
John's daughter Mary expressed sorrow.
</P>
</TEXT>
</DOC>Now run the following command to inspect some of the annotations stored
in that Communication:concrete-inspect.py --ner --pos --dependency serif_dog-bites-man.concreteThis command shows a tokenization, part-of-speech tagging, named entity
tagging, and dependency parse in aCoNLL-like columnar format:INDEX       TOKEN   POS     NER     HEAD    DEPREL
-----       -----   ---     ---     ----    ------
1   John    NNP     PER     2       compound
2   Smith   NNP     PER     10      nsubjpass
3   ,       ,
4   manager NN              2       appos
5   of      IN              7       case
6   ACMÉ    NNP     ORG     7       compound
7   INC     NNP     ORG     4       nmod
8   ,       ,
9   was     VBD             10      auxpass
10  bit     NN              0       ROOT
11  by      IN              13      case
12  a       DT              13      det
13  dog     NN              10      nmod
14  on      IN              15      case
15  March   DATE-NNP                13      nmod
16  10th    JJ              15      amod
17  ,       ,
18  2013    CD              13      amod
19  .       .

1   He      PRP             2       nsubj
2   died    VBD             0       ROOT
3   !       .

1   John    NNP     PER     3       nmod:poss
2   's      POS             1       case
3   daughter        NN              5       dep
4   Mary    NNP     PER     5       nsubj
5   expressed       VBD             0       ROOT
6   sorrow  NN              5       dobj
7   .       .Reading ConcreteThere are even more annotations stored in this Communication, but for
now we move on to demonstrate handling of the Communication in Python.
The example file contains a single Communication, but many (if
not most) files contain several.  The same code can be used to read
Communications in a regular file, tar archive, or zip
archive:from concrete.util import CommunicationReader

for (comm, filename) in CommunicationReader('serif_dog-bites-man.concrete'):
    print(comm.id)
    print()
    print(comm.text)This loop prints the unique ID and text (the same text we saw
before) of our one Communication:tests/testdata/serif_dog-bites-man.xml

<DOC id="dog-bites-man" type="other">
<HEADLINE>
Dog Bites Man
</HEADLINE>
<TEXT>
<P>
John Smith, manager of ACMÉ INC, was bit by a dog on March 10th, 2013.
</P>
<P>
He died!
</P>
<P>
John's daughter Mary expressed sorrow.
</P>
</TEXT>
</DOC>In addition to the general-purposeCommunicationReaderthere is a
convenience function for reading a single Communication from a regular
file:from concrete.util import read_communication_from_file

comm = read_communication_from_file('serif_dog-bites-man.concrete')Communications are broken intoSections, which are in turn broken
intoSentences, which are in turn broken intoTokens(and that’s
only scratching the surface).  To traverse this decomposition:from concrete.util import lun, get_tokens

for section in lun(comm.sectionList):
    print('* section')
    for sentence in lun(section.sentenceList):
        print('  + sentence')
        for token in get_tokens(sentence.tokenization):
            print('    - ' + token.text)The output is:* section
* section
  + sentence
    - John
    - Smith
    - ,
    - manager
    - of
    - ACMÉ
    - INC
    - ,
    - was
    - bit
    - by
    - a
    - dog
    - on
    - March
    - 10th
    - ,
    - 2013
    - .
* section
  + sentence
    - He
    - died
    - !
* section
  + sentence
    - John
    - 's
    - daughter
    - Mary
    - expressed
    - sorrow
    - .Here we usedget_tokens, which abstracts the process of extracting
a sequence ofTokensfrom aTokenization, andlun, which
returns its argument or (if its argument isNone) an empty list
and stands for “list un-none”.  Many fields in Concrete are optional,
includingCommunication.sectionListandSection.sentenceList;
checking forNonequickly becomes tedious.In this Communication the tokens have been annotated with
part-of-speech tags, as we saw previously usingconcrete-inspect.py.  We can print them with the following code:from concrete.util import get_tagged_tokens

for section in lun(comm.sectionList):
    print('* section')
    for sentence in lun(section.sentenceList):
        print('  + sentence')
        for token_tag in get_tagged_tokens(sentence.tokenization, 'POS'):
            print('    - ' + token_tag.tag)The output is:* section
* section
  + sentence
    - NNP
    - NNP
    - ,
    - NN
    - IN
    - NNP
    - NNP
    - ,
    - VBD
    - NN
    - IN
    - DT
    - NN
    - IN
    - DATE-NNP
    - JJ
    - ,
    - CD
    - .
* section
  + sentence
    - PRP
    - VBD
    - .
* section
  + sentence
    - NNP
    - POS
    - NN
    - NNP
    - VBD
    - NN
    - .Writing ConcreteWe can add a new part-of-speech tagging to the Communication as well.
Let’s add a simplified version of the current tagging:from concrete.util import AnalyticUUIDGeneratorFactory, now_timestamp
from concrete import TokenTagging, TaggedToken, AnnotationMetadata

augf = AnalyticUUIDGeneratorFactory(comm)
aug = augf.create()

for section in lun(comm.sectionList):
    for sentence in lun(section.sentenceList):
        sentence.tokenization.tokenTaggingList.append(TokenTagging(
            uuid=aug.next(),
            metadata=AnnotationMetadata(
                tool='Simple POS',
                timestamp=now_timestamp(),
                kBest=1
            ),
            taggingType='POS',
            taggedTokenList=[
                TaggedToken(
                    tokenIndex=original.tokenIndex,
                    tag=original.tag.split('-')[-1][:2],
                )
                for original
                in get_tagged_tokens(sentence.tokenization, 'POS')
            ]
        ))Here we usedAnalyticUUIDGeneratorFactory, which creates generators of
ConcreteUUIDobjects (seeWorking with UUIDsfor more information).
We also usednow_timestamp, which returns a Concrete timestamp representing
the current time.  But now how do we know which tagging is ours?  Each
annotation’s metadata contains atoolname, and we can use it to
distinguish between competing annotations:from concrete.util import get_tagged_tokens

for section in lun(comm.sectionList):
    print('* section')
    for sentence in lun(section.sentenceList):
        print('  + sentence')
        token_tag_pairs = zip(
            get_tagged_tokens(sentence.tokenization, 'POS', tool='Serif: part-of-speech'),
            get_tagged_tokens(sentence.tokenization, 'POS', tool='Simple POS')
        )
        for (old_tag, new_tag) in token_tag_pairs:
            print('    - ' + old_tag.tag + ' -> ' + new_tag.tag)The output shows our new part-of-speech tagging has a smaller, simpler
set of possible values:* section
* section
  + sentence
    - NNP -> NN
    - NNP -> NN
    - , -> ,
    - NN -> NN
    - IN -> IN
    - NNP -> NN
    - NNP -> NN
    - , -> ,
    - VBD -> VB
    - NN -> NN
    - IN -> IN
    - DT -> DT
    - NN -> NN
    - IN -> IN
    - DATE-NNP -> NN
    - JJ -> JJ
    - , -> ,
    - CD -> CD
    - . -> .
* section
  + sentence
    - PRP -> PR
    - VBD -> VB
    - . -> .
* section
  + sentence
    - NNP -> NN
    - POS -> PO
    - NN -> NN
    - NNP -> NN
    - VBD -> VB
    - NN -> NN
    - . -> .Finally, let’s write our newly annotated Communication back to disk:from concrete.util import CommunicationWriter

with CommunicationWriter('serif_dog-bites-man.concrete') as writer:
    writer.write(comm)Note there are many other useful classes and functions in theconcrete.utillibrary.  See the API reference in theonline documentationfor details.concrete-inspect.pyUseconcrete-inspect.pyto quickly explore the contents of a
Communication from the command line.concrete-inspect.pyand other
scripts are installed to the path along with the concrete-python
library.–idRun the following command to print the unique ID of our modified
example Communication:concrete-inspect.py --id serif_dog-bites-man.concreteOutput:tests/testdata/serif_dog-bites-man.xml–metadataUse--metadatato print the stored annotations along with their
tool names:concrete-inspect.py --metadata serif_dog-bites-man.concreteOutput:Communication:  concrete_serif v3.10.1pre

  Tokenization:  Serif: tokens

    Dependency Parse:  Stanford

    Parse:  Serif: parse

    TokenTagging:  Serif: names
    TokenTagging:  Serif: part-of-speech
    TokenTagging:  Simple POS

  EntityMentionSet #0:  Serif: names
  EntityMentionSet #1:  Serif: values
  EntityMentionSet #2:  Serif: mentions

  EntitySet #0:  Serif: doc-entities
  EntitySet #1:  Serif: doc-values

  SituationMentionSet #0:  Serif: relations
  SituationMentionSet #1:  Serif: events

  SituationSet #0:  Serif: relations
  SituationSet #1:  Serif: events

  CommunicationTagging:  lda
  CommunicationTagging:  urgency–sectionsUse--sectionsto print the text of the Communication, broken out
by section:concrete-inspect.py --sections serif_dog-bites-man.concreteOutput:Section 0 (0ab68635-c83d-4b02-b8c3-288626968e05)[kind: SectionKind.PASSAGE], from 81 to 82:



Section 1 (54902d75-1841-4d8d-b4c5-390d4ef1a47a)[kind: SectionKind.PASSAGE], from 85 to 162:

John Smith, manager of ACMÉ INC, was bit by a dog on March 10th, 2013.
</P>


Section 2 (7ec8b7d9-6be0-4c62-af57-3c6c48bad711)[kind: SectionKind.PASSAGE], from 165 to 180:

He died!
</P>


Section 3 (68da91a1-5beb-4129-943d-170c40c7d0f7)[kind: SectionKind.PASSAGE], from 183 to 228:

John's daughter Mary expressed sorrow.
</P>–entitiesUse--entitiesto print the named entities detected in the
Communication:concrete-inspect.py --entities serif_dog-bites-man.concreteOutput:Entity Set 0 (Serif: doc-entities):
  Entity 0-0:
      EntityMention 0-0-0:
          tokens:     John Smith
          text:       John Smith
          entityType: PER
          phraseType: PhraseType.NAME
      EntityMention 0-0-1:
          tokens:     John Smith , manager of ACMÉ INC ,
          text:       John Smith, manager of ACMÉ INC,
          entityType: PER
          phraseType: PhraseType.APPOSITIVE
          child EntityMention #0:
              tokens:     John Smith
              text:       John Smith
              entityType: PER
              phraseType: PhraseType.NAME
          child EntityMention #1:
              tokens:     manager of ACMÉ INC
              text:       manager of ACMÉ INC
              entityType: PER
              phraseType: PhraseType.COMMON_NOUN
      EntityMention 0-0-2:
          tokens:     manager of ACMÉ INC
          text:       manager of ACMÉ INC
          entityType: PER
          phraseType: PhraseType.COMMON_NOUN
      EntityMention 0-0-3:
          tokens:     He
          text:       He
          entityType: PER
          phraseType: PhraseType.PRONOUN
      EntityMention 0-0-4:
          tokens:     John
          text:       John
          entityType: PER.Individual
          phraseType: PhraseType.NAME

  Entity 0-1:
      EntityMention 0-1-0:
          tokens:     ACMÉ INC
          text:       ACMÉ INC
          entityType: ORG
          phraseType: PhraseType.NAME

  Entity 0-2:
      EntityMention 0-2-0:
          tokens:     John 's daughter Mary
          text:       John's daughter Mary
          entityType: PER.Individual
          phraseType: PhraseType.NAME
          child EntityMention #0:
              tokens:     Mary
              text:       Mary
              entityType: PER
              phraseType: PhraseType.OTHER
      EntityMention 0-2-1:
          tokens:     daughter
          text:       daughter
          entityType: PER
          phraseType: PhraseType.COMMON_NOUN


Entity Set 1 (Serif: doc-values):
  Entity 1-0:
      EntityMention 1-0-0:
          tokens:     March 10th , 2013
          text:       March 10th, 2013
          entityType: TIMEX2.TIME
          phraseType: PhraseType.OTHER–mentionsUse--mentionsto show the named entitymentionsin the
Communication, annotated on the text:concrete-inspect.py --mentions serif_dog-bites-man.concreteOutput:<ENTITY ID=0><ENTITY ID=0>John Smith</ENTITY> , <ENTITY ID=0>manager of <ENTITY ID=1>ACMÉ INC</ENTITY></ENTITY> ,</ENTITY> was bit by a dog on <ENTITY ID=3>March 10th , 2013</ENTITY> .

<ENTITY ID=0>He</ENTITY> died !

<ENTITY ID=2><ENTITY ID=0>John</ENTITY> 's <ENTITY ID=2>daughter</ENTITY> Mary</ENTITY> expressed sorrow .–situationsUse--situationsto show the situations detected in the
Communication:concrete-inspect.py --situations serif_dog-bites-man.concreteOutput:Situation Set 0 (Serif: relations):

Situation Set 1 (Serif: events):
  Situation 1-0:
      situationType:    Life.Die–treebankUse--treebankto show constituency parse trees of the sentences in
the Communication:concrete-inspect.py --treebank serif_dog-bites-man.concreteOutput:(S (NP (NPP (NNP john)
            (NNP smith))
       (, ,)
       (NP (NPA (NN manager))
           (PP (IN of)
               (NPP (NNP acme)
                    (NNP inc))))
       (, ,))
   (VP (VBD was)
       (NP (NPA (NN bit))
           (PP (IN by)
               (NP (NPA (DT a)
                        (NN dog))
                   (PP (IN on)
                       (NP (DATE (DATE-NNP march)
                                 (JJ 10th))
                           (, ,)
                           (NPA (CD 2013))))))))
   (. .))


(S (NPA (PRP he))
   (VP (VBD died))
   (. !))


(S (NPA (NPPOS (NPP (NNP john))
               (POS 's))
        (NN daughter)
        (NPP (NNP mary)))
   (VP (VBD expressed)
       (NPA (NN sorrow)))
   (. .))Other optionsUse--ner,--pos,--lemmas, and--dependency(together
or independently) to show respective token-level information in a
CoNLL-like format, and use--textto print the text of the
Communication, as described in a previous section.Runconcrete-inspect.py--helpto show a detailed help message
explaining the options discussed above and others.  All
concrete-python scripts have such help messages.create-comm.pyUsecreate-comm.pyto generate a simple Communication from a text
file.  For example, create a file calledhistory-of-the-world.txtcontaining the following text:The dog ran .
The cat jumped .

The dolphin teleported .Then run the following command to convert it to a Concrete
Communication, creating Sections, Sentences, and Tokens based on
whitespace:create-comm.py --annotation-level token history-of-the-world.txt history-of-the-world.concreteUseconcrete-inspect.pyas shown previously to verify the
structure of the Communication:concrete-inspect.py --sections history-of-the-world.concreteOutput:Section 0 (a188dcdd-1ade-be5d-41c4-fd4d81f71685)[kind: passage], from 0 to 30:
The dog ran .
The cat jumped .

Section 1 (a188dcdd-1ade-be5d-41c4-fd4d81f7168a)[kind: passage], from 32 to 57:
The dolphin teleported .Other scriptsconcrete-python provides a number of other scripts, including but not
limited to:concrete2json.pyreads in a Concrete Communication and prints a
JSON version of the Communication to stdout.  The JSON is “pretty
printed” with indentation and whitespace, which makes the JSON
easier to read and to use for diffs.create-comm-tarball.pylikecreate-comm.pybut for multiple files: reads in a tar.gz
archive of text files, parses them into sections and sentences based
on whitespace, and writes them back out as Concrete Communications
in another tar.gz archive.fetch-client.pyconnects to a FetchCommunicationService, retrieves one or more
Communications (as specified on the command line), and writes them
to disk.fetch-server.pyimplements FetchCommunicationService, serving Communications to
clients from a file or directory of Communications on disk.search-client.pyconnects to a SearchService, reading queries from the console and
printing out results as Communication ids in a loop.validate-communication.pyreads in a Concrete Communication file and prints out information
about any invalid fields.  This script is a command-line wrapper
around the functionality in theconcrete.validatelibrary.Use the--helpflag for details about the scripts’ command line
arguments.Working with UUIDsEachUUIDobject contains a single string,uuidString, which can be used as a universally unique identifier for the
object theUUIDis attached to.  TheAnalyticUUIDGeneratorFactoryproducesUUIDgenerators for aCommunication,one for each analytic (tool) used to
process theCommunication.In contrast to the Pythonuuidlibrary, theAnalyticUUIDGeneratorFactoryyields UUIDs that have common prefixes within aCommunicationand within annotations produced by the same analytic, enabling
common compression algorithms to much more efficiently store the UUIDs in eachCommunication.See theAnalyticUUIDGeneratorFactoryclass in the API
reference in theonline documentationfor more information.Note thatuuidStringis generated by
a random process, so running the same code twice will result in two
completely different sets of identifiers.  Concretely, if you run a parser to
produce a part-of-speechTokenTaggingfor eachTokenizationin aCommunication,save the modifiedCommunication,then run the parser again on
the same originalCommunication,you will get two different identifiers for
eachTokenTagging,even though the contents of each pair ofTokenTaggings—the part-of-speech tags—may be the identical.Validating Concrete CommunicationsThe Python version of the Thrift Libraries does not perform any
validation of Thrift objects.  You should use thevalidate_communication()function after reading and before writing
a Concrete Communication:from concrete.util import read_communication_from_file
from concrete.validate import validate_communication

comm = read_communication_from_file('tests/testdata/serif_dog-bites-man.concrete')

# Returns True|False, logs details using Python stdlib 'logging' module
validate_communication(comm)Thrift fields have three levels of requiredness:explicitly labeled asrequiredexplicitly labeled asoptionalno requiredness label given (“default required”)Other Concrete tools will raise an exception if arequiredfield is
missing on deserialization or serialization, and will raise an
exception if a “default required” field is missing on serialization.
By default, concrete-python does not perform any validation of Thrift
objects on serialization or deserialization.  The Python Thrift classes
do provide shallowvalidate()methods, but they only check for
explicitlyrequiredfields (not “default required” fields) and do
not validate nested objects.Thevalidate_communication()function recursively checks a
Communication object for required fields, plus additional checks for
UUID mismatches.