Package: entitykb
💡 EntityKBPython toolkit for the rapid development of custom knowledge bases.EntityKB Documentation:https://www.entitykb.orgEntityKB Code Repository:https://github.com/genomoncology/entitykbEntityKB Python Package:https://pypi.org/project/entitykb/OverviewEntityKB is a toolkit for rapidly developing knowledge bases (i.e.knowledge graphs)
using the Python programming language. It combines an text processing
pipeline with a graph data store to support rules/string-based
entity extraction and linking.Potential Use CasesEntity Extraction: Pull concepts from unstructured text using using
keyword and pattern matching.Entity Linking: Map concepts to a knowledge graph for "semantic
searching" capabilities for recommendation systems, Q&A, data
harmonization, and validated data entry (i.e. drop downs, typeahead).Data Set Labeling: Overcome the "cold start" training set problem
by using entity extraction capabilities to generate annotations.Knowledge Curation/Integration: Iteratively add new data types
and sources with simple Python model classes and data loaders.CapabilitiesEntityKB provides a focused set of core capabilities that can be
extended and enhanced:Graph data storefor storing of entities (nodes) and their
relationships (edges) as plain-old python objects.Terms and Edge indexfor efficient retrieval and traversal of
entities using their name, synonyms, and relationships.Processing pipelinethat normalizes and tokenizes text and then
resolves entities using string, regex, grammar or python based matching.Searchingwith fluent, pythonic traversal query builder for walking
and filtering graph nodes and their relationships.Importing and exportingof data with CLI tooling and/or Python code.Multiple interfacesincluding embedded Python client, RPC/HTTP servers
and CLI.PrioritiesEntityKB's core goal is to enable the rapid, iterative development of custom
knowledge graphs using Python. The followingquality attributeshave been prioritized in furtherance of this goal:Evolvability: Add, update and remove entity types and
data sources without time-consuming data migrations.Configurability: Activate and deactivate out-of-the-box components and
custom code by editing a simple JSON file.Interoperability: Interact via command-line, RPC, HTTP, or in-memory
Python library based on evolving project needs.Understandability: Create new entity classes, contextual labels/verbs,
and custom resolvers with domain specific language and concepts.Portability: Code and data created for EntityKB should be
transferable to a new technology stack with minimal effort.LimitationsEntityKB is deliberately limited in scope to minimize complexity.
Below are some choices that users should be aware of upfront:Not secure: EntityKB has no authentication or authorization
capabilities. RPC and HTTP services should not be exposed to
untrusted clients. Instead, proxy EntityKB behind your application's
security layer. Also, knowledge bases are stored using pickles
which are not secure, only unpickle data from trusted sources.Python-based: The most critical data structures are powered by
libraries written in the C programming language for high-performance
and tight memory management. However, this library is still mostly
Python-based and will certainly not meet everyone's runtime needs.Not transactional: EntityKB is not designed for ACID-compliant
data storage and should never be used as the "system of record".
EntityKB can be updated during runtime, but care should be taken to
prevent data loss or corruption.Not ML based: EntityKB is a software development platform
without any out-of-the-box machine learning capabilities. However, it
certainly can be used in larger ML-based projects and custom resolvers
can be added that use ML models for their entity detection logic.Not Resilient: EntityKB graph searching capability has no
guards against long-running queries that can impact system
responsiveness. Limit end user's from creating open-ended queries to
prevent service disruption.Getting StartedInstall$pipinstallentitykbInitializeEntityKBinitcreates a KB in the specified "root" directory. The root
directory is determined using the following priorities:Command-line argumentEnvironment variable (ENTITYKB_ROOT)Default path (~/.entitykb)Below are theinitandinfocommands using the default path. Notice
the default configuration specifies implementation classes that can be
overridden using theconfig.jsonfile. Theindex.dbcontains the graph
and terms index data in python pickle format and can be deployed with the
config.json to any server using the same version of EntityKB.$ entitykb init
INFO:     Initialization completed successfully.

$ ls ~/.entitykb/
config.json
edges
edges.dawg
nodes
nodes.dawg

$ cat ~/.entitykb/config.json
{
    "graph": "entitykb.Graph",
    "modules": [],
    "normalizer": "entitykb.LatinLowercaseNormalizer",
    "searcher": "entitykb.DefaultSearcher",
    "terms": "entitykb.TermsIndex",
    "tokenizer": "entitykb.WhitespaceTokenizer",
    "pipelines": {
        "default": {
            "extractor": "entitykb.DefaultExtractor",
            "resolvers": [
                "entitykb.TermResolver"
            ],
            "filterers": []
        }
    }
}

$ entitykb info
+------------------------------------+-----------------------------------+
| config.graph                       |                    entitykb.Graph |
| config.modules                     |                                   |
| config.normalizer                  | entitykb.LatinLowercaseNormalizer |
| config.pipelines.default.extractor |         entitykb.DefaultExtractor |
| config.pipelines.default.filterers |                                   |
| config.pipelines.default.resolvers |             entitykb.TermResolver |
| config.root                        |        /Users/ianmaurer/.entitykb |
| config.searcher                    |          entitykb.DefaultSearcher |
| config.terms                       |               entitykb.TermsIndex |
| config.tokenizer                   |      entitykb.WhitespaceTokenizer |
| entitykb.version                   |                          20.12.14 |
| graph.edges                        |                                 0 |
| graph.nodes                        |                                 0 |
+------------------------------------+-----------------------------------+InteractStart a new Knowledge Base and add two entities:>>>fromentitykbimportKB,Entity>>>kb=KB()>>>nys=Entity(name="New York",label="STATE")>>>kb.save_node(nys)>>>nyc=Entity(name="New York City",label="CITY",synonyms=["NYC"])>>>kb.save_node(nyc)Entity(key='New York City|CITY',label='CITY',data=None,name='New York City',synonyms=('NYC',))Edge objects can be created using the Edge class or the right/left-shift
methods on nodes. TheVerb(or short-handV) can be used or so can
a simple string. The library converts all verbs to uppercase:>>>fromentitykbimportEdge,Verb,V>>>Edge(start="New York City|CITY",verb=V.IS_IN,end=nys)Edge(start='New York City|CITY',verb='IS_IN',end='New York|STATE')>>>nyc>>Verb.IS_IN>>nysEdge(start='New York City|CITY',verb='IS_IN',end='New York|STATE')>>>nys<<V.is_in<<nycEdge(start='New York City|CITY',verb='IS_IN',end='New York|STATE')>>>nys<<"is_in"<<"New York City|CITY"Edge(start='New York City|CITY',verb='IS_IN',end='New York|STATE')Edges still need to be saved to the graph which can be accomplished using
thesave_edgeorconnectmethods:>>>edge=nyc>>Verb.IS_IN>>nys>>>kb.save_edge(edge)>>>kb.connect(start=nyc,verb="IS_IN",end=nyc)Edge(start='New York|STATE',verb='IS_IN',end='New York City|CITY')To index the terms and edges of the Graph store, you must first run thereindexmethod on the KB:>>>kb.reindex()Perform term search using common prefix text:>>>response=kb.search("New Y")>>>len(response)2>>>response[0]Entity(key='New York|STATE',label='STATE',data=None,name='New York',synonyms=())>>>response[1]Entity(key='New York City|CITY',label='CITY',data=None,name='New York City',synonyms=('NYC',))Parse text into a document with tokens and spans containing entities:>>>doc=kb.parse("NYC is another name for New York City")>>>len(doc.tokens)8>>>doc.spans(NYC,NewYorkCity)>>>doc.entities(Entity(key='New York City|CITY',label='CITY',data=None,name='New York City',synonyms=('NYC',)),Entity(key='New York City|CITY',label='CITY',data=None,name='New York City',synonyms=('NYC',)))The dump command generates JSONL:$entitykbdump{"kind":"node","payload":{"key":"New York|STATE","label":"STATE","data":null,"name":"New York","synonyms":[]}}{"kind":"node","payload":{"key":"New York City|CITY","label":"CITY","data":null,"name":"New York City","synonyms":["NYC"]}}{"kind":"edge","payload":{"__root__":["New York City|CITY","IS_IN","New York|STATE"]}}Dump and load the data for safe transfer to a different version of EntityKB:$entitykbdump/tmp/out.jsonl
$wc-l/tmp/out.jsonl2$entitykbclear
Areyousureyouwanttoclear:/Users/ianmaurer/.entitykb/index.db?[y/N]:y
INFO:Clearcompletedsuccessfully.

$# do version switch here.$$entitykbload/tmp/out.jsonl
Loaded2in0.02s[/tmp/out.jsonl,jsonl]Reindexedin0.01sBackgroundHistoryEntityKB was developed byGenomOncologyand is the foundation of the clinical, molecular and genomic knowledge
base that power GenomOncology'signiteIQ data extraction
platformandclinical decision
support API suite. EntityKB
was released as an open source library in November 2020 for the
benefit of GenomOncology's clients and the greater open source
community.MaintainerThe initial version of EntityKB was designed and implemented by Ian Maurer
who is the Chief Technology Officer (CTO) for GenomOncology. Ian has over
20 years of industry experience and is the architect of GenomOncology'signiteIQ data extract platformandAPI Suitethat powers GenomOncology'sPrecision Oncology Platform.Ian can be contacted viaTwitter,LinkedIn, or email
(ian -at- genomoncology.com).Related ProjectsEntityKB was inspired by and is powered by several other projects in the
open source community. Below are the most salient examples:DAWGis used for indexing edge relationships (triples), terms, and labels. It
supports "completion" of terms based on a prefix.DiskCacheis used for storing nodes and edges to disk as python objects.Pydanticfor model
annotations, schema definition and FastAPI documentation.Typerpowers EntityKB's
Command Line Interface (CLI) tool.FastAPIpowers EntityKB's
HTTP Application Programming Interface (API).uvicornandStarlettefor power running FastAPI.MkDocs,Termynal.js, andMaterial for MkDocsfor making the documentation look great.Larkfor powering the date
resolver's grammar.FlashTextfor inspiring
parts of EntityKB's design and approach.LicenseThis project is copyrighted byGenomOncologyand licensed under the terms of the MIT license.StatusEntityKB should be considered beta software. Some caveats:Expect backwards incompatible changes that will break your Knowledge Base.
When that happens, run the following steps:
1. Using the old version of EntityKB, run thedumpCLI command to
generate a JSONL file of your edges and nodes.
2. Switch to the new version of EntityKB, and run theloadCLI command
to import the edges and nodes from JSONL.Monitor changes in therelease notes.To minimize frustration, please pin the version of the software in
your requirements.txt or equivalent file.ContributingSubmit bugs and enhancement suggestions viaGitHub issues.Contributions welcome, please seeDevelopmentfor
setting up a working dev environment.Our goal is to keep EntityKB's code footprint as small as possible
butcontribmodules will be more readily accepted.Separate packages don't require a pull request, but please follow the
naming patternentitykb-<name>to aid in discoverability on pypi.