Package: iknowpy
iKnowiKnowis a library for Natural Language Processing that identifies entities (phrases) and their semantic context in natural language text in English, German, Dutch, French, Spanish, Portuguese, Swedish, Russian, Ukrainian, Czech and Japanese. It was originally developed byi.Knowin Belgium, acquired byInterSystemsin 2010 to be embedded in its Caché andIRIS Data Platformproducts. InterSystems published the iKnow engine as open source in 2020.This readme file has everything you need to get started, but make sure you click through to thewikifor more details on any of these subjects.iKnowUsing iKnowFrom PythonFrom C++From InterSystems IRISFrom Different PlatformsUnderstanding iKnowEntitiesAttributesHow it worksBuilding the iKnow EngineContributing to iKnowUsing iKnowFrom PythonThe easiest way to see for yourself what iKnow does with text is by giving it a try! Thanks to ourPython interface, that only takes two simple steps:Usepipto install theiknowpymodule as follows:pipinstalliknowpyFrom your Python prompt, instantiate the engine and start indexing:importiknowpyengine=iknowpy.iKnowEngine()# show supported languagesprint(engine.get_languages_set())# index some texttext='This is a test of the Python interface to the iKnow engine.'engine.index(text,'en')# print the raw resultsprint(engine.m_index)# or make it a little nicerforsinengine.m_index['sentences']:foreins['entities']:print('<'+e['type']+'>'+e['index']+'</'+e['type']+'>',end=' ')print('\n')If you are looking for another programming language or interface, check out the otherAPIs. For more on the Python interface, move on to theGetting Startedsection in the wiki!From C++The main C++ API file isengine.h, defining the classiKnowEnginewith the main entry point:index(TextSource,language)After indexing all data is stored iniknowdata::Text_Source m_index. "iknowdata" is the namespace used for all classes that contain output data. Fore more details, please refer to theAPI overviewon the wiki.From InterSystems IRISFor many years, the iKnow engine has been available as an embedded service on the InterSystems IRIS Data Platform. The obvious advantage of packaging it with a database is that indexing results from many documents can be stored in a single repository, enabling corpus-wide analytics through practical APIs. See theiKnow documentationfor IRIS or browse the InterSystems Developer Community's articles on setting up aniKnow domain,browsing itandusing iFind(iKnow-powered text search)TheInterSystems IRIS Community Editionis available from Docker Hub free of charge.From Different PlatformsSince version 1.3, a C-interface is available, enabling communication with the iKnow engine in a JSON encoded request/response style:constchar*j_response;iknow_json(R"({"method" : "index", "language" : "en", "text_source" : "HelloWorld"})",&j_response);Most API functionality is available in a serialized json format.Understanding iKnowEntitiesiKnow identifies phrase boundaries that defineEntities, entirely based on the syntactic structure of the sentences, rather than relying on an upfront dictionary or pretrained model. This makes iKnow well-suited for initial exploration of a new corpus.
iKnow Entities are notNamed Entitiesin the NER sense, but rather the word groups that need to be considered together, representing a concept or relationship as coined by the text author in its entirety. The following examples clearly show the importance of this phrase level to fully capture what the author meant:iKnow EntityMeaningDopaminesmall moleculeDopamine receptordrug targetDopamine receptor antagonistchemical drugDopamine receptor genegene, molecular sequenceDopamine receptor gene mutationphysiological processiKnow will label every entity with a simple role that is eitherconcept(usually corresponding to Noun Phrases in POS lingo) orrelation(verbs, prepositions, ...). Typical stop words that have little meaning of their own get categorized asPathRelevant(e.g. pronouns) orNonRelevantparts, depending on whether they play a role in the sentence structure or are just linguistic fodder.In the following sample sentence, we've highlightedconcepts, relations andPathRelevantsseparately.Belgian geuzeiswell-knownacross thecontinentforitsdelicate balance.Read more...CRC'sAs of v1.4, the iKnow engine now also produces Concept-Relation-Concept clusters (aka CRC's)Read more...AttributesBeyond this simple phrase recognition, iKnow also captures the context of these entities throughsemantic attributes. Attributes label spans (of entities) within a sentence that share a semantic context. Most attributes start from amarker termand are then, through linguistic rules, expanded left and right as appropriate per the syntactic structure of the sentence. iKnow's main contribution is in this fine-grained expansion, which has been shown to be more accurate than many ML-based techniques.iKnow supports the following attribute types:Negation: iKnow tags all entities participating in a negation, as opposed to an (implied) affirmative context.After discussinghisnausea, the[patientdidn't report suffering fromchest pain,shortnessofbreathortickling].Sentiment: based on a user-supplied list of marker terms, iKnow will identify spans with either a positive or negative sentiment (through separate attributes). Overlapping negation attributes will reverse the sentiment in some language models.[Ilikedthestriped pijamas], but the[slippersdidn't really fit withit].Measurements,Time,FrequencyandDuration: all entities "participating" in an expression of something measurable or time-related will be tagged, enabling efficient recognition of facts in long stretches of natural language text.Uponexam[two weeks ago]the[patient's weightwas146.5 pounds].Certainty: this attribute is a work in progress. See the correspondingwiki sectionfor more details.Some attributes are not available for all languages yet. See thewiki sectionfor more details.How it worksSome InterSystems-era resources on how iKnow works:A recentintroductory videoA not-so-recentplayliston our video channelA fun animation of our unique bottom-up approach inJapaneseandRussian(English version embeddedhere)Read more...Building the iKnow EngineThesource codefor the iKnow engine is written in C++ and includes .sln files for building withMicrosoft Visual Studio 2019 Community Editionand Makefiles for building in Linux/Unix.Please refer tothis wiki pagefor more on the overall build process.Contributing to iKnowYou are welcome to contribute to iKnow's engine code and language models. Check out theWikifor more details on how they work and theIssuesandProjectssections for any particular work on the horizon.