Package: skipchunk
SkipchunkEasy search autosuggest with NLP magic.Out of the box it provides a hassle-free autosuggest for any corpus from
scratch, and latent knowledge graph extraction and exploration.Free software: MIT LicenseDocumentation:https://skipchunk.readthedocs.io.Installpipinstallskipchunkpython-mspacydownload'en_core_web_lg'python-mnltk.downloaderwordnetYou also need to have Solr or Elasticsearch installed and running
somewhere!The current Solr supported version is 8.4.1, but it might work on other
versions.The current Elasticsearch supported version is 7.6.2, but it might work
on other versions.Use It!See the./example/folder for an end-to-end OSC blog load:SolrStart Solr first! Doesn’t work with Solr cloud yet, but we’re working on
it.You’ll need to start solr using skipchunk’s solr_home directory for now.Then run this:pythonsolr-blog-example.pyElasticsearchStart Elasticsearch first!Then run this:pythonelasticsearch-blog-example.pyFeaturesIdentifies and groups the noun phrases and verb phrases in a corpusIndexes these phrases in Solr or Elasticsearch for a really good
out-of-the-box autosuggestStructures the phrases as a graph so that
concept-relationship-concept can be easily foundMeant to handle batched updates as part of a full stack search
platformLibrary APIEngine configurationYou need an engine_config, as a dict, to create skipchunk.The dict must contain the following entrieshost (the fully qualified URL of the engine web API endpoint)name (the name of the graph)path (the on-disk location of stateful data that will be kept)engine_name (either “solr” or “elasticsearch”)Solr engine config exampleengine_config_solr={"host":"http://localhost:8983/solr/","name":"osc-blog","path":"./skipchunk_data","engine_name":"solr"}Elasticsearch engine config exampleengine_config_elasticsearch={"host":"http://localhost:9200/","name":"osc-blog","path":"./skipchunk_data","engine_name":"elasticsearch"}Skipchunk InitializationWhen initializing Skipchunk, you will need to provide the constructor
with the following parametersengine_config (the dict containing search engine connection details)spacy_model=“en_core_web_lg” (the spacy model to use to parse text)minconceptlength=1 (the minimum number of words that can appear in a
noun phrase)maxconceptlength=3 (the maximum number of words that can appear in a
noun phrase)minpredicatelength=1 (the minimum number of words that can appear in
a verb phrase)maxpredicatelength=3 (the maximum number of words that can appear in
a verb phrase)minlabels=1 (the number of times a concept/predicate must appear
before it is recognized and kept. The lower this number, the more
concepts will be kept - so be careful with large content sets!)cache_documents=Falsecache_pickle=FalseSkipchunk Methodstuplize(filename=source,fields=['title','content',...])(Produces
a list of (text,document) tuples ready for processing by the
enrichment.)enrich(tuples)(Enriching can take a long time if you provide
lots of text. Consider batching at 10k docs at a time.)save(Saves to pickle)load(Loads from pickle)Graph APIAfter enrichment, you can then index the graph into the engineindex(skipchunk:Skipchunk)(Updates the knowledge graph in the
search engine)delete(Deletes a knowledge graph - be careful!)After indexing, you can call these methods to get autocompleted concepts
or walk the knowledge graphconceptVerbConcepts(concept:str,verb:str,mincount=1,limit=100)->list( Accepts a verb to find the concepts appearing in the same context)conceptsNearVerb(verb:str,mincount=1,limit=100)->list( Accepts
a verb to find the concepts appearing in the same context)verbsNearConcept(concept:str,mincount=1,limit=100)->list(
Accepts a concept to find the verbs appearing in the same context)suggestConcepts(prefix:str,build=False)->list( Suggests a list
of concepts given a prefix)suggestPredicates(prefix:str,build=False)->list( Suggests a
list of predicates given a prefix)summarize(mincount=1,limit=100)->list( Summarizes a core)graph(subject:str,objects=5,branches=10)->list( Gets the
subject-predicate-object neighborhood graph for a subject)CreditsDeveloped by Max Irwin, OpenSource Connectionshttps://opensourceconnections.comAll the blog posts contained in the example directory are copyright
OpenSource Connections, and may not be used nor redistributed without
permissionHistory0.1.0 (2019-06-18)Cookie-cutted.0.9.0 (2020-09-25)First release on PyPI.1.0.0 (2020-12-10)Stable API.1.1.0 (2020-12-10)Beta Release.1.1.1 (2020-12-10)Basic Readme doc.1.2.2 (2020-12-14)Configset path fix.Static assets moved.2.0.0 (2021-09-09)SpaCy upgraded to 3.1.2.Batch multiprocess processing for better throughput2.0.1 (2021-09-09)Fixed bulk operator