Package: hashedindex
Fast and simple InvertedIndex implementation using hash lists (python dictionaries).Supports Python 3.5+Free software: BSD licenseInstallingFeaturesText ParsingStemmingIntegration with Numpy and PandasReporting BugsInstallingThe easiest way to install hashindex is through pypipip install hashedindexFeatureshashedindexprovides a simple to use inverted index structure that is flexible enough to work with all kinds of use cases.Basic Usage:importhashedindexindex=hashedindex.HashedIndex()index.add_term_occurrence('hello','document1.txt')index.add_term_occurrence('world','document1.txt')index.get_documents('hello')Counter({'document1.txt':1})index.items(){'hello':Counter({'document1.txt':1}),'world':Counter({'document1.txt':1})}example='The Quick Brown Fox Jumps Over The Lazy Dog'forterminexample.split():index.add_term_occurrence(term,'document2.txt')hashedindexis not limited to strings, any hashable object can be indexed.index.add_term_occurrence('foo',10)index.add_term_occurrence(('fire','fox'),90.2)index.items(){'foo':Counter({10:1}),('fire','fox'):Counter({90.2:1})}Text ParsingThehashedindexmodule comes included with a powerful textparser module with methods to split
text into tokens.fromhashedindeximporttextparserlist(textparser.word_tokenize("hello cruel world"))[('hello',),('cruel',),('world',)]Tokens are wrapped within tuples due to the ability to specify any number of n-grams required:list(textparser.word_tokenize("Life is about making an impact, not making an income.",ngrams=2))[(u'life',u'is'),(u'is',u'about'),(u'about',u'making'),(u'making',u'an'),(u'an',u'impact'),(u'impact',u'not'),(u'not',u'making'),(u'making',u'an'),(u'an',u'income')]Take a look at the function’s docstring for information on how to usestopwords, specify amin_lengthfor tokens, and configure token output using theignore_numeric,retain_casingandretain_punctuationparameters.By default,word_tokenizeomits whitespace from the output token stream; whitespaces are rarely useful to include in a document term index.If you need to tokenize text and re-assemble an output with spacing that matches the input, you may enable thetokenize_whitespacesetting.list(textparser.word_tokenize('Conventions.  May. Differ.',tokenize_whitespace=True))[('conventions',),('  ',),('may',),(' ',),('differ',)]StemmingWhen building an inverted index, it can be useful to resolve related strings to a common root.For example, in a corpus relating to animals it might be useful to derive a singular noun for each animal; as a result, documents containing either the worddogordogscould be found under the index entrydog.Thehashedindexmodule’s text parser provides optional support for stemming by allowing the caller to specify a custom stemmer:classNaivePluralStemmer():defstem(self,x):returnx.rstrip('s')list(textparser.word_tokenize('It was raining cats and dogs',stemmer=NaivePluralStemmer()))[('it',),('wa',),('raining',),('cat',),('and',),('dog',)]Integration with Numpy and PandasThe idea behindhashedindexis to provide a really quick and easy way of generating
matrices for machine learning with the additional use of numpy, pandas and scikit-learn.
For example:fromhashedindeximporttextparserimporthashedindeximportnumpyasnpindex=hashedindex.HashedIndex()documents=['spam1.txt','ham1.txt','spam2.txt']fordocindocuments:withopen(doc,'r')asfp:fortermintextparser.word_tokenize(fp.read()):index.add_term_occurrence(term,doc)# You *probably* want to use scipy.sparse.csr_matrix for better performanceX=np.as_matrix(index.generate_feature_matrix(mode='tfidf'))y=[]fordocinindex.documents():y.append(1if'spam'indocelse0)y=np.asarray(doc)fromsklearn.svmimportSVCclassifier=SVC(kernel='linear')classifier.fit(X,y)You can also extend your feature matrix to a more verbose pandas DataFrame:importpandasaspdX=index.generate_feature_matrix(mode='tfidf')df=pd.DataFrame(X,columns=index.terms(),index=index.documents())All methods within the code have high test coverage so you can be sure everything works as expected.Reporting BugsFound a bug? Nice, a bug found is a bug fixed. Open an Issue or better yet, open a pull request.History0.10.0 (2020-10-19)addcountoptional parameter toadd_term_occurrencemethod (@jayadison)0.9.0 (2020-07-14)support non-ascii characters during tokenization (@jayadison)0.8.0 (2019-05-08)Add option to retain punctuation inword_tokenize(@jayadison)Add option to include whitespace tokens inword_tokenizeresults (@jayadison)0.7.1 (2019-04-30)Fix minor issue in history changelog0.7.0 (2019-04-30)Add support for retaining token casing inword_tokenize(Thanks @jayadison)0.6.0 (2019-12-11)Add support for running stemming operations withword_tokenize(Thanks @jayaddison)Add official support for python 3.80.5.0 (2019-07-21)Drop support for python 2.7 and 3.40.1.0 (2015-01-11)First release on PyPI.