Package: rake-spacy
rake-spacyPython implementation of the RAKE (short forRapidAutomaticKeywordExtraction) algorithm using spaCy.Installationpipinstallrake-spacySince rake-spacy depends on spacy, and to used spacy one has to load a language model, by default, rake-spacy will try to load spacy'sen_core_web_smmodel, so also grab that language model as well.python-mspacydownloaden_core_web_smWhile this is the model used by rake-spacy by default, you can easily provide rake spacy with any language model/pipeline of your chosing. (Just about anynlpobject from the spacy docs.)Getting StartedTo quickly extract some ranked keywordsfromrake_spacyimportRaker=Rake()text="Compatibility of systems of linear constraints over the set of natural numbers. Criteria of compatibility of a system of linear Diophantine equations, strict inequations, and nonstrict inequations are considered. Upper bounds for components of a minimal set of solutions and algorithms of construction of minimal generating sets of solutions for all types of systems are given. These criteria and the corresponding algorithms for constructing a minimal supporting set of solutions can be used in solving all the considered types of systems and systems of mixed types."ranklist=r.apply(text)print(ranklist)# [(8.0, minimal generating sets), (8.0, minimal supporting set), (7.0, minimal set), (5.0, considered types), (5.0, system), (5.0, systems), ...]Differnent Language ModelsTo specify a language model other than spacy'sen_core_web_smmodel, you simply instatiate the language model of your choosing, and pass it toRakein instantiation:fromrake_spacyimportRakeimportspacynlp=spacy.load('en_core_web_lg')# assuming this model is installedr=Rake(nlp=nlp)text='This is a sentence.'# Rake is now using the provided nlp object to prase the documentranklist=r.apply(text)Code Structure and CustomizationAt it's core, the RAKE algorithm is conceptually simple:Extract candidate spans(sequences of contiguous words/tokens) from a piece of text.Create the co-occurance graphbased on the spans extracted from the previous step (represented as an adjacency matrix, where two words/tokens are "adjacent", i.e. share an edge in the graph, if they are found in the same span).Compute a score for each word/tokenbased from the co-occurance graph.Compute a score for each spanby aggregating word scores (determined in the previous step)Order the spansbased on the aggregate span-scores.There is a fairly close mapping between the highlevel steps outlined above, the package structure of rake-spacy, and the available keyword arguments of theRakeobject. To understand the structure of the code we'll start with the first step.PhrasersTo apply RAKE to a piece of text the first thing we need is a way to extract candidate spans from the text. Throughout the remainder of the documentation these spans will be referred to as "phrases".By default, rake-spacy extracts "contiguous non-stop-words" as the candidate phrases. What are "contiguous non-stop-words"? These are just chunks of text (words/tokens that are side-by-side) which contain no stop words. (More on how stop words/tokens are determined below.)This process of extracting candidate phrases for RAKE to consider (i.e. from which to build the co-occurance graph) can be customized by providing a user-specified callable as thephraser_classparameter when instantiatingRake:defmy_phraser_func(doc:spacy.language.Language)->List[spacy.tokens.Span]:...# slice and dice the document as desiredreturnmy_list_of_spansr=Rake(phraser_class=my_phraser_func)Essentially, the object passed as the value to thephraser_classargument must be a callable which accepts aspacy.tokens.Docobject and returns a list ofspacy.token.Spanobjects. Those span objects will be the phrases used to construct the co-occurance graph.Phraser classes included in this package can be found in therake_spacy.phrasersmodule.Aside #1Why call the paramter phraser_class? This is because the "batteries-included" phraser callables (found in therake_spacy.phrasersmodule) inherit fromrake_spacy.phrasers.BasePhraser. The choice to use classes is to allow for "parameterizable callables" which still only take one argument when called. Specifically, whenRakecalls thephraser_classcallable the only parameter that is provided is the spacydoc. By using a class to define this callable additional parameters can be specified in the class's__init__method, and used in the__call__method.Of course, there are other ways to achieve this same "parameterizability", one could define the phraser parameters in the global scope and then reference those variables in the function body.GLOBAL_NAME='World'deff(greeting):print(f'{greeting},{GLOBAL_NAME}!')Or if global variables aren't your thing you could usefunctools.partialimportfunctoolsdeffull_g(greeting,name):print(f'{greeting},{name}')g=functools.partial(full_g,name='World')As with most things in python there's typically more than one way to do something. "Callable classes" seemed like a straight forward choice, and are found throughout the code.(Word) ScorersSkipping over the construction of the co-occurance graph for a moment. The next thing to do is to score each of the words/tokens found in the collection of phrases using the information contained in the co-occurance graph. By default, the score rake-spacy attributes to each word/token is simply the number of times it occured in the text (these are the diagonal elements in the co-occurance graph).To override the default behaviour we simply define a callable that takes the co-occurance graph and a token (vertex in the graph) and returns a numeric score, and then provide that callable as the value to theword_scorer_classargument:defmy_scorer_func(cograph:DefaultDict[str,DefaultDict[str,int]],token:spacy.tokens.Token)->float:...# compute the score for this token using the co-ocrrance graphreturntoken_scorer=Rake(word_scorer_class=my_scorer_func)Word-scorer classes included in this package can be found in therake_spacy.scorersmodule.Aside #2It's worth mentioning here that the co-occurance graph is stored as (essentially) a defaultdict within a defaultdict. Specifically, "indexing" once into the co-occurance graph with a token,cograph[token], returns a dict-like object which maps co-occuring words/tokens to co-occurance counts. That is,cograph[token][cotoken]is an integer denoting the number of timetokenandcotokenappeared in the same phrase.cograph[token]looks weird, you might be thinking.tokenis aspacy.tokens.Tokenobject and it's being used as the key to this dict-looking-thing? Of course,spacy.tokens.Tokenobjects have a__hash__method, and so theycanbe used as dictionary keys. But that's not (necessarily) what is going on under the hood (although itcouldbe - if you really wanted it to be that way). Using tokens as "keys" is facilitated (and, in fact, is customizable as well) by another set of classes we have yet to talk about, but this small detail is precisely why you can think of the co-occurance graph as beingessentiallylike a defaultdict within a defaultdict. In a nut shell, yes, you should index into the co-occurance graph withspacy.tokens.Tokenobjects, directly. More on this below.AggregatorsOK, so we've split our text up into phrases, computed the co-occurance graph and scored the words/tokens, now we have to score the phrases themselves.By default, rake-spacy will just sum up all the token scores in each phrase. But this default behaviour can be changed by providing a callable as the value to theaggregator_classkeyword argument.defmy_aggregator_func(word_scores:List[float])->float:...# compute the phrase-score from the word-scores of its constiuent wordsreturnphrase_scorer=Rake(aggregator_class=my_aggregator_func)Aggregator classes included in this package can be found in therake_spacy.aggregatorsmodule.This covers probably the most important parts of how one might want to customize how the RAKE algorithm is administered to a piece of text, but we did skim over a few things. We'll cover those next.Stop-Words/TokensSpacy is a fantastic library and with basically no code at all we can breakdown a piece of text into fine-grained parts seemingly "for free". However, when applying the RAKE algorithm we'd probably like to ignore some of the more common or less-meaningful words/tokens. Stop-words were alluded to above, how can we customize which words/tokens are considered "stop-words" and should effectively be ignored?By default, rake-spacy assumes that any token for which the following expression evaluates toTrueis a stop-token:(token.is_stoportoken.is_spaceortoken.is_punct)andnottoken.like_numTo customize this behaviour we can implement (you guessed it) acallablewhich accepts a token and returns a boolean. (True if this token should be ignored, False if it should be kept.)defmy_stop_word_detector_func(token:spacy.tokens.Token)->bool:...# determine if this toke is a stop-token (True) or not (False)returnis_stop_tokenr=Rake(stop_token_class=my_stop_word_detector_func)Internally, rake-spacy will use this callable on each token (actually, pair of tokens) before updating the co-occurance graph. If the callable indicates that one of the two tokens is a stop-word/token and should be ignored, that pair is skipped and rake-spacy moves onto the next pair.One thing to note is that a user-specified stop-word callable will likely not have a role in how phrases are extracted (because "phrasing" is also user-specifiable). As such, if you provideRakewith astop_token_classoverride, for consistency, you may want to use that same stop-word callable in the phraser callable. Perhaps something like:classMyStopWordIndicator(rake_spacy.stop_words.BaseStopTokenIndicator):def__call__(self,token):...# your implementationclassMyPhraser(rake_spacy.phrasers.BasePhraser):def__init__(self,stop_word_indicator):self.stop_word_indicator=stop_word_indicatordef__call__(self,doc):...# your implementation, using self.stop_word_indicatorstop_token_class=MyStopWordIndicator()phraser_class=MyPhraser(stop_token_class)r=Rake(stop_token_class=stop_token_class,phraser_class=phraser_class)This is actually what rake-spacy does in theRake.__init__()method.Stop-word/token classes included in this package can be found in therake_spacy.stop_wordsmodule.The Co-Occurance GraphSo, why is the co-occurance graphessentiallylike a defaultdict within a defaultdict? Actually, internally the co-occurance graph uses a subclass ofcollections.defaultdict. This subclass is a thin wrapper around the stdlib defaultdict, but it slides in a (user-specifiable transformation) between the objects that are fed to__getitem__and__setitem__and what is given to the underlying defaultdict. Think of it like some light-weight middleware.You provide acallablewhich accepts an object and converts that object to a string. How this conversion is done is up to you. The Co-Occurance graph simply applies this callable to whatever is given to get/setitem and feeds the output to the underlying defaultdict.By default this callable is just the builtinstrfunction. To customize this behaviour, provide a callable as the value to thetoken_mapper_classargument:defmy_token_mapper_func(token:spacy.tokens.Token)->bool:...# Transform the token to a stringreturntoken_as_stringr=Rake(token_mapper_class=my_token_mapper_func)Why might you want to do this? Perhaps you want the tokens "Hello" and "hello" to be mapped to the same vertex in the co-occurance graph. This is not the default behaviour, but can be achieved by, say, the function:deff(token):returnstr(token).lower()r=Rake(token_mapper_class=f)Or perhaps you wantisandareto be mapped to the same vertex (via the lemmabe). This is not the default behaviour, but can be achieved by, say, the function:defg(token):returntokenifisinstance(str,token)elsetoken.lemma_.lower()r=Rake(token_mapper_class=g)Important: The callable passed astoken_mapper_classmust be idempotent. Specifically, it should satisfy the property:token_mapper(t)==token_mapper(token_mapper(t))To circle back, this how tokens (that are used to "index into" the co-occurance graph) and the graph itself play together nicely.Token mapper classes included in this package can be found in therake_spacy.mappersmodule.