Package: kex
KEXKexis a python library for unsurpervised keyword extractions, supporting the following features:Easy interface for keyword extraction with a variety of algorithmsQuick benchmarking over 15 English public datasetsCustom keyword extractor implementation supportOur paper got accepted by EMNLP 2021 main conference 🎉(camera-ready ishere):In our paper, we conducted an extensive comparison and analysis over existing keyword extraction algorithms and proposed new algorithmsLexRankandLexSpecthat
achieve very competitive baseline with very low complexity. Our proposed algorithms are based on the lexical specificity and we write a short introduction to the
lexical specificityhere.
To reproduce all the result in the paper, please followthese instructions.Get StartedInstall via pippipinstallkexExtract Keywords with KexBuilt-in algorithms inkexis below:FirstN: heuristic baseline to pick up first n phrases as keywordsTF: scoring by term frequencyTFIDF: scoring by TFIDFLexSpec:Ushio et al., 21TextRank:Mihalcea et al., 04SingleRank:Wan et al., 08TopicalPageRank:Liu et al.,10SingleTPR:Sterckx et al.,15TopicRank:Bougouin et al.,13PositionRank:Florescu et al.,18TFIDFRank: SingleRank + TFIDF based word distribution priorLexRank:Ushio et al., 21Basic usage:>>>importkex>>>model=kex.SingleRank()# any algorithm listed above>>>sample='''We propose a novel unsupervised keyphrase extraction approach that filters candidate keywords using outlier detection.It starts by training word embeddings on the target document to capture semantic regularities among the words. It thenuses the minimum covariance determinant estimator to model the distribution of non-keyphrase word vectors, under theassumption that these vectors come from the same distribution, indicative of their irrelevance to the semanticsexpressed by the dimensions of the learned vector representation. Candidate keyphrases only consist of words that aredetected as outliers of this dominant distribution. Empirical results show that our approach outperforms stateof-the-art and recent unsupervised keyphrase extraction methods.'''>>>model.get_keywords(sample,n_keywords=2)[{'stemmed':'non-keyphras word vector','pos':'ADJ NOUN NOUN','raw':['non-keyphrase word vectors'],'offset':[[47,49]],'count':1,'score':0.06874471825637762,'n_source_tokens':112},{'stemmed':'semant regular word','pos':'ADJ NOUN NOUN','raw':['semantic regularities words'],'offset':[[28,32]],'count':1,'score':0.06001468574146248,'n_source_tokens':112}]Compute a statistical priorAlgorithms such asTF,TFIDF,TFIDFRank,LexSpec,LexRank,TopicalPageRank, andSingleTPRneed to compute
a prior distribution beforehand by>>>importkex>>>model=kex.SingleTPR()>>>test_sentences=['documentA','documentB','documentC']>>>model.train(test_sentences,export_directory='./tmp')Priors are cached and can be loaded on the fly as>>>importkex>>>model=kex.SingleTPR()>>>model.load('./tmp')Supported languageCurrently algorithms are available only in English, but soon we will relax the constrain to allow other language to be supported.Benchmark on 15 Public DatasetsUsers can fetch 15 public keyword extraction datasets viakex.get_benchmark_dataset.>>>importkex>>>json_line,language=kex.get_benchmark_dataset('Inspec')>>>json_line[0]{'keywords':['kind infer','type check','overload','nonstrict pure function program languag',...],'source':'A static semantics for Haskell\nThis paper gives a static semantics for Haskell 98, a non-strict ...','id':'1053.txt'}Please take a look anexample scriptto run a benchmark on those datasets.Implement Custom Extractor with KexWe provide an API to run a basic pipeline for preprocessing, by which one can implement a custom keyword extractor.importkexclassCustomExtractor:""" Custom keyword extractor example: First N keywords extractor """def__init__(self,maximum_word_number:int=3):""" First N keywords extractor """self.phrase_constructor=kex.PhraseConstructor(maximum_word_number=maximum_word_number)defget_keywords(self,document:str,n_keywords:int=10):""" Get keywordsParameter------------------document: strn_keywords: intReturn------------------a list of dictionary consisting of 'stemmed', 'pos', 'raw', 'offset', 'count'.eg) {'stemmed': 'grid comput', 'pos': 'ADJ NOUN', 'raw': ['grid computing'], 'offset': [[11, 12]], 'count': 1}"""phrase_instance,stemmed_tokens=self.phrase_constructor.tokenize_and_stem_and_phrase(document)sorted_phrases=sorted(phrase_instance.values(),key=lambdax:x['offset'][0][0])returnsorted_phrases[:min(len(sorted_phrases),n_keywords)]Reference paperIf you use any of these resources, please cite the following paper:@inproceedings{ushio-etal-2021-kex,
    title={{B}ack to the {B}asics: {A} {Q}uantitative {A}nalysis of {S}tatistical and {G}raph-{B}ased {T}erm {W}eighting {S}chemes for {K}eyword {E}xtraction},
    author={Ushio, Asahi and Liberatore, Federico and Camacho-Collados, Jose},
        booktitle={Proceedings of the {EMNLP} 2021 Main Conference},
    year = {2021},
    publisher={Association for Computational Linguistics}
}