Package: pyate
PyATE: Python Automated Term ExtractionPython implementation of term extraction algorithms such as C-Value, Basic,
Combo Basic, Weirdness and Term Extractor using spaCy POS tagging.NEW: Documentation can be found athttps://kevinlu1248.github.io/pyate/. The
documentation so far is still missing two algorithms and details about theTermExtractionclass but I will have it done soon.NEW: Try the algorithms out athttps://pyate-demo.herokuapp.com/, a web app for
demonstrating PyATE!NEW: spaCy V3 is supported! For spaCy V2, usepyate==0.4.3and view thespaCy V2 README.md fileIf you have a suggestion for another ATE algorithm you would like implemented in
this package feel free to file it as an issue with the paper the algorithm is
based on.For ATE packages implemented in Scala and Java, seeATR4SandJATE, respectively.:tada: InstallationUsing pip:pipinstallpyatespacydownloaden_core_web_sm:rocket: QuickstartTo get started, simply call one of the implemented algorithms. According to
Astrakhantsev 2016,combo_basicis the most precise of the five algorithms,
thoughbasicandcvaluesis not too far behind (see Precision). The same
study shows that PU-ATR and KeyConceptRel have higher precision thancombo_basicbut are not implemented and PU-ATR take significantly more time
since it uses machine learning.frompyateimportcombo_basic# source: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1994795/string="""Central to the development of cancer are genetic changes that endow these “cancer cells” with many of thehallmarks of cancer, such as self-sufficient growth and resistance to anti-growth and pro-death signals. However, while thegenetic changes that occur within cancer cells themselves, such as activated oncogenes or dysfunctional tumor suppressors,are responsible for many aspects of cancer development, they are not sufficient. Tumor promotion and progression aredependent on ancillary processes provided by cells of the tumor environment but that are not necessarily cancerousthemselves. Inflammation has long been associated with the development of cancer. This review will discuss the reflexiverelationship between cancer and inflammation with particular focus on how considering the role of inflammation in physiologicprocesses such as the maintenance of tissue homeostasis and repair may provide a logical framework for understanding the Uconnection between the inflammatory response and cancer."""print(combo_basic(string).sort_values(ascending=False))""" (Output)dysfunctional tumor                1.443147tumor suppressors                  1.443147genetic changes                    1.386294cancer cells                       1.386294dysfunctional tumor suppressors    1.298612logical framework                  0.693147sufficient growth                  0.693147death signals                      0.693147many aspects                       0.693147inflammatory response              0.693147tumor promotion                    0.693147ancillary processes                0.693147tumor environment                  0.693147reflexive relationship             0.693147particular focus                   0.693147physiologic processes              0.693147tissue homeostasis                 0.693147cancer development                 0.693147dtype: float64"""If you would like to add this to a spacy pipeline, simply use add Spacy'sadd_pipemethod.importspacyfrompyate.term_extraction_pipelineimportTermExtractionPipelinenlp=spacy.load("en_core_web_sm")nlp.add_pipe("combo_basic")doc=nlp(string)print(doc._.combo_basic.sort_values(ascending=False).head(5))""" (Output)dysfunctional tumor                1.443147tumor suppressors                  1.443147genetic changes                    1.386294cancer cells                       1.386294dysfunctional tumor suppressors    1.298612dtype: float64"""Also,TermExtractionPipeline.__init__is defined as follows__init__(
  self,
  func: Callable[..., pd.Series] = combo_basic,
  *args,
  **kwargs
)wherefuncis essentially your term extracting algorithm that takes in a
corpus (either a string or iterator of strings) and outputs a Pandas Series of
term-value pairs of terms and their respective termhoods.funcis by defaultcombo_basic.argsandkwargsare for you to overide default values for the
function, which you can find by runninghelp(might document later on).Summary of functionsEach ofcvalues, basic, combo_basic, weirdnessandterm_extractortake in a
string or an iterator of strings and outputs a Pandas Series of term-value
pairs, where higher values indicate higher chance of being a domain specific
term. Furthermore,weirdnessandterm_extractortake ageneral_corpuskey
word argument which must be an iterator of strings which defaults to the General
Corpus described below.All functions only take the string of which you would like to extract terms from
as the mandatory input (thetechnical_corpus), as well as other tweakable
settings, includinggeneral_corpus(contrasting corpus forweirdnessandterm_extractor),general_corpus_size,verbose(whether to print a progress
bar),weights,smoothing,have_single_word(whether to have a single word
count as a phrase) andthreshold. If you have not read the papers and are
unfamiliar with the algorithms, I recommend just using the default settings.
Again, usehelpto find the details regarding each algorithm since they are
all different.General CorpusUnderpath/to/site-packages/pyate/default_general_domain.en.zip, there is a
general CSV file of a general corpus, specifically, 3000 random sentences from
Wikipedia. The source of it can be found athttps://www.kaggle.com/mikeortman/wikipedia-sentences. Access it using it using
the following after installingpyate.importpandasaspdfromdistutils.sysconfigimportget_python_libdf=pd.read_csv(get_python_lib()+"/pyate/default_general_domain.en.zip")["SECTION_TEXT"]print(df.head())""" (Output)0    '''Anarchism''' is a political philosophy that...1    The term ''anarchism'' is a compound word comp...2    ===Origins===\nWoodcut from a Diggers document...3    Portrait of philosopher Pierre-Joseph Proudhon...4    consistent with anarchist values is a controve...Name: SECTION_TEXT, dtype: object"""Other LanguagesFor switching languages, simply runTerm_Extraction.set_language({language}, {model_name}), wheremodel_namedefaults tolanguage. For example,Term_Extraction.set_language("it", "it_core_news_sm"})for Italian. By
default, the language is English. So far, the list of supported languages is:English(en)Dutch(nl)French(fr)German(de)Italian(it)Portuguese(pt)Russian(ru)Spanish(es)To add more languages, file an issue with a corpus of at least 3000 paragraphs
of a general domain in the desired language (preferably wikipedia) nameddefault_general_domain.{lang}.zipreplacing lang with the ISO-639-1 code of
the language, or the ISO-639-2 if the language does not have a ISO-639-1 code
(can be found athttps://www.loc.gov/standards/iso639-2/php/code_list.php). The
file format should be of the following form to be parsable by Pandas.,SECTION_TEXT
0,"{paragraph_0}"
1,"{paragraph_1}"
...Alternatively, place the file insrc/pyateand file a pull request.ModelsThough this model was originally intended for symbolic AI algorithms
(non-machine learning), I realized a spaCy model on term extraction can reach
significantly higher performance, and thus decided to include the model here.For a comparison with the symbolic AI algorithms, seePrecision. Note that only
the F-Score, accuracy and precision was taken here yet for the model, but for
the algorithms the AvP was taken so directly comparing the metrics would not
really make sense.URLF-Score (%)Precision (%)Recall (%)https://github.com/kevinlu1248/pyate/releases/download/v0.4.2/en_acl_terms_sm-2.0.4.tar.gz94.7195.4194.03The model was trained and evaluated on theACL dataset, which is
a computer science oriented dataset where the terms are manually picked. This
has not yet been tested on other fields yet, however.This model does not come with PyATE. To install, runpipinstallhttps://github.com/kevinlu1248/pyate/releases/download/v0.4.2/en_acl_terms_sm-2.0.4.tar.gzTo extract terms,importspacynlp=spacy.load("en_acl_terms_sm")doc=nlp("Hello world, I am a term extraction algorithm.")print(doc.ents)"""(term extraction, algorithm)""":dart: PrecisionHere is the average precision of some of the implemented algorithms using the
Average Precision (AvP) metric on seven distinct databases, as tested in
Astrakhantsev 2016.:stars: MotivationThis project was planned to be a tool to be connected to a Google Chrome
Extension that highlights and defines key terms that the reader probably does
not know of. Furthermore, term extraction is an area where there is not a lot of
focused research on in comparison to other areas of NLP and especially recently
is not viewed to be very practical due to the more general tool of NER tagging.
However, modern NER tagging usually incorporates some combination of memorized
words and deep learning which are spatially and computationally heavy.
Furthermore, to generalize an algorithm to recognize terms to the ever growing
areas of medical and AI research, a list of memorized words will not do.Of the five implemented algorithms, none are expensive, in fact, the bottleneck
of the space allocation and computation expense is from the spaCy model and
spaCy POS tagging. This is because they mostly rely simply on POS patterns, word
frequencies, and the existence of embedded term candidates. For example, the
term candidate "breast cancer" implies that "malignant breast cancer" is
probably not a term and simply a form of "breast cancer" that is "malignant"
(implemented in C-Value).:pushpin: TodoAdd other languages and data encapsulation for set languageAdd automated tests and CI/CDAdd a brief CLIMake NER version of this using the datasets from the sourcesAdd PU-ATR algorithm since its precision is a lot higher, though more
computationally expensivePage Rank algorithmAdd sourcesAdd voting algorithm and capabilitiesOptimize perhaps using Cython, however, the bottleneck is POS tagging by Spacy
and word counting with Pandas and Numpy, which are already at C-level so this
will not help muchClearer documentationAllow GPU acceleration with Cupy:bookmark_tabs: SourcesI cannot seem to find the original Basic and Combo Basic papers but I found
papers that referenced them. "ATR4S: Toolkit with State-of-the-art Automatic
Terms Recognition Methods in Scala" more or less summarizes everything and
incorporates several algorithms not in this package.Automatic Recognition of Multi-word Terms: The C-value/ NC-value MethodDomain-independent term extraction through domain modellingATR4S: Toolkit with State-of-the-art Automatic Terms Recognition Methods in ScalaTermExtractor: a Web Application to Learn the Shared Terminology of Emergent Web CommunitiesLearning Domain Ontologies from Document Warehouses and Dedicated Web SitesA Comparative Evaluation of Term Recognition AlgorithmsSemRe-Rank: Improving Automatic Term Extraction By Incorporating Semantic Relatedness With Personalised PageRankTerm extraction: A Review Draft Version 091221:closed_book: AcademiaCitingIf you publish work that uses PyATE, please let me know atkevinlu1248@gmail.comand cite as:Lu, Kevin. (2021, June 28). kevinlu1248/pyate: Python Automated Term Extraction (Version v0.5.3). Zenodo. http://doi.org/10.5281/zenodo.5039289or equivalently with Bibtext:@software{pyate,
	title        = {kevinlu1248/pyate: Python Automated Term Extraction},
	author       = {Lu, Kevin},
	year         = 2021,
	month        = {Jun},
	publisher    = {Zenodo},
	doi          = {10.5281/zenodo.5039289}
}Influences on AcademiaThis package was used in the paper
(Unsupervised Technical Domain Terms Extraction using Term Extractor (Dowlagar and Mamidi, 2021).:coffee: Buy Me a CoffeeIf my work helped you, please consider buying me a coffee athttps://www.buymeacoffee.com/kevinlu1248.