Package: omterms
.. role:: math(raw):format: html latex..Discovery and Representation of Open Making Related Terms=========================================================+-------------+| Bulent      || Ozel, UZH   |+-------------+| ``bulent.oz || el@gmail.co || m``         |+-------------+Support for this work is partly covered by the OpenMaker Project:http://openmaker.eu/Collaborator(s): \* Hamza Zeytinoglu--------------The first objective of this module is to provide a customizable andstandardized text preprocessing prior to further analyses where moreadvanced machine learning and or statistical techniques can be appliedand compared with each other. In that sense, it provides a pipelined setof functionalities (i) to be able to inspect, organize, prune and mergetexts around one or very few specific theme(s) or topic(s), (ii) removeunwanted terms or literals from the texts, (iii) tokenize the texts,(iv) count the terms in texts, and (v) when desired stem the tokenizedterms.The second objective of this module is to be able compare or score aforeground corpus or a specific corpus against a background corpus orreference corpus. Example use cases could be, for instance, exploringthe language of a sub-culture, a community, or a movement looking at towhat extend the specific use of the language of the group differentiatesitself from the common language.In cases when there are more than a few number of themes or topics, andwhere each topic is represented with a large set of documents thatvalidates the employment of standardized matrix decomposition basedmethodologies, then the scoring option of this module can be skippedentirely. More specifically, in use cases where the objective is beingable to classify and differentiate a number of topics or issues fromeach other and where there are sufficient data that fulfills theunderlining assumptions of NMF, LDA or LSI based approaches, then toolsfrom, for instance, Python’s`sklearn.decomposition <http://scikit-learn.org/stable/modules/decomposition.html#non-negative-matrix-factorization-nmf-or-nnmf>`__package are suggested.Nevertheless, the outputs of this module, such as its normalized termfrequencies or the specificity scores it associates to them regarding toa reference background corpus, can be used as input to other matrixdecomposition techniques.Install-------A. Via Python's standard distribution channel PyPI~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.. code:: bashpip install omtermsB. Via from its GitHub source~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.. code:: bashgit clone https://github.com/bulentozel/omterms.git.. code:: bashcd omterms.. code:: bashpip install .A quick use-----------.. code:: python>>> from omterms.interface import *>>> extract_terms("Some input X text to process less then 3 seconds.").head()Configuring the text cleaner ...A single text is provided.Extracting the terms ...Tokenizing the input text ..Done. Number of terms: 10Cleaning process: Initial size of tokens = 10Reduction due to punctuations and stopwords = 3.Reduction due to all numeral terms = 1Reduction due to short terms = 1Reduction due to rare terms = 0Reduction due to partially numeral terms = 0Reduction due to terms with not allowed symbols = 0The total term count reduction during this cleaning process = 5Percentage = 50%COMPLETED.TF     Term  wTF0   1    input  0.21   1     text  0.22   1  process  0.23   1     less  0.24   1  seconds  0.2>>>More on usage-------------`Please see thetutorial. <https://github.com/bulentozel/omterms/blob/master/tutorial.ipynb>`__----------------------------Roadmap on Keyword and Keyphrase Extraction===========================================The method outlined here aims to set-up a base line for futureimprovements.-  It uses a statistical approach combined with standardized proceduresthat are widely applied in standard NLP workflows.-  In this base line, it aims to present a work flow that can be abliedto-  different languages-  differrent problem domains-  analysis on a single theme with limited training set1. Overall work flow--------------------In short, the workflow presented on this notebook is the second stage ona workflow objective of which is being able to measure relevance of agiven external input to a specific theme, issue or topic. The steps ofthe work flow is as follows.1. Forming a specific corpus where the corpus consists of set ofdocuments around a topic. The corpus could be-  a set of blog articles around an issue let say green finance-  or a set of Wikipedia articles around the same subject-  or collection of news articles around the green finance-  or collection of tweets around the same issue.At the moment we have another module that given a set of seedWikipedia articles around an issue the crawler scrapes textual datafrom articles. For the details of the module please `see the scrapermodule. <https://github.com/bulentozel/OpenMaker/tree/master/Scraping>`__.The output of that module is a set of input texts stored in acollection in JSON format.2. Given an input set of texts on a theme a concept or a topic identifyset of terms that more likely or less likely can occur within adiscussion on the topic. This module hereby presents one of thesimple methods for this purpose.3. Given a list of weighted terms which are more likely to occur orreprsent a theme, concept or topic and input query text measure therelevance of the input text to the topic/theme/concept. `The notebookin thislink <https://github.com/bulentozel/OpenMaker/blob/master/Semantics/Score%20Text.ipynb>`__demonstrates one way doing such scoring of a given text against thecurated set of terms of this particular module.2. Suggested future work-------------------------  Comparing and combining this comparison based scoring with matrixdecompostion based topic modelling approaches such as NMF, LDA, LSI.-  Using language specicif term frequency counts of Wikipedia itself forcomparisons. In NLP terminology, the *foreground* corpus around atopic needs to be compared and contrasted to a *background* corpus.-  Improving the semantic crawler of the previous stage to be able toincrease quality of the specific corpusesMethodological Improvements~~~~~~~~~~~~~~~~~~~~~~~~~~~-  Instead of tokenizing all terms, examine possibilities of key-phraseextrcation combining with *tf-idf* and-  experiment with extracting noun phrases and words, for this useNLTK's regular expression module for POS (part of speeach)analysis.-  extract n-grams where n=1,2,33. Definitions and assumptions------------------------------Assumptions~~~~~~~~~~~-  At the comparison stage, it is assumed that a document's terms tendto be relatively frequent within the document as compared to anexternal reference corpus. However, it should be noted thisassumption is contested in the field. See the paper by Chuang et el.-  Condidering the fact that the crawler is used to aggregatesemantically related set of documents into a single document, *tf xidf* is equivalent to *tf*. As can be seen below, we use a normalizedversion of *tf*: *ntS / NS*.-  Fewer number of but relatively more relevant training (input corpus)is prefered in order to reduce term extraction problems due to lengthof documents. However, it should be noted that the crawling depth ofan identiefied wiki article from stage 1 of this document can be usedas an additional weight on relevance/reprsesntation of keywords.-  We have limited ourselves to terms instead of n-grams and phrases oruse of POS to be able to develop a base model that can work ondifferent languages.Term~~~~Given for instance a set of texts around open source software movement aterm that is identified can be a word such as *openness*, a person suchas *Stallman* a license type such as *GNU*, an acronym for anorganization such as *FSF* the Free Software Foundation, or a technologysuch as *Emacs*.Likelihood ratio~~~~~~~~~~~~~~~~It is a simple measure computed comparing frequency count of a term in aspecific corpus versus its frequency count in the reference referencecorpus. Here assumption is that the reference corpus is a large enoughsample of the language at observing the occurance of a term. Then havinga higher/lower observation frequency of a term in the specific corpus isa proxy indicator for the term choice while having a debate on thetopic.The likelihood ratio for a term :math:`P_t` is calculated as::math:`P_t = log ( (ntS/NS) / (ntR/NR) )`where-  *ntS* is the raw frequency count of the term in the entire specificcorpus-  *ntR* is the raw frequenccy count of the term in the reference corpus-  *NS* is the total number of terms in the specific corpus-  *NR* is the total number of terms in the reference corpusIt should be noted that frequency counts are calculated after havingapplied the same tokenization and post processing such as excludingstop-words, pancuations, rare terms, etc both on the reference corpusand the specific corpus.4. Some thoughts on a conceptual approach at using the extracted keywords or phrases to predict topical relevance of a new text-------------------------------------------------------------------------------------------------------------------------------Using the outcome of this technique to score arbitrary input textsagainst a single issue such as financial sustainability or against a setof issues such as the 10 basic human values requires a set ofnormalization of the raw scores and their rescaling/transformation.The factors that need to be considered are:-  **Differing document lengths:** The likelihood of repetion of a keyphrase increases as the size of the input text gets larger. In moreconcrete terms, when a scoring that simply sums up detection ofweighted keyphrases or words within a given input text would be verysensitive to the document length. For isntance, the an executivesummary of an article would very likely get quite lower score thanthe full article on any issue.*Among other methods, this can simply be resolved by computing perword scores, where the word set to be conidered is the tokenized andcleaned set of words that represent the input text.*-  **Topical relevance:** This factor would be important when thesubject matter of the inputs texts vary among each other. In otherwords, this factor would matter to a very high significance, let'ssay when one wants to compare perceptions of indivuduals on the roleof privacy in democracies and when this question is not asked them ina uniform manner, that is under the same social, cultural,environmental and physical conditions.Let’s assume that issue under investigation is again pricacy indemocracies. It is possible that the same individual as a blogger whohas a strong pro-privacy opinion (i) may not touch the issue whiletalking on data science, (ii) he would slightly touch the issue whilehe talks about her preferences in mobile devices (iii) He dives intosubject using all keywords and phrases when he talks about impact ofprivacy in a democratic life. In brief, it is necessary to offset thevariability of the topical relavance of an input text to the issueunder investigation when arbitrary text samples are used for scoring.*An offsetting scheme can be devised when opinion or perception of anactor is to be measured with respect to more than one factor thatdefine the issue under investigation. For instance, when we want tomeasure the position of a political leader on individual liberties vssocial security or when we want to profile discourse of the politicalleader as of a number of basic human values we could employ somesimple statistical methods in order to offset the topical relevanceof the discourses or the speeches of the political figure to what wewould like to measure.**A simple method could be rescaling the scores on each sub factorsuch as the scores of liberty and security that are measured from thesame speech into a range of -1 to 1. This can simply be done bytaking the mean of the two and then deducting the mean from eachscore and scaling them into a scala of -1 to 1. This way it may bepossible to use multiple speeches of the same political figure ondifferent topics to evalaute his or her postion on liberty vssecurity matter.*In statistical terms this problem corresponds to adjusting ornormalizing ratings or scores measured on different scales to anotionally common scale. Given the fact that in most cases a normaldistribution for underlying factors may not be assumed thequantile-normalization technique is suggested. The quantilenormalization sorts and ranks the variables with a non-negativeamplitudes. Then these rankings can be scaled to for instance to a 0-1interval.-  **Level of subjectivity**. This is variability in terms of relevantimportance attributed to each issue out of a given set of issues. Forinstance, it is possible that a great many individuals or politicalleaders would attach a higher importance to individual liberties thansecuirty or otherway around. But the question might be rather tounderstand to what extend one attaches more importance to an issuemore than the others. So when the objective of the scoring is notsimply to make an order of importance, then a comparative importancewith respect to overall observations needs to be tackled.*Observed variances in each query texts can be considered. That is, asimple statistical methods can be used for instance to be able tocompare two or more query texts with respect to each other. Asuggested method would be (1) estimate coefficient of variation foreach input text using per-word scores (2) the rescalequantile-normalized scores that is suggested above using theestimated coefficient of variation in each case.**When this rescaling is applied, for instance, liberty vs securitythe coeffcient of variation would act as a polarization measure.*Scoring a group of variables~~~~~~~~~~~~~~~~~~~~~~~~~~~~When one attempts to use scores generated by this package, usingspecific vs reference corpus comparisons, on a group of variable thenboth ranking of the scores as well as the relevant importance of eachscore from a number of texts from the same source should be taken intoconsideration.5. State of the art--------------------  Survey Paper: Kazi Saidul Hasan and Vincent Ng, 2014. “AutomaticKeyphrase Extraction: A Survey of the State of the Art” Proceedingsof the 52nd Annual Meeting of the Association for ComputationalLinguistics, pages 1262–1273.-  Survey Paper: Sifatullah Siddiqi and Aditi Sharan. Article: Keywordand Keyphrase Extraction Techniques: A Literature Review.International Journal of Computer Applications 109(2):18-23, January2015-  Survey Paper: Z. A. Merrouni, B. Frikh, and B. Ouhbi. Automatickeyphrase extraction: An overview of the state of the art. In 20164th IEEE Colloquium on Information Science and Technology (CiSt),pages 306–313, Oct 2016-  PageRank - Topical: Zhiyuan Liu, Wenyi Huang, Yabin Zheng and MaosongSun, 2010. “Automatic Keyphrase Extraction via Topic Decomposition”.Proceeding EMNLP '10 Proceedings of the 2010 Conference on EmpiricalMethods in Natural Language Processing Pages 366-376-  RAKE (Rapid Automatic Keyword Extraction ): Stuart Rose, Dave Engel,Nick Cramer, and Wendy Cowley. Automatic keyword extraction fromindividual documents. Text Mining, pages 1–20, 2010.-  TextRank - Graph Based : Rada Mihalcea and Paul Tarau. Textrank:Bringing order into texts. Association for Computational Linguistics,2004.-  STOPWORDS: S. Popova, L. Kovriguina, D. Mouromtsev, and I. Khodyrev.Stopwords in keyphrase extraction problem. In 14th Conference-  Corpus Similarity - Keyword frequency based: Adam Kilgarriff. Usingword frequency lists to measure corpus homogeneity and similaritybetween corpora. In Proceedings of ACLSIGDAT Workshop on very largecorpora, pages 231–245, 1997.-  Recommendation - Keyphrase Based: F. Ferrara, N. Pudota and C. Tasso.A keyphrase-based paper recommender system. In: Digital Libraries andArchives. Springer Berlin Heidelberg, 2011. p. 14-25.-  Jason Chuang, Christopher D. Manning, Jeffrey Heer, 2012. "Withoutthe Clutter of Unimportant Words": Descriptive Keyphrases for TextVisualization" ACM Trans. on Computer-Human Interaction, 19(3), 1–29.+--------------------------------------------------------------+| Learn more about the OpenMaker project: http://openmaker.eu/ |+--------------------------------------------------------------+