Package: lexicalrichness
LexicalRichnessis a small Python module to compute textual lexical richness (aka lexical diversity) measures.Lexical richness refers to the range and variety of vocabulary deployed in a text by a speaker/writer(McCarthy and Jarvis 2007). Lexical richness is used interchangeably with lexical diversity, lexical variation, lexical density, and vocabulary richness and is measured by a wide variety of indices. Uses include (but not limited to) measuring writing quality, vocabulary knowledge(Šišková 2012), speaker competence, and socioeconomic status(McCarthy and Jarvis 2007).
See thenotebookfor examples.Table of Contents1. Installation2. Quickstart3. Use LexicalRichness in your own pipeline4. Using with Pandas5. Attributes6. Methods7. Formulation & Algorithmic Details8. Example use cases9. Contributing10. Citing1. InstallationInstall using PIPpipinstalllexicalrichnessIf you encounter,ModuleNotFoundError:Nomodulenamed'textblob'install textblob:pipinstalltextblobNote: This error should only exist forversions <= v0.1.3. Fixed inv0.1.4byDavid LesieurandChristophe Bedetti.Install from Conda-ForgeLexicalRichnessis now also available on conda-forge. If you have are using theAnacondaorMinicondadistribution, you can create a conda environment and install the package from conda.condacreate-nlexcondaactivatelexcondainstall-cconda-forgelexicalrichnessNote: If you get the errorCommandNotFoundError: Your shell has not been properly configured to use 'conda activate'withconda activate lexinBasheither tryconda activate bashin theAnaconda Promptand then retryconda activate lexinBashor just trysource activate lexinBashInstall manually using Git and GitHubgitclonehttps://github.com/LSYS/LexicalRichness.gitcdLexicalRichnesspipinstall.Run from the cloudTry the package on the cloud (without setting anything up on your local machine) by clicking the icon here:2. Quickstart>>>fromlexicalrichnessimportLexicalRichness# text example>>>text="""Measure of textual lexical diversity, computed as the mean length of sequential words in
                a text that maintains a minimum threshold TTR score.

                Iterates over words until TTR scores falls below a threshold, then increase factor
                counter by 1 and start over. McCarthy and Jarvis (2010, pg. 385) recommends a factor
                threshold in the range of [0.660, 0.750].
                (McCarthy 2005, McCarthy and Jarvis 2010)"""# instantiate new text object (use the tokenizer=blobber argument to use the textblob tokenizer)>>>lex=LexicalRichness(text)# Return word count.>>>lex.words57# Return (unique) word count.>>>lex.terms39# Return type-token ratio (TTR) of text.>>>lex.ttr0.6842105263157895# Return root type-token ratio (RTTR) of text.>>>lex.rttr5.165676192553671# Return corrected type-token ratio (CTTR) of text.>>>lex.cttr3.6526846651686067# Return mean segmental type-token ratio (MSTTR).>>>lex.msttr(segment_window=25)0.88# Return moving average type-token ratio (MATTR).>>>lex.mattr(window_size=25)0.8351515151515151# Return Measure of Textual Lexical Diversity (MTLD).>>>lex.mtld(threshold=0.72)46.79226361031519# Return hypergeometric distribution diversity (HD-D) measure.>>>lex.hdd(draws=42)0.7468703323966486# Return voc-D measure.>>>lex.vocd(ntokens=50,within_sample=100,iterations=3)46.27679899103406# Return Herdan's lexical diversity measure.>>>lex.Herdan0.9061378160786574# Return Summer's lexical diversity measure.>>>lex.Summer0.9294460323356605# Return Dugast's lexical diversity measure.>>>lex.Dugast43.074336212149774# Return Maas's lexical diversity measure.>>>lex.Maas0.023215679867353005# Return Yule's K.>>>lex.yulek153.8935056940597# Return Yule's I.>>>lex.yulei22.36764705882353# Return Herdan's Vm.>>>lex.herdanvm0.08539428890448784# Return Simpson's D.>>>lex.simpsond0.0156641604010025053. Use LexicalRichness in your own pipelineLexicalRichnesscomes packaged with minimal preprocessing + tokenization for a quick start.But for intermediate users, you likely have your preferrednlp_pipeline:# Your preferred preprocessing + tokenization pipelinedefnlp_pipeline(text):...returnlist_of_tokensUseLexicalRichnesswith your ownnlp_pipeline:# Initiate new LexicalRichness object with your preprocessing pipeline as inputlex=LexicalRichness(text,preprocessor=None,tokenizer=nlp_pipeline)# Compute lexical richnessmtld=lex.mtld()Or useLexicalRichnessat the end of your pipeline and input thelist_of_tokenswithpreprocessor=Noneandtokenizer=None:# Preprocess the textlist_of_tokens=nlp_pipeline(text)# Initiate new LexicalRichness object with your list of tokens as inputlex=LexicalRichness(list_of_tokens,preprocessor=None,tokenizer=None)# Compute lexical richnessmtld=lex.mtld()4. Using with PandasHere’s a minimal example usinglexicalrichnesswith aPandasdataframewith a column containing text:defmtld(text):lex=LexicalRichness(text)returnlex.mtld()df['mtld']=df['text'].apply(mtld)5. Attributeswordlistlist of wordswordsnumber of words (w)termsnumber of unique terms (t)preprocessorpreprocessor usedtokenizertokenizer usedttrtype-token ratio computed as t / w (Chotlos 1944, Templin 1957)rttrroot TTR computed as t / sqrt(w) (Guiraud 1954, 1960)cttrcorrected TTR computed as t / sqrt(2w) (Carrol 1964)Herdanlog(t) / log(w) (Herdan 1960, 1964)Summerlog(log(t)) / log(log(w)) (Summer 1966)Dugast(log(w) ** 2) / (log(w) - log(t) (Dugast 1978)Maas(log(w) - log(t)) / (log(w) ** 2) (Maas 1972)yulekYule’s K (Yule 1944, Tweedie and Baayen 1998)yuleiYule’s I (Yule 1944, Tweedie and Baayen 1998)herdanvmHerdan’s Vm (Herdan 1955, Tweedie and Baayen 1998)simpsondSimpson’s D (Simpson 1949, Tweedie and Baayen 1998)6. MethodsmsttrMean segmental TTR (Johnson 1944)mattrMoving average TTR (Covington 2007, Covington and McFall 2010)mtldMeasure of Lexical Diversity (McCarthy 2005, McCarthy and Jarvis 2010)hddHD-D (McCarthy and Jarvis 2007)vocdvoc-D (Mckee, Malvern, and Richards 2010)vocd_figUtility to plot empirical voc-D curvePlot the empirical voc-D curvelex.vocd_fig(ntokens=50,# Maximum number for the token/word size in the random samplingswithin_sample=100,# Number of samplesseed=42,# Seed for reproducibility)Assessing method docstrings>>>importinspect# docstring for hdd (HD-D)>>>print(inspect.getdoc(LexicalRichness.hdd))Hypergeometricdistributiondiversity(HD-D)score.Foreachterm(t)inthetext,computetheprobabiltiy(p)ofgettingatleastoneappearanceoftwitharandomdrawofsizen<N(textsize).ThecontributionofttothefinalHD-Dscoreisp*(1/n).ThefinalHD-Dscorethussumsoverp*(1/n)withpcomputedforeachtermt.DescribedinMcCarthyandJavis2007,p.g.465-466.(McCarthyandJarvis2007)Parameters__________draws:intNumberofrandomdrawsinthehypergeometricdistribution(default=42).Returns_______floatAlternatively, just do>>>print(lex.hdd.__doc__)Hypergeometricdistributiondiversity(HD-D)score.Foreachterm(t)inthetext,computetheprobabiltiy(p)ofgettingatleastoneappearanceoftwitharandomdrawofsizen<N(textsize).ThecontributionofttothefinalHD-Dscoreisp*(1/n).ThefinalHD-Dscorethussumsoverp*(1/n)withpcomputedforeachtermt.DescribedinMcCarthyandJavis2007,p.g.465-466.(McCarthyandJarvis2007)Parameters----------draws:intNumberofrandomdrawsinthehypergeometricdistribution(default=42).Returns-------float7. Formulation & Algorithmic DetailsFor details under the hood, please seethis sectionin the docs (orsee here).8. Example use cases[1]SENTiVENTused the metrics that LexicalRichness provides to estimate the classification difficulty of annotated categories in their corpus (Jacobs & Hoste 2020). The metrics show which categories will be more difficult for modeling approaches that rely on linguistic inputs because greater lexical diversity means greater data scarcity and more need for generalization. (h/t Gilles Jacobs)Jacobs, Gilles, and Véronique Hoste. “SENTiVENT: enabling supervised information extraction of company-specific events in economic and financial news.” Language Resources and Evaluation (2021): 1-33.[2]Measuring political media using text data.This chapter of my thesis investigates whether political media bias manifests by coverage accuracy. As covaraites, I use characteristics of the text data (political speech and news article transcripts). One of the ways speeches can be characterized is via lexical richness.[3]Unreadable News: How Readable is American News?This study characterizes modern news by readability and lexical richness. Focusing on the NYT, they find increasing readability and lexical richness, suggesting that NYT feels competition from alternative sources to be accessible while maintaining its key demographic of college-educated Americans.[4]German is more complicated than EnglishThis study analyses a small sample of English books and compares them to their German translation. Within the sample, it can be observed that the German translations tend to be shorter in length, but contain more unique terms than their English counterparts. LexicalRichness was used to generate the statistics modeled within the study.9. ContributingAuthorLucas ShenContributorsContributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.
See here forhow to contributeto this project.
See here forContributor Code of
Conduct.If you’d like to contribute via a Pull Request (PR), feel free to open an issue on theIssue Trackerto discuss the potential contribution via a PR.10. CitingIf you have used this codebase and wish to cite it, here is the citation metadata.Codebase:@misc{lex,author={Shen, Lucas},doi={10.5281/zenodo.6607007},license={MIT license},title={{LexicalRichness: A small module to compute textual lexical richness}},url={https://github.com/LSYS/lexicalrichness},year={2022}}Documentation on formulations and algorithms:@misc{accuracybias,title={Measuring Political Media Slant Using Text Data},author={Shen, Lucas},url={https://www.lucasshen.com/research/media.pdf},year={2021}}The package is released under theMIT
License.