Package: pylats
Python Linguistic Analysis Tools (pylats)Pylats is designed to perform text pre-processing for further linguistic analyses (e.g., measuring lexical diversity and/or lexical sophistication).  Currently, advanced features such as lemmatization and POS tagging are available for English, with other languages to follow (Spanish models will be added in the near future, we have plans to release other models as well). However, pylats CAN currently be used with other languages using basic features (see examples below).Pylats currently usesspacyas a starting point for many of its advanced features if working with English. Pylats was tested using spacy version 3.2 and be default uses the "en_core_web_sm" model. To install spacy and a language model, see thespacy installation instructions.InstallationTo install pylats, you can usepip:pipinstallpylatsGetting StartedImportfrompylatsimportlatsUsing pylatspylatsis designed to be the first step in conducting linguistic analyses using related analysis tools (such aslexical-diversity).pylatsuses theNormalizeclass to take a raw text string and format it:teststr="I love pepperoni pizza."normed=lats.Normalize(teststr)#processed text stringprint(normed.toks)#output:
['i', 'love', 'pepperoni', 'pizza']Paragraphs and sentencesThe.toksmethod will provide a flat list of the tokens in a text. However, it can often be useful to conduct analyses at the sentence and/or paragraph level. The.sentsand.parasmethods provide a representation of text with nested lists.teststr="""I love pepperoni pizza. Sometimes I like to add feta and banana peppers.This is a second paragraph. In the original string there is a newline character before this paragraph."""normedp=lats.Normalize(para_sample)tokensprint(normedp.toks)['i', 'love', 'pepperoni', 'pizza', 'sometimes', 'i', 'like', 'to', 'add', 'feta', 'and', 'banana', 'peppers', 'this', 'is', 'a', 'second', 'paragraph', 'in', 'the', 'original', 'string', 'there', 'is', 'a', 'newline', 'character', 'before', 'this', 'paragraph']sentencesforxinnormedp.sents:print(x)#print tokens in each sentence['i', 'love', 'pepperoni', 'pizza']
['sometimes', 'i', 'like', 'to', 'add', 'feta', 'and', 'banana', 'peppers']
['this', 'is', 'a', 'second', 'paragraph']
['in', 'the', 'original', 'string', 'there', 'is', 'a', 'newline', 'character', 'before', 'this', 'paragraph']paragraphsfor x in normed.paras:
	print(x) #print sentences each paragraph[['i', 'love', 'pepperoni', 'pizza'], ['sometimes', 'i', 'like', 'to', 'add', 'feta', 'and', 'banana', 'peppers']]
[['this', 'is', 'a', 'second', 'paragraph'], ['in', 'the', 'original', 'string', 'there', 'is', 'a', 'newline', 'character', 'before', 'this', 'paragraph']]Changing parametersBy default,Normalizesimply removes punctuation and converts words in the text to lower case. However, a wide range of customizations can be made by adjusting theparametersclass.For example, it may be useful to exclude particular words for some analyses. In studies of lexical diversity, for instance, we probably don't want to include misspelled words (misspelled words would positively contribute to diversity scores, but probably shouldn't). Pylats includes a default list of "real" words drawn from a large corpus of English language which can be used to filter out misspelled ones. Words can also be added to a list of items to remove OR can be added to a list that overrides other lists.Below, we create a copy ofparametersand then make some changes:new_params=lats.parameters()#create a copy of the parameters classnew_params.attested=True#set the attested attribute to TrueOutput with default settings:#with default settingsmsp_default=lats.Normalize("This is a smaple sentence")print(msp_default.toks)['this', 'is', 'a', 'smaple', 'sentence']Output with new settings:msp_new=lats.Normalize("This is a smaple sentence",params=new_params)print(msp_new.toks)['this', 'is', 'a', 'sentence']Default parametersclassparameters:#default English parameterslang="en"model="en_core_web_sm"try:nlp=nlp_en_smexceptNameError:nlp=Nonepunctuation=['``',"''","'",'.',',','?','!',')','(','%','/','-','_','-LRB-','-RRB-','SYM',':',';','"']punctse=[".","?","!"]abbrvs=["mrs.","ms.","mr.","dr.","phd."]splitter="\n"#for splitting paragraphsrwl=en_rwlsp=Truesspl="spacy"pos=None#other options are "pos" for Penn tags and "upos" for universal tagsremovel=['becuase']#typos and other words not caught by the real words listlemma=Falselower=True#treat all words as lower caseattested=False#filter output using real words list?spaces=[" ","  ","   ","    "]#need to add more hereoverride=[]#items the system ignores that should be overriddenposignore=[]numbers=["NUM"]#pos_ tag for numbersnonumbers=Trueconnect="__"#for connecting ngramscontentPOS=[]#can be added, blank for nowcontentLemIgnore=[]#can be added, blank for nowAdding part of speech informationIf spacy is installed (and activated), part of speech tags can be added to each token, which can be useful in disambiguating homographic tokens (e.g.,runas a verb in the sentenceI like torun.versus run as a noun in the sentenceI went for arun.). This is helpful in a number of applications, including calculating indices lexical diversity.pos_params=lats.parameters()pos_params.pos="upos"#for large-grained universal parts of speechrun_sample=lats.Normalize("I like to run. I went for a run.",params=pos_params)forxinrun_sample.sents:print(x)['i_PRON', 'like_VERB', 'to_PART', 'run_VERB']
['i_PRON', 'went_VERB', 'for_ADP', 'a_DET', 'run_NOUN']Changing the spacy language modelTo change the spacy language model that is used by pylats, first make sure that the desired model has been downloaded fromspacy. The preferred method to load models is to create a parameter class that includes the appropriate model (for easier replicability), then use the following command:myparameters.model="en_core_web_trf"myparameters.nlp=lats.load_model(lats.myparameters.model)Using pylats with languages other than EnglishPylats currently has advanced features available for English and Spanish texts and basic features for other languages. As the tool expands, advanced feature support will be added for other languages (let us know what languages you would like to see supported!).To process texts with basic features, simply change parameters.sp toTrue. The processor will treat text between whitespace as a token. Accordingly, some pre-processing may be necessary.Example 1 (Spanish), advanced features:
There are currently two pre-made parameter classes for Spanish. One uses a faster (but slightly less accurate) tagging and parsing model (parameters_es, which uses the "es_core_news_sm" spacy model). The second uses a slower (but more accurate) tagging and parsing model (parameters_es_trf, which uses the "es_dep_news_trf" model). To process Spanish texts, first be sure to download the appropriate model fromspacy.If either of these models are installed prior to importing pylats, you can simply use the appropriate parameter class.span_sample=lats.Normalize("Me gustaría aqua con gas.",lats.parameters_es_trf)#process textprint(span_sample.toks)['me', 'gustaría', 'aqua', 'con', 'gas']If we want to create a lemmatized representation of a text and add part of speech tags, we can create a slightly altered version of our parameters class:parameters_es_lemmas=lats.parameters_es_trf()parameters_es_lemmas.lemma=True#set lemma to Trueparameters_es_lemmas.pos="pos"#set pos tags to universal pos + fine-grained verb POS tags (universal pos + mood/tense)span_sample_lemma=lats.Normalize("Me gustaría aqua con gas.",parameters_es_lemmas)#process textprint(span_sample_lemma.toks)['yo_PRON', 'gustar_VERB_Cnd', 'aqua_ADJ', 'con_ADP', 'gas_NOUN']Example 2 (Korean), basic features:whtsp_params=lats.parameters()#copy parameterswhtsp_params.sp=False#turn off spacy processingkor_sample=lats.Normalize("피자 좀 주세요",whtsp_params)print(kor_sample.toks)['피자', '좀', '주세요']LicenseThis work is licensed under aCreative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.