Package: lltk-dh
Literary Language Toolkit (LLTK)Corpora, models, and tools for the study of complex language.QuickstartSeethis notebookfor a more interactive quickstart (run the code here on Binder).InstallOpen a terminal, Jupyter, or Colab notebook and type:pip install -qU lltk-dh

# or for very latest version:
#pip install -qU git+https://github.com/quadrismegistus/lltkShowavailable corpora:lltk showOr, within python, show in markdown:importlltklltk.show()Play with corporaSeebelowfor available corpora.# Load/install a corpusimportlltkcorpus=lltk.load('ECCO_TCP')# load the corpus by name or ID# Metadatameta=corpus.meta# metadata as data framesmpl=meta.query('1770<year<1830')# easy query access# Datamfw=corpus.mfw()# get the 10K most frequent words as a listdtm=corpus.dtm()# get a document-term matrix as a pandas dataframedtm=corpus.dtm(tfidf=True)# get DTM as tf-idfmdw=corpus.mdw('gender')# get most distinctive words for a metadata groupPlay with texts# accessing text objstexts=corpus.texts()# get a list of corpus's text objectstexts_smpl=corpus.texts(smpl)# text objects from df/list of idstexts_rad=corpus.au.Radcliffe# hit "tab" after typing e.g. "Rad" to autocompletetext=corpus.t# get a random text object from corpus# metadata accesstext_meta=text.meta# get text metadata as dictionaryauthor=text.author# get common metadata as attributestitle=text.titleyear=text.yeardec=text.decade# few inferred attributes toodec_str=text.decade_str# '1890-1900' rather than 1890# data accesstxt=text.txt# get plain text as stringxml=text.xml# get xml as string# simple nlpwords=text.words# get list of words (excl punct)sents=text.sents# get list of sentencescounts=text.counts# get word counts as dictionary (from JSON if saved)# other nlptnltk=text.nltk# get nltk Text objecttblob=text.blob# get TextBlob objecttstanza=text.stanza# get list of stanza objects (one per para)tspacy=text.spacy# get list of spacy objects (one per para)Available corporaLLTK has built in functionality for the following corpora. Some (🌞) are freely downloadable from the links below or the LLTK interface. Some of them (☂) require first accessing the raw data through your institutional or other subscription. Some corpora have a mixture, with some data open through fair research use (e.g. metadata, freqs) and some closed (e.g. txt, xml, raw).namedesclicensemetadatafreqstxtxmlrawARTFLAmerican and French Research on the Treasury of the French LanguageAcademic☂️☂️BPOBritish Periodicals OnlineCommercial☂️☂️CLMETCorpus of Late Modern English TextsAcademic🌞🌞☂️☂️COCACorpus of Contemporary American EnglishCommercial☂️☂️☂️☂️COHACorpus of Historical American EnglishCommercial☂️☂️☂️☂️ChadwyckChadwyck-Healey Fiction CollectionsMixed🌞🌞☂️☂️☂️ChadwyckDramaChadwyck-Healey Drama CollectionsMixed☂️☂️☂️☂️☂️ChadwyckPoetryChadwyck-Healey Poetry CollectionsMixed☂️☂️☂️☂️☂️ChicagoU of Chicago Corpus of C20 NovelsAcademic🌞🌞☂️DTADeutsches Text ArchivFree🌞🌞🌞🌞🌞DialNarrDialogue and Narration separated in Chadwyck-Healey NovelsAcademic🌞🌞☂️ECCOEighteenth Century Collections OnlineCommercial☂️☂️☂️☂️☂️ECCO_TCPECCO (Text Creation Partnership)Free🌞🌞🌞🌞🌞EEBO_TCPEarly English Books Online (curated by the Text Creation Partnership)Free🌞🌞🌞🌞ESTCEnglish Short Title CatalogueAcademic☂️EnglishDialoguesA Corpus of English Dialogues, 1560-1760Academic🌞🌞🌞EvansTCPEarly American FictionFree🌞🌞🌞🌞🌞GaleAmericanFictionGale American Fiction, 1774-1920Academic🌞🌞☂️☂️GildedAgeU.S. Fiction of the Gilded AgeAcademic🌞🌞🌞HathiBioBiographies from Hathi TrustAcademic🌞🌞HathiEngLitFiction, drama, verse word frequencies from Hathi TrustAcademic🌞🌞HathiEssaysHathi Trust volumes with "essay(s)" in titleAcademic🌞🌞HathiLettersHathi Trust volumes with "letter(s)" in titleAcademic🌞🌞HathiNovelsHathi Trust volumes with "novel(s)" in titleAcademic🌞🌞HathiProclamationsHathi Trust volumes with "proclamation(s)" in titleAcademic🌞🌞HathiSermonsHathi Trust volumes with "sermon(s)" in titleAcademic🌞🌞HathiStoriesHathi Trust volumes with "story/stories" in titleAcademic🌞🌞HathiTalesHathi Trust volumes with "tale(s)" in titleAcademic🌞🌞HathiTreatisesHathi Trust volumes with "treatise(s)" in titleAcademic🌞🌞InternetArchive19th Century Novels, curated by the U of Illinois and hosted on the Internet ArchiveFree🌞🌞🌞LitLabLiterary Lab Corpus of 18th and 19th Century NovelsAcademic🌞🌞☂️MarkMarkMark Algee-Hewitt's and Mark McGurl's 20th Century CorpusAcademic🌞🌞☂️OldBaileyOld Bailey OnlineFree🌞🌞🌞🌞RavenGarsideRaven & Garside's Bibliography of English Novels, 1770-1830Academic☂️SOTUState of the Union AddressesFree🌞🌞🌞Sellers19th Century Texts compiled by Jordan SellersFree🌞🌞🌞SemanticCohortCorpus used in "Semantic Cohort Method" (2012)Free🌞SpectatorThe Spectator (1711-1714)Free🌞🌞🌞TedJDHCorpus used in "Emergence of Literary Diction" (2012)Free🌞🌞🌞TxtLabA multilingual dataset of 450 novelsFree🌞🌞🌞🌞DocumentationIncomplete for now. Seethis sample notebookfor some examples.New corpusImport a corpus into LLTK:lltk import                           # use the "import" command \
  -path_txt mycorpus/txts             # a folder of txt files  (use -path_xml for xml) \
  -path_metadata mycorpus/meta.xls    # a metadata csv/tsv/xls about those txt files \
  -col_fn filename                    # .txt/.xml filename col in metadata (use -col_id if no ext)Or create a new one:lltk createMost frequent wordscorpus.mfw_df(n=None,# Number of top words overallby_ntext=False,# Count number of documents not number of wordsby_fpm=False,# Count by within-text relative sumsmin_count=None,# Minimum count of wordyearbin=None,# Average relative counts across `yearbin` periodscol_group='period',# Which column to periodize onn_by_period=None,# Number of top words per periodkeep_periods=True,# Keep periods in output dataframen_agg='median',# How to aggregate across periodsmin_periods=None,# minimum number of periods a word must appear inexcl_stopwords=False,# Exclude stopwords (at `PATH_TO_ENGLISH_STOPWORDS`)excl_top=0,# Exclude words ranked 1:`not_top`valtype='fpm',# valtype to compute top words by**attrs)Document term matrixcorpus.dtm(words=[],# words to use in DTMn=25000,# if not `words`, how many mfw?texts=None,# set texts to use explicitly, otherwise use alltf=False,# return term frequencies, not countstfidf=False,# return tfidf, not countsmeta=False,# include metadata (e.g. ["gender","nation"])**mfw_attrs,# all other attributes passed to self.mfw())Most distinctive wordscorpus.mdw(groupby,# metadata categorical variable to group bywords=[],# explicitly set words to usetexts=None,# explicitly set texts to usetfidf=True,# use tfidf as mdw calculationkeep_null_cols=False,# remove texts which do not have `groupby` setremove_zeros=True,# remove rows summing to zeroagg='median',# aggregate by `agg`)