Package: litcorpt
litcorptLITeraryCORpus inPorTuguese is a API to access a literary
corpus in portuguese language.The API provides access to the corpus without all the fuzz to download and write
a loader for different types of data sources. It is exposed as a simple document
database.How to install.Simply:pip install litcorptGetting startedAfter installation in you Python justimport litcorpt
from pprint  import pprint as pp
corpus_db = litcorpt.corpus_load()
print(f'There are {len(corpus_db)} documents in corpus')It will load the whole corpus. When running by the first time, it will download
from internet, process and build the whole dataset.The download size is around 600MB and is automaticly handled by the library. It
is downloaded just at first time you load it. After the first time it will load
from local disk. The time to load data locally takes around 6 seconds. This value
was measured in my own computer (your mileage may vary).Basic UsageMost of time you just want to retrieve the whole corpus as alistof text
documents.  You can do that with this one liner.corpus = litcorpt.corpus(corpus_db)This operation just append to a list all contents for all documents. Since a
document may have more than one content.Advanced usageBesides the fetchall usage, many custom queries can be done. Is possible to
search by matches, regexes, fields.Book metadataTo retrieve metadata from all booksmetadata = litcorpt.metadata(corpus_db)Metadata will be a list of books (model.Book). Each i-th element is a metadata
related to i-th text fromcorpusvariable from last example.You can convert this book metadata to a dictionary withmetadata[0].dict(exclude_none=True, exclude_defaults=True)or to a JSON objectmetadata[0].json(exclude_none=True, exclude_defaults=True)All book titles of an author (Eça de Queirós)We are ignoring documents where Queirós is an editor.As a regularforloopq = litcorpt.Query()
result = corpus_db.search(q.creator.any(q.name == 'Eça de Queirós'))

titles = []
for document in result:
  titles.append(document['title'])

pp(titles)As a list comprehension shorter but harder to read.q = litcorpt.Query()
titles = [ document['title'] for document in corpus_db.search(q.creator.any(q.name == 'Eça de Queirós'))]
pp(titles)Building a corpus with Eça de Queirósq = litcorpt.Query()
search = (q.creator.any(q.name == 'Eça de Queirós'))
queiros_corpus = litcorpt.corpus(corpus_db, search)
pp(queiros_corpus)Building a bibliographyHere we handle the case where there is no author.bibliography = []
for document in iter(corpus_db):
    creators = []
    for creator in document.get('creator', [{'name': 'Anonymous'}]):
        creators.append(creator['name'])
    bibliography.append(f'{" and ".join(creators)}. {document["title"].strip()}.')

pp(bibliography)Count documents by AuthorHere we use Python's Counter to count the surnames and using a dict
comprehension to filter the authors that occurs more than 5 times. You still can
access the whole counting thenamevariableAs a list comprehensionq = litcorpt.Query()
from collections import Counter
names = Counter([ creator['name'] for document in corpus_db.search(q.creator.exists()) for creator in document['creator'] ])
most_common_names = {name: count for name, count in names.items() if count >= 5}

print(most_common_names)Unrolling the comprehensionq = litcorpt.Query()
from collections import Counter

names = []

for document in corpus_db.search(q.creator.exists()):
  for creator in document['creator']:
    names.append(creator['name'])

names = Counter(names)

most_common_names = {}
for name, count in names.items():
  if count >= 5:
    most_common_names[name] = countExtra: Sorting by decreasing frequency, then alphabeticaly.sorted(most_common_names.items(), key=lambda item: (-item[1], item[0]))Display all SubjectsFirst we group all subjectsq = litcorpt.Query()
subjects = [subject
            for document in iter(corpus_db)
            if document['subject'] is not None
            for subject in document['subject']]Then we can count, and sort by descending frequency (Python 3.6> dicts are
ordered by default).from collections import Counter
subject_frequency = Counter(subjects)
subject_frequency = dict(sorted(subject_frequency.items(), key=lambda item: -item[1]))And also group the unique items for reference.subject_list = list(subject_frequency.keys())Building a corpus given a list of SubjectsFirst we pick a list of subjects (this is just an example with a few valid
entries, and some not valid).subjects = [ 'portuguese drama',
             'france',
             'drama',
             'women',
             '<INVALID SUBJECT>' ]Then we proceed with search and corpus buildingq = litcorpt.Query()
result = corpus_db.search(q.subject.any(subjects))
drama_corpus = [book['contents'] for book in result]If we want we can easily list the titles in our newdrama_corpustitles = [ document['title'] for document in result ]Of course we can do the same by any of the fields in document.Retrieving a document by IDq = litcorpt.Query()
search  = q.creator.any(q.name == 'Joaquim Manuel de Macedo')
doc_ids = litcorpt.doc_id(corpus_db, search)

for doc_id in doc_ids:
  print(corpus_db.get(doc_id=doc_id)['title'])ExtraYou can check the 'tests' dir in source_code for examples.The structure of a document.The corpus database is a list of documents. A document is often related with a
literary document (book, text, play, etc) and contains the following fields:FieldExplanationindexAn unique string to internaly identify the entry.titleA title associated to the entry.subtitleDocument subtitle (if exists)creatorA list of creators. Each creator contains:roleCreator relationship with the book entrynamecreator namebirthCreator's birth year.deathCreator's death year.placeCreator's birth place.languageA list of ISO entry with language, pt_BR or pt are the most common here.  A document can contain many languages.  Most of time just one.publishedDate of first publish. Multiple edition should use the date of first edition. Except when large changes happened in document, as change of translator, change of ortography.identifierA unique global identifier, often a ISBN13 for books.original_languageOriginal language of document. Using ISO entry for language.subjectA list entry subjects. As example: Fantasy, Science-Fiction, Kids.  Use lower caps always.genreA list of literary genre: Novel, Poetry, Lyrics, Theather.  Use lower caps always.ortographyReference to which Portuguese ortography is being used.abstractThe book abstract/resume.notesAny notes worth of note.contentsBook contents. The text itself.Caution: Date fields must contain a datetime.date or a string in format "YYYY-MM-DD"CustomizingBy default, the corpus is stored at${HOME}/litcorpt_dataIf you wish to put in a different place, just set theCORPUS_DATAPATHenvironment variable in your system configuration. For example for bash, add
this to your~/.bashrcexport CORPUS_DATAPATH="/whatever/place/you/want"or you can create a.envfile in your project root withCORPUS_DATAPATH="/whatever/place/you/want"This file will be loaded by litcorpt.Then call your programs usinglitcorptor youripythonsessionTODORewrite DOCSTRINGS using numpy style:https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standardFrom a Iterator of books (as the new package search functions) return:
a. contensts only (litcorpt._c)
b. Metadata only (litcorpt._m)Add the new module search functions to README documentationCheck if tests still work with new book. model.Check if functions are returning generators instead lists.