Package: htrc-feature-reader
HTRC-FeaturesTools for working with theHTRC Extracted Features dataset, a dataset of page-level text features extracted from 17 million digitized works.This library provides aFeatureReaderfor parsing files, which are handled asVolumeobjects with collections ofPageobjects. Volumes provide access to metadata (e.g. language), volume-wide feature information (e.g. token counts), and access to Pages. Pages allow you to easily parse page-level features, particularly token lists.This library makes heavy use ofPandas, returning many data representations as DataFrames. This is the leading way of dealing with structured data in Python, so this library doesn't try to reinvent the wheel. Since refactoring around Pandas, the primary benefit of using the HTRC Feature Reader is performance: reading the json structures and parsing them is generally faster than custom code. You also get convenient access to common information, such as case-folded token counts or part-of-page specific character counts. Details of the public methods provided by this library can be found in theHTRC Feature Reader docs.Table of Contents:Installation|Usage|Additional NotesLinks:HTRC Feature Reader Documentation|HTRC Extracted Features DatasetCitation:
Peter Organisciak and Boris Capitanu, "Text Mining in Python through the HTRC Feature Reader,"Programming Historian, (22 November 2016),http://programminghistorian.org/lessons/text-mining-with-extracted-features.InstallationTo install,pipinstallhtrc-feature-readerThat's it! This library is written for Python 3.0+. For Python beginners, you'll needpip.Alternately, if you are usingAnaconda, you can install withcondainstall-chtrchtrc-feature-readerThecondaapproach is recommended, because it makes sure that some of the hard-to-install dependencies are properly installed.Given the nature of data analysis, using iPython with Jupyter notebooks for preparing your scripts interactively is a recommended convenience. Most basically, it can be installed withpip install ipython[notebook]and run withipython notebookfrom the command line, which starts a session that you can access through your browser. If this doesn't work, consult theiPython documentation.Optional:installing the development version.UsageNote: for new Python users, a more in-depth lesson is published by Programming Historian:Text Mining in Python through the HTRC Feature Reader. That lesson is also the official citation associated the HTRC Feature Reader library.Reading feature filesThe easiest way to start using this library is to use the Volume interface, which takes a path to an Extracted Features file.fromhtrc_featuresimportVolumevol=Volume('data/ef2-stubby/hvd/34926/hvd.32044093320364.json.bz2')volThe Nautilus.byDelaware Museum of Natural History.(1904, 222 pages) -hvd.32044093320364The FeatureReader can also download files at read time, by reference to a HathiTrust volume id. For example, if I wantboth of volumes of Pride and Prejudice, I can see that the URLs are babel.hathitrust.org/cgi/pt?id=hvd.32044013656053and babel.hathitrust.org/cgi/pt?id=hvd.32044013656061. In the FeatureReader, these can be called with theids=[]argument, as follows:forhtidin["hvd.32044013656053","hvd.32044013656061"]:vol=Volume(htid)print(vol.title,vol.enumeration_chronology)Pride and prejudice. v.1
Pride and prejudice. v.2This downloads the file temporarily, using the HTRC's web-based download link (e.g.https://data.analytics.hathitrust.org/features/get?download-id={{URL}}). One good pairing with this feature is theHTRC Python SDK's functionality for downloading collections.For example, I have a small collection of knitting-related books athttps://babel.hathitrust.org/cgi/mb?a=listis&c=1174943610. To read the feature files for those books:fromhtrcimportworksetvolids=workset.load_hathitrust_collection('https://babel.hathitrust.org/cgi/mb?a=listis&c=1174943610')FeatureReader(ids=volids).first().titleRemember that for large jobs, it is faster to download your dataset beforehand, using thersyncmethod.VolumeAVolumecontains information about the current work and access to the pages of the work. All the metadata fields from the HTRC JSON file are accessible as properties of the volume object, includingtitle,language,imprint,oclc,pubDate, andgenre. The main identifieridandpageCountare also accessible, and you can find the URL for the Full View of the text in the HathiTrust Digital Library - if it exists - withvol.handle_url."Volume{}is a{}page text from{}written in{}. You can doublecheck at{}".format(vol.id,vol.page_count,vol.year,vol.language,vol.handle_url)'Volume hvd.32044013656061 is a 306 page text from 1903 written in eng. You can doublecheck at http://hdl.handle.net/2027/hvd.32044013656061'This is theExtracted Featuresdataset, so the features are easily accessible. To most popular is token counts, which are returned as a Pandas DataFrame:df=vol.tokenlist()df.sample(10)<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>countpagesectiontokenpos201bodyabodeNN1117bodyheadNN1126bodyforIN1210bodythreeCD1224bodywouldMD189bodyTheDT1283bodyanyDT163bodysurpriseNN1152bodymakeVB1170bodyIPRP3Other extracted features are discussed below.The full included metadata can be seen withvol.parser.meta:vol.parser.meta.keys()dict_keys(['id', 'metadata_schema_version', 'enumeration_chronology', 'type_of_resource', 'title', 'date_created', 'pub_date', 'language', 'access_profile', 'isbn', 'issn', 'lccn', 'oclc', 'page_count', 'feature_schema_version', 'ht_bib_url', 'genre', 'handle_url', 'imprint', 'names', 'source_institution', 'classification', 'issuance', 'bibliographic_format', 'government_document', 'hathitrust_record_number', 'rights_attributes', 'pub_place', 'volume_identifier', 'source_institution_record_number', 'last_update_date'])These fields are mapped to attributes inVolume, sovol.oclcwill return the oclc field from that metadata. As a convenience,Volume.yearreturns thepub_dateinformation andVolume.authorreturns thecontributor information.vol.year,vol.author('1903', ['Austen, Jane 1775-1817 '])If the minimal metadata included with the extracted feature files is insufficient, you can fetch HT's metadata record from the Bib API withvol.metadata.
Remember that this calls the HTRC servers for each volume, so can add considerable overhead. The result is a MARC file, returned as apymarcrecord object. For example, to get the publisher information from field260:vol.metadata['260'].value()'Boston : Little, Brown, 1903.'At large-scales, usingvol.metadatais an impolite and inefficient amount of server pinging; there are better ways to query the API than one volume at a time. Read about theHTRC Solr Proxy.Another source of bibliographic metadata is the HathiTrust Bib API. You can access this information through the URL returned withvol.ht_bib_url:vol.ht_bib_url'http://catalog.hathitrust.org/api/volumes/full/htid/hvd.32044013656061.json'Volumes also have direct access to volume-wide info of features stored in pages. For example, you can get a list of words per page throughVolume.tokens_per_page(). We'll discuss these featuresbelow, after looking first at Pages.Note that for the most part, the properties of thePageandVolumeobjects aligns with the names in the HTRC Extracted Features schema, except they are converted to followPython naming conventions: converting theCamelCaseof the schema tolowercase_with_underscores. E.g.beginLineCharsfrom the HTRC data is accessible asPage.begin_line_chars.The fun stuff: playing with token counts and character countsToken counts are returned byVolume.tokenlist()(orPage.tokenlist(). By default, part-of-speech tagged, case-sensitive counts are returned for the body.The token count information is returned as a DataFrame with a MultiIndex (page, section, token, and part of speech) and one column (count).print(vol.tokenlist()[:3])count
page section token  pos       
1    body    Austen .        1
             Pride  NNP      1
             and    CC       1Page.tokenlist()can be manipulated in various ways. You can case-fold, for example:tl=vol.tokenlist(case=False)tl.sample(5)<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>countpagesectionlowercasepos218bodywhatWP130bodypemberleyNNP1213bodycomesVBZ2183bodytookVBD151bodynecessaryJJ1Or, you can combine part of speech counts into a single integer.tl=vol.tokenlist(pos=False)tl.sample(5)<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>countpagesectiontoken264bodyfamily247bodyjourney198bodyPerhaps149bodyat2227bodyso1Section arguments are valid here: 'header', 'body', 'footer', 'all', and 'group'tl=vol.tokenlist(section="header",case=False,pos=False)tl.head(5)<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>countpagesectionlowercase9header's1and1austen1jane1prejudice1You can also drop the section index altogether if you're content with the default 'body'.vol.tokenlist(drop_section=True,case=False,pos=False).sample(2)<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>countpagelowercase247suppose176would2The MultiIndex makes it easy to slice the results, and it is althogether more memory-efficient. For example, to return just the nouns (NN):tl=vol.tokenlist()tl.xs('NN',level='pos').head(4)<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>countpagesectiontoken1bodyprejudiceJane19bodyVolume110bodyvol312body./■1If you are new to Pandas DataFrames, you might find it easier to learn by converting the index to columns.simpler_tl=df.reset_index()simpler_tl[simpler_tl.pos=='NN']<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>pagesectiontokenposcount31bodyprejudiceJaneNN1199bodyVolumeNN14010bodyvolNN35112body./■NN15312body/NN1..................43178297bodyspiteNN143187297bodyuncleNN143191297bodywarmestNN143195297bodywifeNN143226305bodyNON-RECEIPTNN17224 rows × 5 columnsIf you prefer not to use Pandas, you can always convert the object, with methods liketo_dictandto_csv).tl[:3].to_csv()'page,section,token,pos,count\n1,body,Austen,.,1\n1,body,Pride,NNP,1\n1,body,and,CC,1\n'To get just the unique tokens,Volume.tokensprovides them as a set. Here I select a specific page for brevity and a minimum count, but you can run the method without arguments.vol.tokens(page_select=21,min_count=5){'"', ',', '.', 'You', 'been', 'have', 'his', 'in', 'of', 'the', 'you'}In addition to token lists, you can also access other section features:vol.section_features()<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>tokenCountlineCountemptyLineCountcapAlphaSeqsentenceCountpage14101121510421300000400000500000..................302000003030000030400000305491123330623111306 rows × 5 columnsChunkingIf you're working in an instance where you hope to have comparably sized document units, you can use 'chunking' to roll pages into chunks that aim for a specific length. e.g.by_chunk=vol.tokenlist(chunk=True,chunk_target=10000)print(by_chunk.sample(4))# Count words per chunkby_chunk.groupby(level='chunk').sum()count
chunk section token      pos       
5     body    husbands   NNS      3
2     body    frequently RB       3
              domestic   JJ       3
3     body    :          :       10<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>countchunk1124532988839887410129510054610065712327MultiprocessingFor large jobs, you'll want to use multiprocessing or multithreading to speed up your process. This is left up to your preferred method, either within Python or by spawning multiple scripts from the command line. Here are two approaches that I like.DaskDask offers easy multithreading (shared resources) and multiprocessing (separate processes) in Python, and is particularly convenient because it includes a subset of Pandas DataFrames.Here is a minimal example, that lazily loads token frequencies from a list of volume IDs, and counts them up by part of speech tag.importdask.dataframeasddfromdaskimportdelayeddefget_tokenlist(vol):''' Load a one volume feature reader, get that volume, and return its tokenlist '''returnFeatureReader(ids=[volid]).first().tokenlist()delayed_dfs=[delayed(get_tokenlist)(volid)forvolidinvolids]# Create a daskddf=(dd.from_delayed(delayed_dfs).reset_index().groupby('pos')[['count']].sum())# Run processingddf.compute()Here is an example of 78 volumes being processed in 24 seconds with 31 threads:This example used multithreading. Due to the nature of Python, certain functions won't parallelize well. In our case, the part where the JSON is read from the file and converted to a DataFrame (the light green parts of the graphic) won't speed up because Python dicts lock the Global Interpreter Lock (GIL). However, because Pandas releases the GIL, nearly everything you do after parsing the JSON will be very quick.To better understand what happens whenddf.compute(), here is a graph for 4 volumes:GNU ParallelAs an alternative to multiprocessing in Python, my preference is to have simpler Python scripts and to use GNU Parallel on the command line. To do this, you can set up your Python script to take variable length arguments of feature file paths, and to print to stdout.This psuedo-code shows how that you'd use parallel, where the number of parallel processes is 90% the number of cores, and 50 paths are sent to the script at a time (if you send too little at a time, the initialization time of the script can add up).findfeature-files/-name'*json.bz2'|parallel--eta--jobs90%-n50pythonyour_script.py>output.txtAdditional NotesInstalling the development versiongit clone https://github.com/htrc/htrc-feature-reader.git
cd htrc-feature-reader
python setup.py installIterating through the JSON filesIf you need to do fast, highly customized processing without instantiating Volumes, FeatureReader has a convenient generator for getting the raw JSON as a Python dict:fr.jsons(). This simply does the file reading, optional decompression, and JSON parsing.Downloading files within the libraryutilsincludes an Rsyncing utility,download_file. This requires Rsync to be installed on your system.Usage:Download one file to the current directory:utils.download_file(htids='nyp.33433042068894')Download multiple files to the current directory:ids = ['nyp.33433042068894', 'nyp.33433074943592', 'nyp.33433074943600']
utils.download_file(htids=ids)Download file to/tmp:utils.download_file(htids='nyp.33433042068894', outdir='/tmp')Download file to current directory, keeping pairtree directory structure,
i.e../nyp/pairtree_root/33/43/30/42/06/88/94/33433042068894/nyp.33433042068894.json.bz2:utils.download_file(htids='nyp.33433042068894', keep_dirs=True)
    ```

### Getting the Rsync URL

If you have a HathiTrust Volume ID and want to be able to download the features for a specific book, `hrtc_features.utils` contains an [id_to_rsync](http://htrc.github.io/htrc-feature-reader/htrc_features/utils.m.html#htrc_features.utils.id_to_rsync) function. This uses the [pairtree](http://pythonhosted.org/Pairtree/) library but has a fallback written with that library is not installed, since it isn't compatible with Python 3.


```python
from htrc_features import utils
utils.id_to_rsync('miun.adx6300.0001.001')'miun/pairtree_root/ad/x6/30/0,/00/01/,0/01/adx6300,0001,001/miun.adx6300,0001,001.json.bz2'See theID to Rsync notebookfor more information on this format and on Rsyncing lists of urls.There is also a command line utility installed with the HTRC Feature Reader:$htid2rsyncmiun.adx6300.0001.001
miun/pairtree_root/ad/x6/30/0,/00/01/,0/01/adx6300,0001,001/miun.adx6300,0001,001.json.bz2Advanced FeaturesIn the beta Extracted Features release, schema 2.0, a few features were separated out to an advanced files. However,this designation is no longer present starting with schema 3.0, meaning information likebeginLineChars,endLineChars, andcapAlphaSeqare always available:# What is the longest sequence of capital letter on each page?vol.cap_alpha_seqs()[:10][0, 1, 0, 0, 0, 0, 0, 0, 4, 1]end_line_chars=vol.end_line_chars()print(end_line_chars.head())count
page section place char       
2    body    end   -         1
                   :         1
                   I         1
                   f         1
                   t         1# Find pages that have lines ending with "!"idx=pd.IndexSliceprint(end_line_chars.loc[idx[:,:,:,'!'],].head())count
page section place char       
45   body    end   !         1
75   body    end   !         1
77   body    end   !         1
91   body    end   !         1
92   body    end   !         1TestingThis library is meant to be compatible with Python 3.2+ and Python 2.7+. Tests are written for py.test and can be run withsetup.py test, or directly withpython -m py.test -v.If you find a bug, leave an issue on the issue tracker, or contact Peter Organisciak atorganisciak+htrc@gmail.com.