Diverse Digital Collections Meet Diverse Uses: Applying Natural
Language Processing to Born-Digital Primary Sources
Christopher A. Lee

Kam Woods

University of North Carolina at ChapelHillcallee@unc.edu

University of North Carolina at ChapelHillkamwoods@unc.edu

ABSTRACTUse of primary sources often focuses on identifying and tracking
entities (e.g. people, places, organizations, events) and othervalues (e.g. dates and times) across documents. There are many
existing open-source natural language processing (NLP) tools that
can identify and report on named entities, and projects in thedigital humanities have previously demonstrated the scholarly
value of NLP approaches when working with digitized materials.
To date, there has been relatively little adoption of NLP tools forthe analysis of born-digital materials by libraries, archives andmuseums (LAMs). There are a variety of challenges associatedwith applying NLP tools to born-digital primary source
collections, including those forensically acquired from removable
media. Many of the challenges relate to the diversity of materials
and potential use cases. This paper reports on the BitCurator NLP
project, which is developing software for LAMs to extract andexpose features in text extracted from such materials. The
resulting services and methods can be used by LAM professionals
and the users they serve.

CCS CONCEPTS• Information systems~Digital libraries and archives•Computing methodologies~Natural language processing•Security and privacy~Data anonymization and sanitization

ADDITIONAL KEYWORDS AND PHRASES
BitCurator NLP; archival processing; named entities; text
processing

INTRODUCTION
A growing body of materials with significant cultural value are“born digital.” Libraries, archives and museums (LAMs) are
increasingly called upon to move born-digital materials from their
original locations – whether those are networked environments orremovable media (e.g. floppy disks, flash drives, CD-ROMs, hard
drives) – into more sustainable preservation environments.
Information professionals must be prepared to extract digitalmaterials from removable media in ways that reflect the richmetadata and ensure the integrity of the materials. They must also
support and mediate appropriate access: allowing users to makesense of materials and understand their context, while also
preventing inadvertent disclosure of sensitive data.
There has been a significant shift in recent years toward theadoption of digital forensics tools and methods by LAMs in order
to meet the above goals. This process has been facilitated by the
BitCurator project (2011-2014), funded by the Andrew W. Mellon
Foundation, which has packaged and disseminated an open-source
software environment1 that allows users to create disk images;
extract data and metadata from disks or directories; scan
bitstreams for the presence of potentially sensitive data values;
characterize the contents of disks; and perform other practicaltasks, such as scanning for viruses, finding duplicate files,
creating and working with forensically packaged disk images,
generating cryptographic hashes, and viewing hexadecimal
representations of bitstreams.
The BitCurator Access project (2014-2016), also funded by the
Andrew W. Mellon Foundation, has further advanced these
activities by investigating mechanisms for providing access toforensically-acquired data. A major product of the project has
been BitCurator Access Webtools, which allows users todynamically navigate filesystems of disk images, as well as
searching over the content of many common files types contained
within the images.2 The project also created BitCurator Access
Redaction Tools to redact strings and byte sequences identified indisk images.3 This includes the ability to overwrite specific strings
or regular expression matches, or byte runs that match specificfiles or directory entries.
Members of the BitCurator user community and other interested
LAM parties have expressed a need for tools to help in identifying
and exploring information based on specific entities (e.g. people,
places, organizations, events) of interest to curators andresearchers. These needs can be addressed by building existing
natural language processing (NLP) and information visualization
tools on top of the existing BitCurator environment and
BitCurator Access Webtools. This combination of functions canbeneficial both LAM staff and a variety of end users of digitalcollections.

RATIONALE
1

https://wiki.bitcurator.net/
https://github.com/bitcurator/bitcurator-access-webtools
3
https://github.com/BitCurator/bitcurator-access-redaction
2

One of the primary motivations for using the BitCurator and
BitCurator Access software is to capture and provide access tocontextual information. For example, the original filesystem
attributes associated with files (e.g. directory paths, timestamps)
can be essential to understanding their provenance and originalorder. There are many other types of contextual information thatcan be vital to making sense and meaningful use of digital objects.
Lee’s “Framework for Contextual Information in Digital
Collections” identifies nine classes of contextual entities: object,
agent, occurrence, purpose, time, place, form of expression,concept/abstraction and relationship [1]. In a study of reference
questions submitted to archives, Duff and Johnson found thatmost information requests were based on “proper names, dates,
places, subject, form, and, occasionally, events when composing
their information request” [2]. In their study of genealogists, Duffand Johnson identified information seeking practices that werefocused primarily on names, places and time periods [3].

PREVIOUS WORK
There have been several recent initiatives to better exploit specifictypes of contextual information from within archival descriptions.
Much of this work has focused on what Lee’s framework wouldclassify as agents: individuals, families and organizations. Twostandardization efforts have focused on characterization of suchcontextual information: Encoded Archival Context (EAC) [4] andthe International Standard Archival Authority Record for
Corporate Bodies, Persons, and Families (ISAAR (CPF)) [5]. The
Social Networks and Archival Context (SNAC) project has
extended the work of EAC-CPF by exploring methods to better
create, combine and disseminate name records for persons,families and corporate bodies. While SNAC has been valuable in
exploiting and exposing metadata from within human-generated
archival descriptions, it has focused on one specific set ofcontextual entities and it has not addressed the automatic
extraction of entities from the content of digital objects
themselves.
Open source natural language processing platforms have matured
rapidly during the past decade. These include platforms thatprovide web services and RESTful application programming
interfaces (APIs) and integration with industry-standard testing
and training corpora. Popular open-source toolkits for natural
language processing include OpenNLP, NLTK, Pattern, andspaCy. Some of these platforms have been used in projects
specifically targeted at LAMs, but the use cases are often narrow,and none include facilities specifically designed to process content
from disk images.
One project that incorporates NLP functionality is ePADD (email
Processing, Appraisal, Discovery, and Delivery), developed by
Stanford University's Special Collections and University Archives[7]. The ePADD software allows LAMs to process collections ofemail by using a customized Named Entity Recognition (NER)
engine to identify correspondents within email. The NLP
functions developed for ePADD have been customized to thedomain of email materials to ensure high-quality results.

Another project with goals closely related to BitCurator NLP was
ArchExtract, conducted (2014-2015) at the University ofCalifornia, Berkeley. Bancroft Library staff worked with a student
at the Berkeley iSchool, Janine Heiser, to develop a prototype for
applying topic modeling, named entity extraction, and analysis ofother common terms to collections of text documents, which
Heiser tested against a collection of digitized materials. Topic
modeling, in this instance, is a text mining technique used tocluster words identified in the corpus into abstract “topics.” Theseclusters may include entities extracted using NER, and can becross-referenced to entities of interest to the researcher. The
ArchExtract software, available through GitHub,4 is a proof ofconcept and not currently being actively maintained.
In the digital humanities, there have been many years of work on
applying NLP to the content of primary sources. Projects in thefield often focus on specific areas of NLP, such as NER and topic
modeling to provide researchers with meaningful views of thepeople, organizations, and events described within a formal
collection or data gathered from the Web. There is great potential
to apply these methods more widely to LAM collections in order
to identify and expose the sorts of contextual entities discussed
above. Examples of such digital humanities projects include
Perseus,5 WordHoard,6 Nora,7 Metadata Offer New Knowledge(MONK),8 and the Software Environment for the Advancement of
Scholarly Research (SEASR)9 being used by the HathiTrust
Research Center (HTRC).10 SEASR provides a modular analytics
approach to improve scholars’ access to digital research materials
through a text mining application capable of summarization, an
ngram viewer, named entity extraction, and entity-to-graphvisualization. This project is no longer active; the last major
release was published in 2013.
Other projects have focused on specific types of contextual
entities, such as PeriodO (Periods, Organized), which is creating agazetteer of scholarly assertions about the spatial and temporalextents of historical periods in order to facilitate linking acrossdata sources [8]. These activities have contributed to the available
infrastructure for processing texts to facilitate research, but they
have been applied predominately to digitized primary sources,
rather than to born-digital sources.

BITCURATOR NLP PROJECT
BitCurator NLP (2016-2018), funded by the Andrew W. Mellon
Foundation and led by the School of Information and Library
Science at the University of North Carolina, Chapel Hill (SILS), isdeveloping and disseminating software that responds directly tothe user needs expressed above by identifying, extracting andexposing contextual entities from the wide diversity of borndigital materials that LAMs already hold and continue to receive.
4

https://github.com/j9recurses/archextract
http://www.perseus.tufts.edu/
http://wordhoard.northwestern.edu/
7
http://web.archive.org/web/20070123113233/http://www.noraproject.org/
8
http://monk.library.illinois.edu/
9
http://www.seasr.org/
10
https://www.hathitrust.org/htrc_collections_tools
56The BitCurator and BitCurator Access projects have defined andtested support for a digital curation workflow that begins at the
point of encountering holdings that reside on removable media –either new acquisitions or materials that are within a repository’s
existing holdings – and extends this to the point of interaction
with end users, providing and supporting a variety of accessmechanisms. BitCurator NLP is extending this earlier work byhelping LAMs to identify and explore information based on
specific entities (e.g. people, places, organizations, events) ofinterest to curators and researchers.
Our target use cases differ from previous work in two
fundamental ways. First, disk images are internally complex(containing filesystems and sometimes bootable operating
systems) and require a significant software dependency stack thatis already provided by the BitCurator environment and BitCurator
Access Webtools. These include the ability to read, mount andprovide access to the contents of various filesystems, as well as
extracting, presenting and reporting on files and metadata.
A second distinguishing factor in the application of NLP to disk
images in LAM collections is that disks may contain a broad
range of file types and data encodings, requiring substantial preprocessing to extract content so that it can be processed by NLP
tools and organized into meaningful reports, access points andvisualizations. By contrast, most previous applications of NLP
methods have focused on more “well-behaved” collections or
corpora with more consistent types of content. BitCurator NLP isbuilding from and extending a variety of tools and initiatives
discussed above to provide services that LAMs can be run
independently or integrate into existing software environments
and access portals via simple application programming interfaces(APIs).
Some LAM access systems in use today incorporate NLP to
describe the contents of collections, but this technology is often
tightly coupled to the platform being used or is applied strictly to
file types that tend to share common structures and metadata. An
example of this is email, which contains raw text but alsosignificant structured metadata in the header that can be used tocue identification of persons and organizations and describe theirrelationships. Identifying entities, relationships, and other featuresof interest by processing open text from heterogeneous collectionsof files (such as those extracted from a disk image) is inherently“noisier,” as the extracted text will often contain patterns offeatures (such as persons, places, and organizations) common to awide range of devices and production environments (e.g.,documentation of system files).
BitCurator NLP is exploring approaches that focus on improving
the utility of reports produced about the contents of born-digitalcollections. Using data extracted from open text using NLP tools,
along with techniques described in recent digital forensics
research to eliminate or deemphasize those that appear to beirrelevant or common to the system rather than the documents
themselves (e.g., names and email addresses of developers ororganizations that created the software used to produce a given
document), the project team will also develop guidelines
describing how to apply the tools in ways that support common
access and research use cases.

The BitCurator NLP team is ensuring close integration between
the existing functionality of the BitCurator environment,
BitCurator Access Webtools and the software developed by the
BitCurator NLP project. For example, we are increasingly making
elements of the BitCurator environment available as selfcontained software installers (specifically software packages that
may be installed in Ubuntu and Debian Linux environments using
existing package managers), so users can selectively install andupdate them as they find most useful. Institutions could load eachof the access tools onto the same machine (or virtual machine) as
the one they are using for the initial processing, or they could
instead decide to run those tasks in different environments inorder to better manage and allocate their resources.

CONCLUSION
Given the numerous projects that have focused on application of
NLP to enable scholarship on older primary sources, it is striking
how little investigation there has been into use of NLP to thegrowing volume of more contemporary born-digital materials
acquired by LAMs. While these professionals may have access tosome tools that apply a subset of NLP techniques (such as topic
modeling, NER, and word or phrase frequency analysis) to
specific collections or file types, they generally lack software
allowing them to execute these techniques in aggregate for large
sets of heterogeneous files (including complex files such as disk
images). This is particularly important for materials that containthousands or hundreds of thousands of files, when it is intractableto manually inspect materials to determine which of the files are
relevant preservation targets and what relationships hold between
the files that could be exposed in archival description. Providing
mechanisms that allow LAM professionals to conduct this type ofanalysis in the existing BitCurator environment and in BitCurator
Access Webtools will streamline this process. The BitCurator
NLP software is enabling LAMs to incorporate (or improve) basic
NLP capabilities within existing access environments (or establish
them as a dedicated service) by providing a service layer that
reads files from a digital archival store and produces reports forend users on demand.

ACKNOWLEDGEMENTS
This work has been funded by the Andrew W. Mellon Foundation.
The BitCurator NLP team is composed of Jacob Hill, Christopher
A. Lee, Sunitha Misra and Kam Woods. We have also benefited
from contributions of the project Advisory Board.

REFERENCES[1] Lee, C. A. Lee, "A Framework for Contextual Information in
Digital Collections." Journal of Documentation 67, no.1 (2011):
95-143.
[2] Duff, W. M. Johnson, C. A. A Virtual Expression of Need: An
Analysis of E-Mail Reference Questions. American Archivist 64(2001): 43-60.

[3] Duff, W.M. and Johnson, C. A. Johnson. Where is the List
with All the Names? Information-Seeking Behavior of
Genealogists. American Archivist 66 (2003): 79-95.
[6] Pitti, D., Hu, R. Larson, R., Tingle, B. and Turner A. Social
Networks and Archival Context: From Project to Cooperative
Archival Program. Journal of Archival Organization 12 (2015):
77-97.
[7] Hangal, S., Piratia, V., Manovit, C., Chan, P., Edwards, G. andLam, M. S. Historical Research Using Email Archives.
Proceedings of the 33rd Annual ACM Conference Extended
Abstracts on Human Factors in Computing Systems. ACM Press,
New York, 2015.
[8] Golden, P. Shaw, R. Nanopublication Beyond the Sciences:
The PeriodO Period Gazetteer. PeerJ Computer Science 2:e44(2016).