and Information Science University of North Texas , Denton , TX 76203

The field of Web Archiving exists in a fluid, fragmented, and heterogeneous state. Part of the problem is that this field is relatively new and its literature is scattered across a wide range of journal and conference venues. This makes the state of Web Archiving as a discipline particularly difficult to ascertain. This paper presents an approach to building a collection of articles about the subject. We begin with a small dataset of articles taken from a Web Archiving Bibliography and then proceed to expand it by crawling the Web and collecting additional documents. The crawled documents are then classified using machine learning classification techniques. We show that by extracting the documents’ titles and abstracts and representing them using the “bag of words” approach, we are able to accurately identify documents from the Web crawler as documents that are about Web Archiving. We also discuss our results in the context of Web Archiving as an emerging field. INTRODUCTION The field of Web Archiving arose to address fears of a Digital Dark Age, caused by the gradual disappearance of digital information (Kuny, 1997). Many institutions to date have implemented Web Archiving programs, a notable example being the Internet Archive, which in 1996 began to capture snapshots of the entire Web with the purpose of preserving them for future generations. Additionally, many national libraries began archiving their own national domains as part of an effort to preserve their digital cultural heritage. University libraries also followed suit, often looking to expand on the strengths of their existing physical collections. As a nascent field, Web Archiving exists in an uncertain and continuously evolving state. Although there have been several efforts to establish standards and coordinate Web Archiving initiatives, for example, through the foundation of the International Internet Preservation Consortium (IIPC) in 2003, the field remains fluid, fragmented, and heterogeneous, and consequently, so does its literature. Published articles on Web Archives are relatively few compared to older, more established disciplines, and are scattered across a wide range of journals and conferences, including the ACM Web Science Conference (WebSci), the Joint Conference on Digital Libraries (JCDL), and the D-Lib Magazine. For example, a query for the phrase “Web Archiving” in a well-known database such as Web of Science (Thomson Reuters, 2014) returns 27 results, whereas queries for phrases “information retrieval” and “information science” in the same database return 880 and 3,586 results, respectively. Authors who do research in Web Archiving generally do not have official scholarly journals or publication venues, which can provide a sense of the progress or evolution of their field. In short, the state of Web Archiving as a discipline is currently almost impossible to discern. This fact presents a challenge to a researcher interested in understanding the field: What is the current state of scholarly publication in the field of Web Archiving? The current state of a field cannot be ascertained without a corpus of publications in that field that can be examined. To address the above challenge, we pose our main research question: How do we gather and understand a corpus of Web Archiving research articles, given the scattered nature of the field? In this paper, we present a process, grounded in information retrieval and machine learning techniques, for gathering a corpus of literature about an emerging field. RELATED WORK In the Information Science field, there has been much work done on the subject of exploring and analyzing academic disciplines, usually by making use of bibliometric data. In their prominent study, White and McCain (1998) conducted an extensive domain analysis of the field of Information Science utilizing data from Social Scisearch. They presented a variety of visualizations of the field, such as the most prominent authors, major sub-disciplines, and paradigm shifts over time. Chen (2006) utilized the Java application CiteSpace II to provide an overview of the trends and patterns in the scientific literature of the research fields of mass extinction and terrorism. More recently, Wang and Tang (2013) mapped the development of the emerging field of open innovation using data from Web of Science and CiteSpace II. We would like to highlight the fact that these research efforts differ from ours in one key factor: the aforementioned authors were working in fields with a strong presence in academic databases and citations indexes. This abundance of bibliometric data and research publications made the task of compiling data and corpus building a substantially less difficult task. This situation is not the case with the field of Web Archiving, and so we ASIST 2014, November 1-4, 2014, Seattle, WA, USA. Copyright is retained by the author(s). Towards Building a Collection of Web Archiving Research Articles were forced to look for other alternatives of building a research corpus such as employing machine learning techniques for document classification. Crawling the Web for relevant articles to assemble a dataset seemed like a potentially effective strategy. In the literature, there have been several studies on focused web crawling, a strategy that collects only Web pages that satisfy some specific property, e.g., they belong to a particular topic. Focused crawling first proposed by De Bra et al. is a rich area of research on the Web (Bra et al., 1994), (Junghoo Cho et al., 1998). Chakrabarti et al. (1999) present a discussion on the main components involved in building a focused crawler. Bergmark, Lagoze, and Sbityakov (2002) discuss some of the crawling technologies for building document collections as well as ways to make the crawler highly effective. Batsakis, Petrakis, and Milios (2009) propose state-of-the-art crawlers strategies that use the content of Web pages as well as the link information in order to estimate the relevance of Web pages tied to specific topics. Other works on focused crawling include (Li, Wang, and Du, 2013); (Yang, Kang, and Choi, 2005). Wu et al. described the evolution of a crawling strategy for CiteSeer, which is an academic document search engine (Wu et al., 2012a). CiteSeer actively crawls the Web for academic and research documents primarily in Computer and Information Sciences. The authors experimented with using a whitelist (a list of only certain domains that should be crawled) to improve the crawling efficiency of the CiteSeer crawler. They found that crawling the whitelist significantly increased the crawl precision by reducing a large amount of irrelevant requests and downloads. In another study, Wu et al. developed a middleware, the Crawl Document Importer (CDI), which selectively imports documents and their associated metadata to the CiteSeer crawl repository and database. This middleware provides a universal interface to the crawl database and is designed to support input from multiple open source crawlers and archival formats (Wu et al., 2012b). Caragea et al. (2014) presented a record linkage approach to building a scholarly big dataset, derived from the CiteSeer dataset, which is substantially cleaner than the entire set. More precisely, the authors’ approach was to integrate information from an external data source to remove noise in CiteSeer that results due to automated techniques used for metadata extraction from Web crawled documents. In contrast to the above works, we make use of information retrieval and machine learning techniques such as focused crawling and text classification to construct a scholarly dataset of Web Archiving research articles. The dataset is available for download to the research community and will particularly be useful to researchers interested in Web Archiving and newcomers to this field. 1 1 http://digital.library. unt.edu/ark:/67531/metadc330569/ BUILDING A COLLECTION OF WEB ARCHIVING RESEARCH ARTICLES In this section, we present our crawling strategy for building a collection of research articles gathered from the Web, that are related to the topic of Web Archiving. We describe the main steps of the crawling process: 1. Compile an initial set of documents related to Web Archiving, which represents the seed set. 2. Similarly, compile a set of documents, which are not related to Web Archiving. 3. Train a classifier to accurately discriminate between Web Archiving versus non-Web Archiving documents. 4. Extract the authors from the articles related to Web Archiving in our seed set and perform a crawling using these authors’ names as well as all their found coauthors as queries that are input to a generic search engine and download other research articles that these authors have published previously. 5. Use the trained classifier in Step 3 to automatically identify the documents related to Web Archiving. We present further details of these steps in what follows. We start with an initial corpus (our seed set) composed of 124 documents about Web Archiving that we extracted from a comprehensive bibliography on the subject (Reyes Ayala, 2013). This bibliography was put together over the course of several months using a variety of methods, such as querying search engines and downloading the publications of prominent authors in the field. We also gathered a separate corpus of randomly chosen documents from many different disciplines. At the end of this process, we had 124 articles about Web Archiving and 206 randomly chosen articles, for a total of 330 articles. We used a Python library to extract their titles and abstracts. Some documents did not have abstracts, and in such cases, we instead used the document’s first 300 words. The motivation for extracting only the title and abstract from a document was that in many cases, documents on the Web are not available as full text, but only as title and abstract. We refer to this set of documents as Original. During the Step