Scraping the ACM Digital Library
Donna Bergmark (bergmark@cs.cornell.edu)
Cornell Digital Library Research Group
Paradee Phempoonpanich (pp73@cornell.edu) and Shumin Zhao (sz37@cornell.edu)
Cornell Computer Science Dept.
4128 Upson Hall, Ithaca NY 14853
Abstract
As part of a larger project to automatically reference link the online scholarly literature, an attempt toanalyze PDF documents was undertaken. The ACM
Digital Library was used as the corpus for these experiments. With the current PDF and HTML analysis tools, roughly 80% accuracy was obtained in theautomatic extraction of reference linking information.

1

Introduction

The Association for Computing Machinery (ACM)
has put a large portion of their literature online. This
makes reference linking from and to the ACM literature possible. Reference linking [14] allows one to go
directly from the “active paper” (which one is reading) to a referenced paper, assuming the referenced
paper is also online.
Often, authors of newer “born digital” documents
insert links to online references. However, this does
not always happen, and it was not done with older
literature. ACM has reference linked its new digital
library to other items in digital library[15], but thisstill leaves many unlinked references.
For more aggressive reference linking[8], it is necessary to “scrape” the active document to extract thereferences and their bibliographic data; the bibliographic data is then used to look up one or more online locations for the referenced work; the references
are turned into “actionable links” overlaid on the active document; finally if the user selects the link, acopy of the referenced work is retrieved and shown tothe user.
This paper reports on experiences doing the firstof the four tasks: scraping ACM papers for reference
data. Other projects (CrossRef[13], OpCit[10, 1],
ResearchIndex[5, 12, 6], SCI[2, 9]) provide alternative
approaches to this task. In fact, much work of interest is going on in reference linking today [7] becauseof the huge growth in the amount of online scholarly
literature. The principal need is for automatic techniques such as those used in this project.

2

Methodology

ACM was kind enough to provide us with a copy of
their digital library to see what we could do in theway of scraping it for reference data. We already havesoftware to do this for HTML documents [4], but papers from ACM journals, proceedings, and magazines
are in PDF format.
Rather than manipulating the PDF directly, we decided to take the approach of converting the PDF
to HTML and analyze it using our existing software.
This paper reports on our experiences so far with thisendeavor. We describe the results for some of the papers we looked at and point out where improvements
are needed.
It should be noted up front that the produced
HTML need not present a viewable version of the
PDF paper; it needs only to have the right tags andcontent to allow it to be mined for reference linking
information.
The first tool we considered was Access from Adobe
Systems Inc. This acrobat plugin exports PDF files
as HTML for use by sight-impaired persons wishingto use text-to-audio devices. However, we quickly
ran into two unsolved problems: (1) Adobe explicitly forbids the use of their Access server merely forconverting PDF to HTML, and (2) no command line
interface to Access was known, nor was the source
available.
We then discovered the PJ package, a very goodbut mostly undocumented PDF parser from Etymon
Systems, Inc (<http://www.etymon.com>). We used
this as our workhorse and wrote our code in Java
(Version 1.4 beta of Java 2 Standard Edition from
Sun). We used Java because that is the language inwhich PJ is implemented, and we used version 1.4
because of its support for regular expressions.1

2.1

Deflation

Two Java programs are used to pre-condition the
PDF source. The first one “deflates” the PDF source,
and is the uncompression routine distributed with the
PJ package. This does not work on all papers in the
ACM digital library because a variety of compression
schemes were used. However, it appears that none of
the ACM papers is encrypted, which is a good thing.
We have no hope of processing encrypted material.
The program UncompressPdf takes as input a
PDF file amd deflates it, that is, deflates any flatecompressed stream object in the PDF input.

2.2

Normalization

If the decompression is successful, the document isnext run through a PDF normalizer. This program,
PdfTrim, straightens out some of the more complex
PDF syntax, and prepares the document for further
analysis. The main thing it does is to separate multiple PDF operators, along with their arguments, onto
separate lines.

2.3

Parsing

The conditioned PDF document is then sent throughour parser, PdfParse. The parser is a complex program first developed by Phempoonpanich and thengreatly extended by Zhao. It makes two passes overthe PDF object tree. During the first pass, the firstpage is parsed, to extract the title and authors. The
title is the line drawn in the largest font. The authors
are assumed to come next.
The second pass traverses the page tree to extract
the content of the document, which is written out asan HTML file. The title is enclosed within <h1> tags,
and the authors are denoted by <center> markup.
Each section heading is written out as an <h3> element. The remaining content is written out as text.
The reference section is split up into one <p> element
per reference string. Embedded images are ignored,as they are not relevant to reference data analysis; weare interested only in textual reference data.
In addition, each text fragment that begins with [,(, or { is preceded by an HTML comment giving thepage number and object number of that text fragment. This is to facilitate insertion of locators backinto the original PDF file, to make links “active”.
1 Xpdf is a similar package, written in C and C++.

(That part of this project remains to be implemented,
but in principle it should work.)
The resulting HTML file is handed to our Reference
Linking API [4], which then attempts to extract reference linking information. The remainder of this paper
first outlines some specific techniques used in our reference data extraction software and then presents ourdata on how well the approach worked with ACM’s
source.

3

Title and Author Section

How well the reference data extraction software canwork for the HTML generated from PDF depends on
how well the PDF parser worked. It is important for
the title and author section of a paper to be extracted,
as this information is required to identify the work
citing each one of the references.
A PDF string must satisfy two conditions to berecognized as the title of the paper being analyzed.
First, it needs to have the largest font size on the first
page. Second, if there are more than one such strings(separated by text in other font sizes), only the firstone is taken as the title. This method works wellfor most PDF files. Emitting this string as an <h1>
HTML element is sufficient for the reference data extraction software to pick it up.
After the title section is found, the PDF text rightafter the title is taken as the author section. Finding
the author section is thus dependent on the success offinding the title. The author section is emitted as aseries of <center> HTML elements, one per line. Disambiguating the contents of the author section is left
up to the reference linking data extraction software.
Determining where the author section ends depends on finding a section title. This is a string on
one line, with a rather large font, and often contains
the word “Introduction”. Also, the string “Abstract”
in any font size ends the author section.
As part of determining where the author section
ends, an attempt is made to determine the format ofsection titles. This is important in order to determine
where the reference section starts. The reference section (1) begins with a section title, and (2) has a title
such as “References”, “Bibliography”, etc. The PDF
parser finds too many strings that look like section
titles, but this is all right so long as one of them isthe title for the reference section.

4Contexts and References

Although many reference linking applications need
only deal with the reference section of a paper, it
is quite useful to be able to collect reference contexts
as well. Contexts are sentences that contain one ormore references. For example, a sentence like
“Other authors have presented alternative proofs of nonergodicity (Kaplan [11],Rosenkrantz and Towsley [12]).”is a context that contains two references (also calledreference anchors). Useful information associatedwith each reference includes the following:• Data regarding what work this reference represents (title, authors, year of publication, etc.)• Locations of online copies of this work
• Contexts in which this reference appeared

4.1

The Range Problem

Sometimes it happens that references such as“[1]-[4]” are encountered. The context containing
this reference anchor belongs to references 1, 2, 3, and
4. In this case our existing reference linking software
simply expands the range into the four separate references, and the context is associated with each oneof them.

4.2

Reference Formats

The reference format is selected from one ofsix, depending on which format is first encountered.
square brackets around numerals
parsesreferences
like
[11].
brackets around commaed names and years parses
references such as [Bradford and Whaley, 1999; Ford,1998]. The remaining four formats deal with othervariations [3]. For finding the contexts, it is best
to nail down the reference format as accurately as
possible. Once set, it is not changed for the rest of
the paper.
The way in which reference format is determined is
as follows:
1. Starting at the top of the document (after title
and authors), locate the next sentence (chunk oftext from one full stop to the next).
2. Run six format parsers conceptually “in parallel”
3. If one succeeds in finding a reference, continue
using that reference anchor parser for the rest of
the paper, one sentence at a time.

4.3

Matching Reference Literals to
Their Contexts

A reference literal is the actual reference string, withall of its details, as contained (typically) in the reference section at the end of the paper. More details
can be found in [3], but briefly a reference literal ismatched to its contexts as follows:
1. First, the tag is stripped from the literal.
2. A lookup is performed to see if this tag, e.g. [11]in canonicalized form, is contained in any of thecontexts. If so, those contexts are added to thisreference’s context list.
3. Otherwise, if the tag matches none of thecontext anchors and the format is something likeparentheses around acronyms or brackets around commaed names and years,
an approximate match is made between thecontext’s reference anchors and the reference
literal. For example, if a context contains thereference (Braley and Hill, 1989), it would
match up with a reference literal like Braley,
A. and Hill, J. ... 1989, because every
significant word in the anchor appears in thereference literal.
Of course, the success with which the references areparsed is totally dependent on the PDF parser having located the reference section in the first place, andthen having successfully split each reference into its
own paragraph (<p> HTML element) in the second
place. The PDF page markup specifies positions ofvarious textual elements. We assume that the firstline of each reference literal is aligned to the left.
Other lines of the reference might be indented to theright by a little amount of space. Properties such as
intra- and inter- line spacing are helpful in separating the reference literals from each other. The main
remaining problem is handling references that occur
in two columns. The second column is usually intermixed with the first column, since they are renderedon the page “at the same time”. This results in somereference literals in the HTML file to be combined.

4.4

Bibliographic Data

References’ bibliographic data is extracted from reference literals by the Deciter tool written at U. ofSouthampton [11]. We extract the following information:• title of the referenced work
• each author of the referenced work• year of publication• any urls that are present
The first author’s last name, date, and part of thetitle are used to identify the work. It is possible also
for the Deciter to extract the journal name, pages,
etc. These additional data might be helpful in locating online copies of references.

4.5

Accuracy Metrics

As an objective measure of how well our algorithms
work, we use metrics called Item Accuracy and Reference Accuracy. They are both computed as percentages: elements correctly extracted divided by total number of elements. Item accuracy is 100% if
the following are correctly determined: title, eachauthor, date of publication, each context, and eachreference. The number of correct references is basedon number of reference literals in the paper
× average reference accuracy for this paper.
The average reference accuracy is the number of elements extracted correctly from the reference section
divided by the total number of elements therein.

5

What the Reference
Should Be[1]-[4]
[8]
[10]
[11]
[12]
[13]
[6]
[13]
[18]
[18]
[19], [20], [21]
[20]
[5]
[5]
[22]
[5]
[16]
[5]
[7]

What the Reference was
Scanned As[l]-[4]
[s]
[lo]
[ll]
121
113
161
1131
[H]
[IS]
WI, POI, Pll
[ZO]
(51
[S]
$221
(51
1161
(51)
171

Figure 1: Table of mis-scanned references. Each reference (as it happens) is in a different context fromeach other. Thus 19 of the 29 contexts had an incorrect anchor.

Results – an Example Paper

First we present the results of scraping one paper
in order to give a concrete feeling for how hard thisproject is.
A paper was randomly selected from the ACM Digital Library. After decompression and preconditioning, the paper was converted to HTML as describedin Section 2. The paper was then subjected to analysis, with the goal of extracting all useful reference
linking data from it.
The examined paper contained 29 contexts and 22
references. It appears that flate-compressed streams
deal with bit-mapped images, and deflation attempts
to convert bit-mapped characters into their ASCII
equivalent. This translation did not always go right.
Figure 1 lists the improperly rendered anchors. The
first column has the reference anchors as they appeared in the published paper; the second column
shows the characters into which they were converted
during deflation. Errors are frequent, with some letters being mere approximations to the characters intended to be “drawn” on the printed page.
Figure 2 gives some of the consequences of the reference anchors being scanned incorrectly. We won’t
show the complete table, but it seems clear that thepoor translation of square brackets and digits withinthem hampered the detection of contexts.

Particularly devastating was that the very
first reference anchor in the paper was misspelled (the letter “ell” rather than the digit“1”). Thus despite the references being in theform square brackets around numerals, themuch looser format brackets around commaednames and years was assumed instead. This format does not even require the references to be numeric.
This led to “s”, “ll”, etc. being treated as reference anchors. This led to some references acquiring
many bogus contexts. In other words, since the anchor never matches the tag on the reference literal(which in fact is numeric), the fallback of approximate matching is used. In the case of the first anchor,
for example, since [l] does not match the tag “1.”(or any of the other tags for that matter), it causes
the reference to be searched for the letter “l”, and if
found, then this is added to the reference as one ofits contexts.2
Fixing the brackets and digits by hand resulted inall 29 contexts being found and matched up with their
2 This suggests that some programming changes should be
made. First, reference anchors consisting of only 2 or fewer
lower case letters should not be considered as anchors. Secondly, when searching for anchor words within the reference
string, it should match a whole word, not just part of one.

Context
Number1

Reference
Anchors[l]-[4]3171

4[s],[9]

5

6

7

[lo]

[ll],
Rosenkrantz
and Towsley(121).

113)

Consequences
References 1, 2, and3 will not match up
with this context.
This should havebeen [7].
This
sentence is not recognized as a context, letalone be matched up
with reference literal
7.
Should have been[8], [9]. Picked up
as a context but only
for reference 9., notfor 8..
Should have been[10]. Was picked up
as a context because
we are not requiring
numerics, but it will
not match reference
literal 10..
Should have been[11], Rosenkrantz
and Towsley [12]).
Was picked up as acontext (because of[ll]) but no reference literal will matchwith these non-digits.
Should be [13]; was
not recognized as acontext.

Figure 2: Analyzing contexts from PDF source which
has been uncompressed and trimmed – typical problems with finding references in uncompressed text.
corresponding reference literals. This improved theItem Accuracy (because all the contexts were found)and the Reference Accuracy (because each reference’s contexts were found).
Figure 3 shows the actual results of analyzing
the 22 references in this paper, from the precision
point of view. Precision is how many of the recognized elements were correct. Recall is the percentage of reference data elements that are determinedcorrectly. Figure 4 shows recall. Of the 22 references, 20 were detected and analyzed (the other two,
which appeared in the second column of the paper,
were accidentally merged with the fifth reference).

Actually the results weren’t all that bad (77.73% Average Reference Accuracy and 88.89% average Item
Accuracy). The measurements were lower than they
could have been primarily because of the referencecontext problem caused by poor scan translation.
The bibliographic data was extracted quite well.
Occasionally titles and journal names were scrambled(see Table 1). Categories 3 and 4 of the erroneousjournal names are probably impossible to use as a
URL lookup mechanism. An example of Class 3 error
is:IEEE !I!rans. Inform. Theory...
With some further editing, Class 2 journal names
probably could be accurately detected and used. A
Class 2 example is:Proc. Fifth Data Comm. Symp.,
Snowbird, Utah, pp
Here, only the final “pp.” has to be trimmed off. So,
13 of the 20 references, or 65%, could probably beconverted into URLs of online copies, if any exist.
Table 1: Number of Correctly Extracted Journal
Names
Class 1
Class 2
Class 3
Class 4

6

No journal name in reference
Name extracted, extra information
Name extracted, with scan errors
Name failed to be extracted

0
13
6
1

Collection Level Results

To get a general feeling for how well ACM papers
could be processed using our methodology, we went
to the journals and proceedings and selected at random one paper from each of the different collections,of which there were 60. It became immediately
apparent that PDF files are not all encoded in thesame way. Here are the results:
ResultPapers which did not deflate
Deflated papers that broke PJ
Reference Section not found
References not individualized
Produced analyzable HTML output

Total

Number
38
4
4
6
8
60

Papers which did not deflate actually decreaseda bit in size when run through the decompression

Recall of Data Retrieved

Precision of Recognized Data
Reference
(1)

Bib
Data
(2)

Real
Context(3)

Bogus
Context(4)

Precision(2+3)/(2+3+4)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

4
3
4
5
3
3
3
3.5
4
5
3
4
5
3
3
3
3
3
3
3
0
0
70.5

1
1
1
1
7
2
1
1
1
0
0
0
0
0
1
1
2
0
1
1
0
0
22

11
8
7
0
0
0
0
13
0
13
12
7
14
15
0
0
0
7
0
0
0
0
107

.45
.50
.71
1.00
1.00
1.00
1.00
.35
1.00
.38
.25
.57
.36
.20
1.00
1.00
1.00
.43
1.00
1.00
None
found
.46

(out of 90)(out of 40)

TotalsFigure 3: The accuracy with which 22 reference literals were parsed. Bib Data refers to authors, title andyear. Real contexts are contexts that indeed contained this reference. Bogus contexts are contexts
that did not really belong to this reference. If all thecorrect BibData and Contexts were found, we would
have had 119 elements in all. With the bogus contexts, we add 107 more elements. So, precision for
full recall would have been 119/226 = 0.53, which
is pretty disappointing. The actual scraping had aprecision of 92.5/199.5 = 0.46.

program. Most likely these were compressed using analgorithm other than “flate”. None of these papers
could be processed further.
The papers that broke PJ got a class cast exception involving a PjReference object. We will look
more into this at some future date. The remaining 18
papers, or 30% of the starting set, produced HTMLoutput. This is not an encouraging result. Although
this is a statistically small sample, it does seem toindicate that with current tools we can process only
a quarter of the ACM digital library.

reference

(1)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
Totalsnumberofelements
correcttotalelements

recall

(2)
5
4
5
6
10
5
4
4.5
5
5
3
4
5
3
4
4
5
3
4
4
0
0
92.5

(3)
5
4
5
6
14
6
5
5
5
6
4
5
7
3
4
5
5
5
5
6
4
5
119

(2)/(3)
1.00
1.00
1.00
1.00
.71
.83
.80
.90
1.00
.83
.75
.80
.71
1.00
1.00
.80
1.00
.60
.80
.67
.00
.00
0.78

Why recall isless than 1.00

Title has
B&table(should be
Bistable)

References
not foundFigure 4: Unless otherwise noted, the less than perfect recall is due to contexts not found because thereference anchor was misspelled (see Figure 1). Column 2 includes both Bib Data and Real Context (seeFigure 3 headings). Since the overall recall, or average reference accuracy, is 92.5/119 = 0.78, we say on
average that 17 of the 22 references parsed correctly
when computing Item Accuracy.

7 Other Converters
Although we wrote our own pdf-to-html converter,
there are other converters available on the Internet.
Many of these simply convert the PDF source into
images which are then served up as HTML pages.
These are not useful for our purposes.
The pdftohtml converter from the Greenstone
digital library project (<http://nzdl.org>) did
work for papers that did not deflate, but the HTMLthat was produced was rather poor. This code also
produces a .ppm file for each page, but the images
were upside down and backwards.

The pdftohtml converter from Stuttgart, basedon Xpdf, is a C program downloadable from<http://www.ra.informatik.uni-stuttgart.
de/~gosho/pdftohtml>. It is much faster thanGreenstone’s, perhaps because it does not generate.ppm files.
Neither of the pdftohtml converters avoided thescan errors gotten from the deflation algorithm. Also
both converters missed the second column of references (21 and 22), just as our translator did. They
were either appended to the 5th reference, or interleaved with references 5 and 6.

8

Conclusion

Acknowledgements

This work was funded in part by NSF grantIIS-9907892, ”Integrating and Navigating Eprint
Archives through Citation-Linking”. This project ispart of the International Digital Library Initiative.

References[1] The Open Citation Project.
eprints.org>.

[5] K. Bollacker, S. Lawrence, and C. Giles. CiteSeer: an autonomous web agent for automatic
retrieval and identification of interesting publications. In Proceedings of the 3rd ACM Conference
on Digital Libraries, pages 116–123, 1998.
[6] K. Bollacker, S. Lawrence, and C. Giles. Discovering relevant scientific literature on the web.
IEEE Intelligent Systems, 15(2):42–47, 2000.
[7] C. L. Borgman and J. Furner. Scholarly Communication and Bibliometrics. 2001.

Automatic techniques applied to the ACM digital library will only go so far. Analyzing PDF sourceis really hard. We like the PJ package very much.
We must locate more PDF-decompression tools, ones
which will handle the remaining formats, such as
LZW. The prevalence of scan errors indicates that
ACM was wise to use a proprietary, fuzzy matchingalgorithm to match reference literals against existing
bibliographic data.

9//link.springer-ny.com/link/service/series/0558/papers/2163/21630115.pdf>.

<http:opcit.

[2] H. Atkins. The ISI Web of Science: Links
and electronic journals: How links work today in the Web of Science.
D-Lib Magazine: The Magazine of Digital Library Research, 5(9), 9. <http://www.dlib.org/dlib/
september99/atkins/09atkins.html>.
[3] D. Bergmark. Automatic extraction of reference linking information from online documents.
Technical Report TR 2000-1821, Cornell Computer Science Department, Nov. 2000.
[4] D. Bergmark and C. Lagoze. An architecture
for automatic reference linking.
In ECDL,
September 2001, Sept. 2001. <http://www.cs.
cornell.edu/bergmark/www10.pdf>,<http:[8] P. Caplan and W. Arms. Reference linking forjournal articles. D-Lib Magazine: The Magazineof Digital Library Research, 5(7/8), July/August
1999. <http://www.dlib.org/dlib/july99/
caplan/07caplan.html>.
[9] B. Cronin. Bibliometrics and beyond: Somethoughts on webometrics and influmetrics, 2000.
[10] S. Harnad and L. Carr. Integrating, navigating
and analyzing open eprint archives throughopen citation linking (the OpCit project).
Current Science Online, 79(5), Sept. 2000.
<http://www.cogsci.soton.ac.uk/~harnad/
Papers/Harnad/harnad00.citation.htm>.
[11] S. Hitchcock, L. Carr, S. Harris, J. Hey, andW. Hall.
Citation linking: Improving access to online journals. In R. B. Allen and
E. Rasmussen, editors, Second ACM International Conference on Digital Libraries, pages
115–122, Philadelphia,PA, July 1997.
[12] S. Lawrence, C. L. Giles, and K. Bollacker.
Digital libraries and autonomous citation indexing. IEEE Computer, 32(6):67–71, 1999.
<http://www.researchindex.com.
[13] I. Pila. CrossRef: The central source for reference linking. <http://www.crossref.org/>.
[14] B. Schatz. Information analysis on the net:
The interspace of the 21st century. In Proc.
1997 (34th) Annual Clinic on Library Applications of Data Processing, pages 110–126. Graduate School of Library & Information Science, U.
of Illinois, Oct. 1995.
[15] J. White. ACM opens portal to computing literature. Communications of the ACM, 44(7):14–16,
July 2001.