Natural Language Processing
for Intelligent Accessto Scientific Information
Francesco Ronzano and Horacio Saggion
Natural Language Processing Group (TALN)
Universitat Pompeu Fabra, Barcelona, Spain
Tutorial @ COLING 2016
11th December 2016

TALN @ UPF

Natural Language Processing Group• Francesco Ronzano, Senior Post-doc• Horacio Saggion, Research Professor

What it’s all about
Scientific literature is growing at an unprecedented rate
Automated approaches to extract, enrich, aggregate and summarize
information from scientific publications are essential to enable any
careful and comprehensive assessment of scientific literature
Natural Language Processing and Text Mining play a fundamental rulesince they are key technologies to analyze scientific publications
This tutorial provides an overview of the core content analysis challenges
and opportunities of Scientific Literature Mining
showing how we can characterize and take advantage of implicit and explicit
traits of scientific publications to better organize and provide access toscientific literature
http://taln.upf.edu/pages/coling2016tutorial/

Outline• SCIENTIFIC INFORMATION OVERLOAD
How much scientific literature is there out there? How can we search for and access to scientificinformation?

• DOCUMENT STRUCTURE ANALYSISHow can we extract textual contents from PDF papers? Which tools are there? How can we mineand link data form headers and bibliography?

• SCIENTIFIC DISCOURSE CHARACTERIZATION
How can we spot the contributions of a piece of research? Where do the authors present their
future work?

• CITATION ANALYSISHow can citations improve our access to scientific information? Are all citations equals? How can
we suggest citations?

• SCIENTIFIC DOCUMENT SUMMARIZATION
How can we take advantage of peculiar traits of scientific documents to generate bettersummaries?

• CHALLENGES, DATASETS AND ARCHITECTURES
Which datasets are available for scientific text mining? Which tasks have been proposed?

• DR. INVENTOR TEXT MINING FRAMEWORK
Whic scientific data analyses are supported? How can the framework be used in practice?

• GLOBAL CONCLUSIONS AND DISCUSSIONSCIENTIFIC INFORMATIONOVERLOAD

Outline• How much scientific literature is there?
• How ‘open’ is scientific literature?
• Who publish scientific articles?
• How researchers search and read publications?
• Academic social networks• Social Media in academic communication• Text mining opportunities and challenges

Scientific literature overload

Scientific Literature Overload

How much scientific literature is there?
28,100 active peer-reviewed scholarly journals in English + 6,450 non English journals
All of them are publishing 2,5 million papers a year
(more than one new article every 13 seconds)
Looking inside some citation database…
90 million articles

55 million articles
22,000 journals from about 5,000 publishers(in 2013 about 2 million of new articles)

80 million DOI
about 58 million refer to journal articles from 36,ooo journals

between 100 and 160 million docs
(journal articles, books and grey literature, etc.)

11,365 journals
26 million publications

from more than 2,600 publishers

STM Report 2015 / citation database queryScientific Literature Overload

How much scientific literature is there?
28,100 active peer-reviewed scholarly journals in English + 6,450 non English journals
All of them are publishing 2,5 million paper a year
(more than one new article every 13 seconds)

The number of paper publishedexperimented an exponential growthduring the last decades
The growth in number of papers is
proportional to the growth in numberof scientific researchers all over theworld (now between 7 and 9 million,
only 20% repeated authors)

Global scientific publication growth (articles by year)

STM Report 2015
Bornmann & Mutz (2015). Growth rates of modern science.

Scientific Literature Overload

How much scientific literature is there?
28,100 active peer-reviewed scholarly journals in English + 6,450 non English journals
All of them are publishing 2,5 million paper a year
(more than one new article every 13 seconds)

PubMed: from 1980 to 2003 the average
growth is 2,9% per year, while from 2003
to 2013 it raised up to 6,7% per year

Web of Science: in 2000, 8,684 journals.
In 2005, 9,467 journals, an increase of
9%. In 2010 11,519 journals, a further
increase of 22%.
PubMed growth (articles by year)
STM Report 2015 / PubMed

Scientific Literature Overload

How much scientific literature is there?
…by country• The growth of scientific throughput of China: from 4,5% in 2002 to 17% currently• The citation count is dominated by USA (36%) with China in 11th place (6%), because ofrecent increase in scientific production
Elsevier, 2013

Scientific Literature Overload

Who publish scientific articles?
There are about 10,000 journal publishers globally
64% commercial publishers (including publishing for societies),
30% society publishers, 4% university publishers, 2% other publishers
Revenue:• $10 billion in 2013 ($8 billion in 2008)• 55% from USA, 28% Europe/Middle east, 14% Asia/Pacific, 4% Others
Employers:• 110,000 people globally directly employed, 40% in EU (+ 20-30,000 people indirectly)
The long tail of publishing:• the top 100 journal publisher publish 67% of all journals• top 5 publishers are Springer, Elsevier, Wiley, Taylor&Francis (35% of all journals)• many publishers with 1 or 2 journals
STM Report 2015

Scientific Literature Overload

How ‘open’ is scientific literature?
The Open Access publishing model is consistently growing

22,000 peer-reviewed journals,
13% open-access

8,200 peer-reviewed journals,
9% open-access

3,257 peer-reviewed journals,
17% open-access

2,500 peer-reviewed English journals,
13% open-access

9,237 journals, 2,330,000 articles
Archambault et al. (2014). Proportion of open access papers published in peer-reviewed journals at the European and world levels—1996–2013.
Lewis, D. W. (2012). The inevitability of open access.

Scientific Literature Overload

How ‘open’ is scientific literature?
The Open Access publishing model is consistently growing

Open access growth

Before 2021, globally more thanhalf of the papers will be
published as Open Access

PlosONE, one of the biggest Open
Access journals:• more than 34,000 articles per
year (94 new articles per day)
• 2015 IF: 3.057

Archambault et al. (2014). Proportion of open access papers published in peer-reviewed journals at the European and world levels—1996–2013.
Lewis, D. W. (2012). The inevitability of open access.

Scientific Literature Overload

How researchers search and read publications?
More reading, less time dedicated to each paper
Average number of articles read by year: about 270(with several variations, depending on discipline - more in medicine
and science, fewer in humanities and social sciences,increased from 188 in mid-1990s)

Reading times of an article: about 30 minutes(went down from 45-50 minutes in the mid-1990s)

Clear growing importance of online literature search engines• About 60% of article referrals of major publishers
comes form one search engine, Google Scholar• Publisher sites are accessed at article level(reduced importance of publisher site browsing)
STM Report 2015
Tenopir (2007) What does usage data tell us about our users?

Scientific Literature Overload

How researchers search and read publications?
Online search and access to scientific literature“The forced browsing of print archives may have stretched scientists and scholars toanchor findings deeply into past and present scholarship.
Searching online is more efficient and following hyperlinks quickly puts researchers intouch with prevailing opinion, but this may accelerate consensus and narrow therange of findings and ideas built upon.”Pros:• more comprehensive searches• more information to more extended audienceCons:• the articles cited tend to be more recent• there are fewer citations• most citations are to fewer journals and articles• weakening ability to explore scientific literature laterally finding in otherstudies and disciplines information potentially relevant to our current research
Source: STM Report 2015
Source: Evans, J. A. (2008). Electronic publication and the narrowing of science and scholarship. science, 321(5887)

Scientific Literature Overload

How researchers search and read publications?
…and the coverage of search engines
About 24% of documents are freely
available online with several differences
across fields:

Source: Khabsa & Giles (2014). The number of scholarly documents on the public web. PloS one

Scientific Literature Overload

How researchers search and read publications?
How publishers are dealing with
online scientific literature search and access?
• Improving and enriching their online offer and user experience (new tools:
analytics, expertise search, etc.)• Switching to exclusive online publishing:• currently all STM (International Association of Scientific, Technical, and Medical
Publishers) journals can be accessible on-line – in 2003: 83%, in 2008: 96%• the number of established research journals dropping their print editions looks
likely to accelerate over the coming few years• Offering enhanced article-level access and easing the integration of theircontents into third-party platforms by:• providing enriched and linked versions of publications• exposing Open APIs

Source: STM Report 2015

Scientific Literature Overload

Academic social networks

more than 11 million users

more than 44 million registered users(150.000 members in August 2008, 700.000 in December 2010, one
million by May 2011, and 2 million in September 2012)(16,205,767 papers added and 1,953,015 research interests
specified) Academia.edu attracts over 36 million unique visitors amonthabout 4 million users(part of Elsevier from 2013) – 470 million documents

Why academic social networks are used?
• connecting with other researchers• make own research more visible and follow the updates of other researchers• like an ‘online business card’
Sources: STM Report 2015 / factsheets of academic networks

Scientific Literature Overload

Social Media in academic communication
Social Media are experimenting an increasing adoption
as complementary channel to promote and discuss scientific publications and events

But there is still a long way to go to achieve broad Social Media adoption…
• growing impact but still limited when compared to conventional channels (in several surveys thepercentage of researchers that actively use Social Media ranges from 3% to 32%)

• used mainly as complementary channels to make more visible the research than as means
to interact, discuss with other users or to keep updated with new findings• proliferation of too many potentially useful platforms to consider

Main obstacles to the use of Social Media:• lack of clearly compelling benefits with respect to the time needed to publishing material
and manage interactions• quality and trust issues
A wider adoption of altmetrics can foster the use of Social Media in scholarly communication
Sugimoto et al. (2016). Scholarly use of social media and altmetrics: a review of the literature.

Scientific Literature Overload

Text mining opportunities and challenges
Natural Language Processing and Text Mining are starting to emerge as
key technologies able to help scientists to deal with scientific literature overload
BACKGROUNDAPPROACH
BACKGROUND
FUTURE WORK

Citation network / data linking
Entities and relations
evaluated on
implemented by

Algorithm

Tool

DatasetThe analysis of the structure
and the semantics of fulltextual contents of scientificpublications enables a wide
range of new approaches toeasily retrieve, compare andsummarize scientific literature

Scientific discourse
Content relevance

SUMMARY

Scientific Literature Overload

Text mining opportunities and challenges
Natural Language Processing and Text Mining are starting to emerge as
key technologies able to help scientists to deal with scientific literature overload

Opportunities (use cases):• search for information scoped to specific sectionsof the discursive structure• validating how scientific contents are exposed• scientific-discourse driven summaries

BACKGROUNDAPPROACH
BACKGROUND
FUTURE WORKSUMMARY• automated generation of state-of-the-art reviews• support fast scientific literature screening, thus
reducing the efforts needed for literature reviews

Challenges:• huge, evolving amounts of data
• data collection, extraction, normalization• error rate of automated approaches• high variety of knowledge domains• diversified information needs• data often protected by copyrights

Scientific Literature Overload

Text mining opportunities and challenges
Natural Language Processing and Text Mining are starting to emerge as
key technologies able to help scientists to deal with scientific literature overload

Access to full texts of publications: copyright issues• Major publishers have defined their “Text and data mining policy” (limitation to
text for restricted access, subject to license restriction for OA)• Questions and answers on the modernization of EU copyright rules for the digitalage - European Union, 14 September 2016:
http://europa.eu/rapid/press-release_MEMO-16-3011_en.htm“The Commission proposes a new mandatory exception, which would require all Member States topermit research organizations acting in the public interest – such as universities and research institutes –to carry out text and data mining of copyright protected content to which they have lawful access, forexample scientific publications they have subscribed to, without the need of a prior authorization. The
exception will not apply to commercial companies.”

Scientific Literature Overload

Text mining opportunities and challenges
Natural Language Processing and Text Mining are starting to emerge as
key technologies able to help scientists to deal with scientific literature overload

Access to full texts of publications: copyright issues

http://tdmsupport.crossref.org/

Include in CrossRef metadata that describe each bibliographic entry
a standard set of license information fields that clearly specify
the limitations and the way to access and mine the full textual contents of papers• 12 publishers involved in the definition of license
metadata (Elsevier, Springer, Wiley, etc.)• common mechanism for providing automated text anddata mining tools with direct links to full text on the
publisher’s site

Scientific Literature Overload

Text mining opportunities and challenges
Natural Language Processing and Text Mining are starting to emerge as
key technologies able to help scientists to deal with scientific literature overload

This tutorial provides an overview of the core content analysis challenges
and opportunities of Scientific Literature Mining
showing how we can characterize and take advantage of implicit andexplicit traits of scientific publications to better organize and provide
access to scientific literature

Document
Structure Analysis

Scientific Discourse
Characterization

Citation Analysis

Scientific Document
Summarization

DOCUMENT STRUCTURE
ANALYSIS

Outline• Document formats: dealing with PDF• Genral-purpose PDF-to-text tools• PDF-to-text for scientific publications• Comparing PDF-to-text for scientificpublications• Bibliographic entry parsing• Annotated datasets• Conclusions

Document Structure Analysis

Document formats: dealing with PDF
Despite the many XML dialects and scientific publishing technologies proposed during the last few years,

PDF still constitutes the most widespread distribution formatof scientific publications(80% of scientific literature is accessed as PDF documents)

Why PDF are so popular?
• mature technologies (1993, Adobe)• preserved format across platform with several tools to visualize andannotate it• easy to store and organize for off-line reading• self contained files: capture the article in a stable, read-only form
• can include high-resolution images• print-friendly• can be reasonably protected without the use of dedicated servers

Document Structure Analysis

Document formats: dealing with PDF
Despite the many XML dialects and scientific publishing technologies proposed during the last few years,

PDF still constitutes the most widespread distribution formatof scientific publications(80% of scientific literature is accessed as PDF documents)

Some drawbacks• manipulation is dependent on a commercial software (even if someopen-source alternative is available)• impossible to include multimedia material / low level of interactivity
with contents (internal / external hyperlinks)• visualization not customized to the device• difficult to extract structured textual information
Not
Not
Do you expect that the way they access and use articles
today to change in the future?
(281 responses of researchers in a variety of fields)

Prob.
not
10%

Maybe
46%

3%

sure
5%
Yes
36%

Document Structure Analysis

Document formats: dealing with PDF
PDF is a layout based data format for professional document rendering• Each PDF document is characterized by abody made of a set of objects that areusually grouped into pages: numbers,
strings, streams, arrays, dictionaries, etc.

BODY

Text: This
Postion: (x1, y1)• Objects are declared in the body of the
PDF files, often non sequentially

Image: binEnco
Postion: (x2, y2)(x1, y1)This(x2, y2)

Object n
Text: Reference
Postion: (xn, yn)

XREF• A cross-reference table (xref) lists all theobjects providing the file offset of each ofthem (optimized for reading, no need to
explore the contents of the whole PDF file)

Object 2…

• Each object in the stream is assigned a
special location inside a page viewport (aswell as a special size and style if applicable)

Object 1

Offset of Obj. 1
Offset of Obj. 2…
Offset of Obj. n(xn, yn)
ReferenceDocument Structure Analysis

Document formats: dealing with PDF
Customized approaches are necessary to extract
structured textual information from the PDF of scientific publications
Title: A Comparison of Layout based Bibliographic
Metadata Extraction Techniques
Author 1 Name: Michael Grantizer
Author 1 Affiliation: University of Passau, Germany
Author 1 Email: Michael.Grantizer@uni-passau.de
Abstract: Social research networks such as
Mendeley and CiteULike offer various services forcollaboratively managing bibliographic metadata…
…

The quality of scientific text mining often depends in the first place to the quality ofthe extraction of semi-structured textual contents from the original PDF file• robust with respect to different document layoutsa random sample of 125,000 publications from PubMed contains articles of 500 publishers, each
one with its own layout and style• covering and customized to the wide set of structural elements of scientific articles

Document Structure Analysis

General purpose PDF-to-text software
Convert PDF files to plain text, XML / HTML files with some layout information

Java library
Apache Project (actively maintained)
2.0.3 released on 17/9/2016

C ++ code
Freeware, based on xpdf
2.1 released on 14/6/2014

JPedalC++ code
Open source, based on xpdf
0.49.0 released on 15/11/2016
Used by GIMP, Okular,
Pdf2HTMLex, etc.

Commercial
Specialized to extract tables,
word lists and other elements…also, https://github.com/itext/itext7 , etc.

Try and compare these tools and others at: http://backingdata.org/pdfconv/

Document Structure Analysis

General purpose PDF-to-text software
Example of XML output
generated by

Document Structure Analysis

General purpose PDF-to-text software
Used by:
Open source project (GPL v3) of layout-preserving PDF to HTML converter(C++ mainly, based on Poppler to process PDF files)

Each PDF file is converted in an HTML file made of a set of DIV elements
properly positioned inside the page viewport
HTML source…
<div class="t x1 h1 y1 ff1 fs0 fc0
ws0">Languages</div>…
CSS CLASSES TO DEFINE THE LAYOUT.x1 {left: 441.233569px; }.y1 {bottom: 1139.024816px; }.h1 {height: 49.637990px; }.ff1 {
font-family: ff1;
line-height: 0.898000;}.fs0 {
font-size: 71.731200px; }.fc0 {
color: rgb(0,0,0); }.ws0 {
word-spacing: 0.000000px; }.t { position: absolute;
white-space: pre; }Document Structure Analysis

PDF-to-text for scientific publications• Integrating a general purpose PDF-to-text converter and postprocessing its output• Implementing customized approaches to identify structuralelements proper of scientific publications like: title, authors andaffiliations, abstract, section heading and contents, figures, tables,
formulas, bibliographic entries, in-line citation markers, etc.
• Robust to varied publishing styles

SectLabelDocument Structure Analysis

PDF-to-text for scientific publications
http://cermine.ceon.pl/
https://github.com/CeON/CERMINE• Java, open-source (GitHub) - libSVM• PDF analysis: based on both layout features and contents• Output: JATS XML
EXTRACTION OF HIERARCHY OF BASIC STRUCTURAL ELEMENTS

PDF-TO-XML
CONVERSION

Split words into chars (height, width, x, y), then apply Docsum:

https://github.com/itext/itext7• hierarchical grouping of elements relying on: nearest neighbor pairs of chars,
averages distances among chars and lines, line parallelness and overlap, etc.
• hierarchy of structural elements: chars, words, lines, zones, pages

METADATA ZONE CLASSIFICATION

Algorithm: SVM
Classes: title, author, affiliation, editor,
correspondence, type, abstract,
keywords, bib_info, dates
BIB. ENTRIES PARSING
Algorithm: CRF
Classes: firstname,surname, title, source,
volume, infopage, year, etc.
Feats: 42 like special chars,
match in dict. lists, etc.

BIB. ENTRIES IDENTIFICATION

Algorithm: kMeans (k=2) togroup reference lines into firstline of bib. entry / otherFeats: line length, indentation,
space between lines, etc.

HIGH-LEVELZONECLASSIFICATIONTEXT READING ORDER
IDENTIFICATION

Algorithm: SVM
Classes:
metadata,
references, body
Feats: geometric,
sequential,
lexical, heuristics
on content

Tkaczyk, D., Szostek, P., Fedoryszak, M., Dendek, P. J., & Bolikowski, Ł. (2015). CERMINE: automatic extraction of structured metadata from scientific literature.
nternational Journal on Document Analysis and Recognition (IJDAR), 18(4), 317-335.

Document Structure Analysis

PDF-to-text for scientific publications
http://cermine.ceon.pl/
https://github.com/CeON/CERMINE
TRAINING AND EVALUAITON OF SVM / CRF
Zone classifier (high-level and metadata) trained and evaluatedon the GROTOAP2 dataset:• 2,651 document from PubMed available as PDF + JATS XML• for each PDF + JATS XML pair of files:• PDF processed by CERMINE• JATS XML annotations used to label the zones
LAYOUT
identified by CERMINE (text sequence alignment algorithm)• analyzing a sample of the subset of the transferred annotations, a set of heuristic rules toimprove the quality of annotation transfer is developed and applied to each document
Citation parser trained and evaluated on the three datasets: 4,000 parsed citations: 2,000
from CiteSeer and Cora-ref and 2,000 from 1991 different PMC documents
RESULTS:• Citation parsing F-score: 93,3%• Metadata and Bibliography extraction F-score (47,983 PDF + metadata records): 77,5%
Tkaczyk, D., Szostek, P., Fedoryszak, M., Dendek, P. J., & Bolikowski, Ł. (2015). CERMINE: automatic extraction of structured metadata from scientific literature.
nternational Journal on Document Analysis and Recognition (IJDAR), 18(4), 317-335.

Document Structure Analysis

PDF-to-text for scientific publications
https://grobid.readthedocs.io/
https://github.com/kermitt2/grobid• Java (with JNI call to native CRF libraries: CRF++ or Wapiti) open-source (GitHub)• PDF analysis with CRF exploiting both layout features and contents• Output: TEI XML
HIERARCHY OF CRF SEQUENCE TAGGERS• position information (begin/end of line, in the doc.)• lexical information (vocabulary, large gazetteers)• layout information (font size, block, etc.)
Document segmentation hierarchy

Bibliographic entry segmentation hierarchy

PDF-TO-XML
CONVERSION(token info)• 9 CRF models for full texts• 14 intermediary labels in total• 55 final labels
Lopez, P. (2009, September). GROBID: Combining automatic bibliographic data recognition and term extraction for scholarship publications. In International Conference on Theory and Practiceof Digital Libraries (pp. 473-474). Springer Berlin Heidelberg.

Document Structure Analysis

PDF-to-text for scientific publications
https://grobid.readthedocs.io/
https://github.com/kermitt2/grobid
TRAINING AND EVALUATION OF CRF• Each model with its own set of features,
specialized to tag certain fields• Training sets specific to each model includedin the software (see table)• Trainer framework to generate and manuallyvalidate new training examples from acollection of PDF files• Best performing header metadata extraction
tool over 7 (Lipinski et al. 2013)

Model

Nº training
examplesExploit layout

A customized versions of GROBID is exploited by:
Extraction of bibliographic entries and matching against internal DB• about 300,000 PDF processed monthly (16 nodes Hadoop cluster)• failure rate of 1% of user uploaded PDF
Lopez, P. (2009, September). GROBID: Combining automatic bibliographic data recognition and term extraction for scholarship publications. In International Conference on Theory and Practiceof Digital Libraries (pp. 473-474). Springer Berlin Heidelberg.

Document Structure Analysis

PDF-to-text for scientific publications
http://pdfx.cs.man.ac.uk/• Online Web Service (Max 5Mb)
• PDF analysis is rule-based, relying on both layout features and contents• Output: JATS compliant XML files
Two steps PDF analysis:
STEP 1: a geometrical model of the textual contents and the layout of the information
contained in the PDF is build:• each word described is by orientation, position, font, etc.
• global document stats: most frequent font size and style, average line spacing
and font spacing, etc.
• neighbor words sharing similar text features are merged into blocks
STEP 2: based on the layout features previously
spotted, a set of rules is exploited to merge blocks
into regions and iteratively identify 18 elements
Inside the document (also on the basis o thesurrounding elements)
Constantin, A., Pettifer, S., & Voronkov, A. (2013, September). PDFX: fully-automated PDF-to-XML conversion of scientific literature. In Proceedings of the 2013 ACM symposium on Document
engineering (pp. 177-180). ACM.

Document Structure Analysis

PDF-to-text for scientific publications
http://pdfx.cs.man.ac.uk/

EVALUATION• 50,000 PDF + XML articles published by Elsevier in 2008
• 1,943 PDF + XML articles published in the PMC Open Access Subset in 2011 each one froma different journal

F-score per class – 0,95 similarity threshold between extracted and original textual contents of each field• Elsevier dataset more curated• PMC dataset suffers the high variation in style due to the presence of 1,943 articles eachone from a different journal• tables are with both datasets difficult to identify, while title and email are the easiest to
spot
Constantin, A., Pettifer, S., & Voronkov, A. (2013, September). PDFX: fully-automated PDF-to-XML conversion of scientific literature. In Proceedings of the 2013 ACM symposium on Document
engineering (pp. 177-180). ACM.

Document Structure Analysis

PDF-to-text for scientific publications
SectLabel

https://github.com/knmnyn/ParsCit/tree/master/bin/sectLabel• Perl and Ruby (CRF++), open-source (GitHub), process Omnipage output• PDF analysis relying on both layout features and contents
ALL LINES LABELINGPDF-TO-XML
CONVERSION(line based repr.)

EVALUATION
Adding to textual /content features also
layout features
improves the F-score ofabout 10 points (up to84%) and is particularly
beneficial for sections
like metadata, captions,hierarchical headers

Algorithm: CRF
Classes (26): address, affiliation, author, bodyText, categories, construct, copyright, email,equation, figure, figureCaption, footnote, keywords, listItem, note, page, reference,
sectionHeader, subsectionHeader, subsubsectionHeader, table, tableCaption, title
Line feats: location, number, punctuation, length, format / layout, differences in format /layout with previous and following lines
HEADER LINE LABELING

Algorithm: CRF, applied only to lines classified as Header in the previous stepClasses (13): abstract, categories, general terms, keywords, introduction, background,
related work, methodology, evaluation, discussions, conclusions, acknowledgements,
references
Line feats: location, number, punctuation, length, format / layout, differences in format /layout with previous and following lines

Luong, M. T., Nguyen, T. D., & Kan, M. Y. (2012). Logical structure recovery in scholarly articles with rich document features. Multimedia Storage and Retrieval Innovations for Digital Library
Systems, 270.
Councill, I. G., Giles, C. L., & Kan, M. Y. (2008, May). ParsCit: an Open-source CRF Reference String Parsing Package. In LREC (Vol. 8, pp. 661-667).

Document Structure Analysis

PDF-to-text for scientific publications
https://github.com/BMKEG/lapdftextProject• Java, open-source (GitHub)• PDF analysis is rule-based, relying on both layout features and contents• Output: JATS XML
1. Word blocks merged into text blocks by relying on page-level anddocument-level features like distance between words and lines, fontsizes and weight, etc.
PDF-TO-XML
CONVERSION(word blocks
detection)

JPedal

2. Rules exploited to assign to each text blocks a specificclass among: title, abstract,
heading, sub-heading,references, etc.
3. Other rules are used to define
the reading order of classified
text blocks and eventually merge
together contiguous text blocks
belonging to the same class

Ramakrishnan, C., Patnia, A., Hovy, E., & Burns, G. A. (2012). Layout-aware text extraction from full-text PDF of scientific articles. Source code for biology and medicine, 7(1), 1.

Document Structure Analysis

PDF-to-text for scientific publications
https://www.mendeley.com/download-mendeley-desktop/
How header metadata are extracted from a PDF document?
Initial approach:
PDF to text by means of PDFnet software(commercial) then apply iterative multi-step SVM
classifier (RBF kernel) as in:Han, H., Giles, C. L., Manavoglu, E., Zha, H., Zhang, Z., & Fox, E. A. (2003)
Automatic document metadata extraction using support vector machines.
In Digital Libraries proceedings. 2003 Joint Conference IEEE.

1) SVM that indipendently classifies each header line with respect to textual features: position,
number of words, number of capitalized words, % of words in specific dictionares, % of words occurringin a specific class of traning data
2) Contextual iterative classification by SVM where each header line is described by the feature set atstep 1 and the class assigned to the previous L and next N header lines – stop condition: label
assignments to header lines changes less than a predefined threshold with respect to previous iteration
3) Proper heuristics to segment lines with multiple authors

Document Structure Analysis

PDF-to-text for scientific publications
https://www.mendeley.com/download-mendeley-desktop/
How header metadata are extracted from a PDF document?
Later approach:• Trained on a large set of papers• Fields: title, authors, DOI, publication,
volume, issue, year, page, ranges

Evaluation:• Dataset: 26,000 PDFs with perfect metadata record in
Mendeley Catalogue• 2,4% couldn’t be converted to XML by pdftoxml• 83,9% can be processed extracting perfect metadata
records: authors, title, year, and publication venue (e.g.
journal, conference, magazine)“If you drop 10 PDFs into your Mendeley Library then, on
average, you’ll get perfect, citable metadata for 8-9 of them.”
https://mendeleyapi.wordpress.com/2014/10/15/pdf-extraction-gets-a-boost-with-our-new-api-service/https://krisjack.wordpress.com/2015/03/12/how-well-does-mendeleys-metadata-extraction-work/Document
Structure
Analysis

Comparing PDF-to-text
for scientific publications
Lipinski, M., Yao, K., Breitinger, C., Beel, J., & Gipp, B. (2013, July).
Evaluation of header metadata extraction approaches and tools for scientific PDF documents.
In Proceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries (pp. 385-386). ACM.

Dataset:1,153 random PDF articles from arXiv together with their metadata (title, authors, year,
abstract) dealing with Physics, Mathematics, Computer Science, Quantitative Biology,
Quantitative Finance and Statistics
Evaluation:
A100: 100 randomly selected articles and manual evaluation: 1 perfect match, 0.5 accent or ligature issues,
0.25 partial match, 0 no match
B100: 100 randomly selected articles and automated evaluation: Levenshtein distance normalized by thelength of the reference value for the field
B1153: whole dataset and automated evaluation: Levenshtein distance normalized by the length of thereference value for the field

SectLabel

Accuracy values

Document
Structure
Analysis

Comparing PDF-to-text
for scientific publications
Tkaczyk, D., Szostek, P., Fedoryszak, M., Dendek, P. J., & Bolikowski, Ł. (2015).
CERMINE: automatic extraction of structured metadata from scientific literature.
nternational Journal on Document Analysis and Recognition (IJDAR), 18(4), 317-335.

Dataset:1,943 pairs of PDF + JATS XML documents retrieved from PubMed Open Access Subset

Evaluation:
metadata extraction of CERMINE and other 4 similar tools (exact match)

In every cell there is precision, recall and F-score value

Document Structure Analysis

Bibliographic entry parsing
Tools:• most of them based on: rules or sequence taggers like HMM and CRF• some example:
 FreeCite: CRF++ library, CORA dataset, open-source (ruby), Web API
http://freecite.library.brown.edu/
 ParsCit: CRF++ library, open source (perl and C++)
https://github.com/knmnyn/ParsCit
Web API:• Web services that match against a citation database: CrossRef Metadata Search API: find metadata by DOI or by bibliographic entry
stringhttp://search.crossref.org/help/api
 Bibsonomy REST API – search posts by string:https://bitbucket.org/bibsonomy/bibsonomy/wiki/documentation/api/REST%20A
PI
https://www.bibsonomy.org/api/posts?resourcetype=bookmark&search=SemKey

Document Structure Analysis

Annotateddatasets
CORA Field Extraction dataset
https://people.cs.umass.edu/~mccallum/data/cora-ie.tar.gz
Seymore, K., McCallum, A., & Rosenfeld, R. (1999, July). Learning hidden Markov model structure for information extraction. In AAAI99 Workshop on Machine Learning for Information Extraction (pp. 37-42).

• 500 tagged references: author, title, journal, volume, pages, date
• 937 headers: title, author, affiliation, address, email, abstract, keywords
FluXcim Citation dataset
https://github.com/knmnyn/ParsCit/blob/master/doc/flux-cim-cs.tagged.txt
Cortez, E., da Silva, A. S., Gonçalves, M. A., Mesquita, F., & de Moura, E. S. (2007, June). FLUX-CIM: flexible unsupervised extraction ofcitation metadata. In Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries(pp. 215-224). ACM.

• 300 citation strings randomly from ACM Digital Library - CORA format
UMASS Citation dataset
http://www.iesl.cs.umass.edu/data/umasscitationfield
Anzaroot, S., & McCallum, A. (2013). A new dataset for fine-grained citation field extraction. In ICML Workshop on Peer Reviewing andPublishing Models.

• from arXiv papers in physics, mathematics, computer science and quantitative
biology• venue  publisher, note, web, institution,• 1,800 citations hierarchically labeled as:
department, etc.
• ref-markers• author  first, middle, last and affix• title• date  year and month• reference-id

Document Structure Analysis

Annotated datasets
CiteSeer Citation dataset
https://github.com/knmnyn/ParsCit/blob/master/doc/citeseerx.tagged.txt
Lawrence, S., Giles, C. L., & Bollacker, K. D. (1999, April). Autonomous citation matching. In Proceedings of the third annual conference on
Autonomous Agents (pp. 392-393). ACM.

• 200 tagged references: author, titile, journal, volume, pages, date
GROTOAP2 - GROund Truth for Open Access Publications
http://cermine.ceon.pl/grotoap2/
Tkaczyk, D., Szostek, P., & Bolikowski, L. (2014). GROTOAP2 The Methodology of Creating a Large Ground Truth Dataset of Scientific
Articles. D-Lib Magazine, 20(11), 13.

• 13,210 ground truth files in TrueViz XML format (1,640,973 zones in total) – each one
corresponding to a PDF + JATS XML of the Open Access Subset of PubMed Central• thanks to TrueViz, each file is represented as a hierarchy of structural elements:• a list of pages• each page contains a list of zones• each zone contains a list of lines• each line contains a list of words• and finally each word contains a list of characters

Structural elements have: text content, position on the page and dimensions. Also
the natural reading order for all structure elements is specified.
Each zone has labels (imported from PubMed) describing the role in the document areassigned to zones. There are 22 labels including: abstract, aknowledgments, affiliation ,author, bib_info, body_content, conflict_statement, copyright, dates, editor, equation, etc.

Document Structure Analysis

Conclusions• A precise extraction of structured textual contents from the PDF of
scientific publications is essential to enable any further text processingof their contents• Several PDF-to-text conversion tools are available, both general
purpose and customized to scientific publications• Such tools usually rely on both layout and textual features of scientificpublications and are rule-based or rely on supervised machine learning
approaches• A rich set of annotated corpora is freely available for further
experimentation

SCIENTIFIC DISCOURSE
CHARACTERIZATION

Outline• What is scientific discourse?
• Scientific discourse characterization– Annotation procedures and annotated corpora– Automated annotation of scientific texts• Overview of available datasets• Conclusions

Scientific Discourse Characterization

What is scientific discourse?
Scientific discourse concerns the characterization of how content ispresented, discussed and motivated in scientific literature

The most effective sources ofsupervision for training statisticalparsers are treebanks.

CONTRAST
Unfortunately, treebanks are
expensive, time-consuming tocreate, and not available for mostdomains.

BACKGROUNDAPPROACH
OUTCOME

Scientific Discourse Characterization

What is scientific discourse?
Scientific discourse concerns the characterization of how content ispresented, discussed and motivated in scientific literature

Why making scientific discourse explicit?
Provide new dimensions to drive the automatedanalysis of scientific publications• ease the interpretation of the information flow
• contextualize contents and characterize theirconnections with related pieces of research• discover relevant aspects, novelties and future
directions• support tasks like targeted information
extraction, content retrieval and summarization• assess the quality of content exposition

Scientific Discourse Characterization

Scientific discourse characterization
Steps towards the characterization and automated annotationof scientific discourse
Annotation procedure
and annotated corpus• definition of annotation schema• corpus annotation:• corpus content selection• annotation guidelines andprocedure• annotation results
BACKGROUNDAPPROACH
BACKGROUND
FUTURE WORK

Automated annotationof scientific texts• algorithmic approach• feature engineeringScientific Discourse Characterization

Scientific discourse characterization
Annotation procedure and annotated corpus

Two main approaches to the rhetorical analysis of a text:
Rhetorical Structure Theory:
relations between clauses orlarger text segments
The most effective sources ofsupervision for training statisticalparsers are treebanks.

CONTRAST
Unfortunately, treebanks are
expensive, time-consuming tocreate, and not available for mostdomains.

Zone Analysis: characterizationof the global rhetorical status ofeach text unit (sentence)

Scientific Discourse Characterization

Scientific discourse characterization
Annotation procedure and annotated corpus
Rhetorical Structure Theory (Mann & Thompson)
Coherent texts consist of minimal units, which are linked
to each other, recursively, through rhetorical relations
thus generating a tree-like representation of a text
Many neo-pagan religions, such as
Wicca, use aspects of ancient Greek
religions in their practice; Hellenic
polytheism
insteadfocuses
exclusively on the old religions, as
far as the fragmentary nature of thesurviving source material allows.

Many neopagan religions,such as Wicca,
use aspects ofancient Greek
religions in theirpractice;

Hellenic
polytheism
instead focuses
exclusively on
the old religions,as far as thefragmentary
nature of thesurviving source
material allows.

Scientific Discourse Characterization

Scientific discourse characterization
Annotation procedure and annotated corpus
Rhetorical Structure Theory (Mann & Thompson)
Coherent texts consist of minimal units, which are linked
to each other, recursively, through rhetorical relations
thus generating a tree-like representation of a text
• 23 relations (symmetric and not;several extensions proposed)• the text is represented by a recursive
tree structure (the most relevant
minimal units are usually placed on
the top)
Among others, exploited to:• Check text coherence• Natural Language Generation• Corpus analysis and study of discourse
phenomena• Text summarization

Many neopagan religions,such as Wicca,
use aspects ofancient Greek
religions in theirpractice;

Hellenic
polytheism
instead focuses
exclusively on
the old religions,as far as thefragmentary
nature of thesurviving source
material allows.

Scientific Discourse Characterization

Scientific discourse characterization
Annotation procedure and annotated corpus
Zone Analysis

Several annotations schemes and procedures have been proposed
to characterize text units with respect to:• the type and complexity of the discourse elements identified• the type of text units to which the discourse is applied (sentences,segments of sentences, specific relations or events occurring inthese sentences)
Knowledge Claim discourse Model and Argumentative Zoning
Core Scientific Concepts
IMRAD structure
Dr. Inventor Scientific Discourse Schema

Scientific Discourse Characterization

Scientific discourse characterization
Annotation procedure and annotated corpus
Zone Analysis: Knowledge Claim Discourse Model
The argumentative structure of a scientific article is basedon the need of authors to convince the reader of their contributions
by claiming the ownership of a new piece of knowledge
Scientific discourse develops throughout a set of ‘rhetorical moves’ that are explicit statements, referred toas Knowledge Claims, useful to characterize and justify the contributions of a specific piece of work
EXAMPLES OF RHETORICAL MOVES

Properties of research space

Open domain word sense disambiguation presents several
interesting challenges both semantic and computational.

Properties of new solution (US)The proposed methodologies solves the issues related withthe high computational cost of knowledge analysis.

Properties of existing solution(THEM)The method proposed by Ray et. al., 2010 stressed theimportance of correctly dealing with semantic draft.

Relationship between existing andnew solution (US and THEM)

Our solution improves the previous state-of-the-art method (Gilet al., 2012) by exploiting a new set of data sources.

Scientific Discourse Characterization

Scientific discourse characterization
Annotation procedure and annotated corpus
Zone Analysis: Argumentative Zoning
Argumentative Zoning Annotation Schema bundles together similarrhetorical moves casting the general argumentation recognition
Knowledge Claim Discourse Model into a sentence classification task
AZ Annotation Schema

AZ Corpus
2%
6%

2%

• 80 conference articles in computational
linguistics• 12,188 sentences assigned to one of 7
Categories• Avg. annotator agreement K: 0,71

5%
16%
67%

Online at (SciXML format):
http://www.cl.cam.ac.uk/~sht25/AZ_corpus.html

2%

Annotation categories are defined on the basis of who owns the knowledge claim

Scientific Discourse Characterization

Scientific discourse characterization
Annotation procedure and annotated corpus
Zone Analysis: Argumentative Zoning cross-domain validity
Test if non-expert humans can annotate the sentences of text fromdifferent domains (Computational Linguistics and Chemistry) withrespect to an extended version of the Argumentative Zoning Schema
Argumentative Zoning II Schema (to model typical Chemistry argumentation)
AZ AimAZ OwnAZ Background

AZ Contrast

AZ OtherAZ Basis
AZ OwnData:• 30 Chemistry papers• 9 Computational
Linguistics papers (CL)

Annotation:
3 annotators experts
in CL with different
levels of expertise in
Chemistry chemistry intro

Scientific Discourse Characterization

Scientific discourse characterization
Annotation procedure and annotated corpus
Zone Analysis: Argumentative Zoning cross-domain validity
Test if non-expert humans can annotate the sentences of text fromdifferent domains (Computational Linguistics and Chemistry) withrespect to an extended version of the Argumentative Zoning Schema
Argumentative Zoning II Schema (to model typical Chemistry argumentation)
AZ AimAZ OwnAZ Background

AZ Contrast

AZ OtherAZ Basis
AZ OwnInter-annotator agreement
is comparable acrossdomains (k is 0.65 in CL
and 0.71 in Chemistry)
Higher agreement among
Chemistry experts 
a little improvement ofannotation quality withdomain knowledge

Scientific Discourse Characterization

Scientific discourse characterization
Automated annotation of scientific texts
Zone Analysis: Argumentative Zoning
Teufel, S., & Moens, M. (2002). Summarizing scientific articles: experiments with relevanceand rhetorical status. Computational linguistics, 28(4), 409-445.

Classifier: Naïve Bayes
Sentence features:• Structural:• Absolute sentence location• Position of sentence within section and paragraph• Type of headline of current section (15 prototypical types)• Words shared with title or headlines• Significant words (sentences that contain one of the 18 highest TF*IDF words)• contain self-citation• Sentence-scoped:• Verb (voice, tense, modal)• Contain citation• Most probable previous sentence category• Meta-discourse expression (formulaic expressions, type of agent, type of action)

Scientific discourse characterization
Automated annotation of scientific texts
Zone Analysis: Argumentative Zoning
Teufel, S., & Kan, M. Y. (2011). Robust argumentative zoning for sensemaking
in scholarly documents (pp. 154-170). Springer Berlin Heidelberg.

Argumentative zoning sentence classifier robustwith respect to noisy input: plain text or textual input generated from
PDF to text conversion or OCR
Explicit structure (SciXML)<TITLE>Paper title</TITLE><HEDER>Section title</HEADER><S>First sentence of the paper.</S><S>Second sentence of the paper.</S>….

Plain textual contents
Paper title
Section title
First sentence of the paper. Second
sentence of the paper.
….

Classification without using structural features: Absolute sentence location / Position ofsentence within section and paragraph / Type of headline of current section / Words sharedwith title or headlines / Significant words (TF*IDF) / Is self-citation

Scientific Discourse Characterization

Scientific discourse characterization
Automated annotation of scientific texts
Zone Analysis: Argumentative Zoning
Teufel, S., & Kan, M. Y. (2011). Robust argumentative zoning for sensemaking
in scholarly documents (pp. 154-170). Springer Berlin Heidelberg.

Argumentative zoning sentence classifier robustwith respect to noisy input: plain text or textual input generated from
PDF to text conversion or OCR
Classifier: Maximum entropy (automatically spotted andPOS tagged sentences)

Sentence-scoped features:• Normalized number of sentencesfrom the beginning• Overlap with first 100 words of text
• Verb (voice, tense, modal)• Contain citation, is self-citation• Set reduced to agent type
• Raw tokens, bigrams, trigrams

F-score withstructuralfeatures
52%
38%
45%
26%
44%
86%
61%

The agreement with theGold Standard, even withnoisy input data (PDF totext extractor, automatic
sentence and paragraphidentification and POS
tagging) is still respectable
and the classifier is stillrobust and fast to execute

Scientific Discourse Characterization

Scientific discourse characterization
Automated annotation of scientific texts
Zone Analysis: Argumentative Zoning
Séaghdha, D. O., & Teufel, S. (2014). Unsupervised learning of rhetorical structurewith un-topic models. In COLING (pp. 2-13).

The linguistic constructs that are used to express the rhetoricalfunctions in a paper are independent from the topic

The problem of _______________________ has received a lot of attention because of its
relevance to __________________________________________________. __ proposed an
approach based on ____________________________________________________________.
In this paper we present a method to __________________________________________________________________________________. We demonstrate the empirical effectiveness of
our method reporting experiment on ______________________________________.
Topic-independent template for abstracts of NLP papers

Scientific Discourse Characterization

Scientific discourse characterization
Automated annotation of scientific texts
Zone Analysis: Argumentative Zoning
Séaghdha, D. O., & Teufel, S. (2014). Unsupervised learning of rhetorical structurewith un-topic models. In COLING (pp. 2-13).

Two language models can be composed by a binary-valued latent variable
to generates the words of a paper:
LDA topic model: to generate the topic dependent words of a document
Word distribution of a rhetorical zone: to represent transition probabilities acrossrhetorical categories of sentences a Markov model is used since the probability of azone is dependent on the zone of the previous sentence
The problem of Word Sense Disambiguation has received a lot of attention because of its
relevance to the correct interpretation and integration of textual contents. We proposed an
approach based on knowledge resources built with unsupervised approaches from a corpus.
In this paper we present a method to extend semantic networks to improve theireffectiveness on Word Sense Disambiguation. We demonstrate the empirical effectiveness of
our method reporting experiment on a wide collection of sense annotated corpora.
Topic-independent template for abstracts of NLP papers, filled

Scientific Discourse Characterization

Scientific discourse characterization
Automated annotation of scientific texts
Zone Analysis: Argumentative Zoning
Séaghdha, D. O., & Teufel, S. (2014). Unsupervised learning of rhetorical structurewith un-topic models. In COLING (pp. 2-13).

Given a number collection of documents, a number of topics and a numberof rhetorical zones to discover, this unsupervised approach assign eachsentence to:• a distribution of topics  most likely topic• a distribution of rhetorical zones  most likely zone
The problem of Word Sense Disambiguation has received
a lot of attention because of its relevance to the correctinterpretation and integration of textual contents.

TOPIC N. 20 (over 100)
ZONE N. 3 (over 10)

Scientific Discourse Characterization

Scientific discourse characterization
Automated annotation of scientific texts
Zone Analysis: Argumentative Zoning
Séaghdha, D. O., & Teufel, S. (2014). Unsupervised learning of rhetorical structurewith un-topic models. In COLING (pp. 2-13).

How good is this approach to cluster sentences into rhetorical zones?
Dataset: 1000 abstracts annotated with Argumentative Zoning
Zone clustering approaches:• Boilerplate-LDA (presented in the paper)• Boilerplate-LDA with probability of zone transition independent from adjacentsentences (no Markov model for zone transition)• Boilerplate-LDA without topics• kMeans (FEATURES: tf-idf-transformed lexical frequencies, part-of-speech tags and a location feature computed by
dividing the abstract into 5 bins)

Compared with Gold Standard sentence clustering into zones

Results:
Boilerplate-LDA (presented in the paper) generates clusters of sentences thatare more consistent with Gold Standard clusters

Scientific Discourse Characterization

Scientific discourse characterization
Automated annotation of scientific texts
Zone Analysis: Argumentative Zoning
Séaghdha, D. O., & Teufel, S. (2014). Unsupervised learning of rhetorical structurewith un-topic models. In COLING (pp. 2-13).

Can we use learned zones as features to improve supervised classification?
Dataset: 1000 abstracts annotated with Argumentative Zoning
Classification approaches: Logistic Regression with history feature and CRF• Base features: tf-idf-transformed lexical frequencies, part-of-speech tags and alocation feature computed by dividing the abstract into 5 bins extended with:
1. Boilerplate-LDA zone feature (index of the zone from 1 to 10)
2. Topics that are assigned to the words of a sentence by LDA are set to true (onefeature per topic)
3. Only topic that is assigned with more frequency set to true (one feature per
topic)

Results:
Performance of a Logistic Regression and CRF classifier improves with theaddition of zone features (item 1, helps to identify the rhetorical zone) and notwith the addition of topic features

Scientific Discourse Characterization

Scientific discourse characterization
Automated annotation of scientific texts
Zone Analysis: Argumentative Zoning
Merity, S., Murphy, T., & Curran, J. R. (2009, August). Accurate argumentative zoning with maximum
entropy models. In Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly
Digital Libraries (pp. 19-26). Association for Computational Linguistics.
• sentence features: unigram, bigram, section counter, location inside section and paragraphand length• improvement of sentence classification performance on Argumentative Zoning corpus by usinga maximum entropy classifier• by using an HMM with only unigrams and bigrams the classification accuracy improvement isrelevant up to an history of the four previous decisions
Feltrim, V. D., Teufel, S., das Nunes, M. G. V., & Aluísio, S. M. (2006). Argumentative zoning applied to
critiquing novices’ scientific abstracts. In Computing Attitude and Affect in Text: Theory andApplications (pp. 233-246). Springer Netherlands.
• SciPo: tools that applies a set of rules to evaluate the coherence of scientific abstracts ofnovices on the basis of their rhetorical structure spotted by AZ classifier• Argumentative Zoning schema ported to scientific abstract in Portuguese: the category OWN
divided into Methodology, Results and Conclusion• Corpus of 52 abstracts annotated• Automated classification experiments with Teufel’s features ported to Portuguese: Classifier:Naïve Bayes (13-folds cross validation) accuracy 74%, K with gold standard 0.65

Scientific Discourse Characterization

Scientific discourse characterization
Automated annotation of scientific texts
Zone Analysis: Argumentative Zoning
Mizuta, Y., & Collier, N. (2004, May). An Annotation Scheme for a Rhetorical Analysis of Biology
Articles. In LREC (pp. 1737-1740).

• 20 online articles taken from major biology journals annotated in order to develop
and refine the annotation schema on the bases of Teufel’s Argumentative Zoning• extended modified version of AZ Schema to include:• a finer grained classification of the author’s own work
• an explicit relation between the data presented and the findings
Hachey, B., & Grover, C. (2006). Extractive summarisation of legal texts. Artificial Intelligence andLaw, 14(4), 305-345.

• adaptation of Argumentative Zoning to the legal domain• unlike scientific texts, the fundamental communicative purpose of a judgment is tolegitimise a decision, by showing that it derives, by a legitimate process, from
authoritative sources of law• schema categories: FACTS, PROCEEDING, BACKGROUND, FRAMING, DISPOSAL,TEXTUAL, OTHERS

Scientific Discourse Characterization

Scientific discourse characterization
Annotation procedure and annotated corpus
Zone Analysis: Core Scientific Concepts
A paper is a human readable representation of a scientific investigation:a scientific discourse annotation schema should point out thecomponents of the scientific investigation
Category

Description

Hypothesis

An unconfirmed statemen which is a stepping stone of the investigation

Motivation

The reason behind the investigation

Background

Generally expected background knowledge and previous work

Goal

A target state of the investigation where intended discoveries are made19%
1%

ObjectAn entity which is the product or main theme of the investigation (advantage /disadvantage)

3%

MethodMeans by which the authors seek to achieve the goal of the investigation (old/ new – advantage / disadvantage)

11%

Experiment

An experimental method

Model

A statement about a theoretical model or framework

Observation

The data / phenomena recorded in an investigation

ResultFactual statements about the outputs, interpretation of an observation

https://www.aber.ac.uk/en/cs/r
14% esearch/cb/projects/art/art21% corpus/

Conclusion

Statements inferred from observations and results

9%

2%
1%

10%
9%

ART Corpus• 265 papers from thedomains of chemistry andbiochemistry• 39,915 sentences• Avg. annotator agreement
K: 0,55
Online at (SciXML format):Scientific Discourse Characterization

Scientific discourse characterization
Automated annotation of scientific texts
Zone Analysis: Core Scientific Concepts
Liakata, M., Saha, S., Dobnik, S., Batchelor, C., & Rebholz-Schuhmann, D. (2012). Automatic recognition ofconceptualization zones in scientific articles and two life science applications. Bioinformatics, 28(7)Classifiers: SVM (linear), CRF
Sentence features:• Structural:• Absolute sentence location• Section ID (incremental integer, up to 10)• Length and position of sentence within section andparagraph• Type of headline of current section (16 types ofprototypical headers)• Sentence-scoped:• No citations, one citation, +1 citation• Category of previous sentence (not CRF)
• Unigrams, bigrams and trigrams lemmatized• Verb POS, passive or not, presence• Verb class (10 classes) obtained by clustering verbs withfrequency > 150• Grammatical triples from dependency tree

Results:• Accuracy: SVM: 51,6% CRF: 50,4%• Most relevant feature sets: bigrams,
triples from dependency tree, verbs aswell as structural features as history andsection heading type (ngram 65,000
features vs 13,000 all other features)• There is not always a direct correlationof annotator agreement and classifier
performance: Experiment and Model
have an higher F-score but low interannotator agreement

Scientific Discourse Characterization

Scientific discourse characterization
Annotation procedure and annotated corpus
Zone Analysis: Core Scientific Concepts, multi-class
James Ravenscroft, Maria Liakata, Anika Oellrich, and Shyamasree Saha (2016). Multi-label annotation inscientific articles – The Multi-label Cancer Risk Assessment Corpus. LREC

Dealing with the case in which more than one Core Scientific Concept
appears in a single sentence
Bone marrow stromal cells were treated with AhR agonists and bacterial
lipopolysaccharide (LPS) to mimic innate inflammatory cytokine responses.

METHOD
GOAL

Multi-CoreSC Corpus
50 papers from the domain of cancer risk assessment Environmental Health Perspectives (21),
Carcinogenesis (15), Toxicological Sciences (9), Journal of Biological Chemistry (3), Occupational andEnvironmental Medicine (1), PlosOne (1)• 8,501 sentences
Online at (SciXML format):
http://www.sapientaproject.com/wp-content/uploads/2016/05/consensus_annotated.zip

Scientific Discourse Characterization

Scientific discourse characterization
Annotation procedure and annotated corpus
Zone Analysis: Core Scientific Concepts, multi-class
James Ravenscroft, Maria Liakata, Anika Oellrich, and Shyamasree Saha (2016). Multi-label annotation inscientific articles – The Multi-label Cancer Risk Assessment Corpus. LREC

Dealing with the case in which more than one Core Scientific Concept
appears in a single sentence
Multi-CoreSC CRA Corpus

Is CoreSC CRF classifier domain indep.?

Old: trained on ART corpus, New: trained and• 3 biology expert annotators
tested on CRA corpus• weighted kappa > 0.55 for each ann. pair tested on CRA corpus• 12.5% of sentences obtained a multiMost influential features of
CoreSC label
CoreSC annotation are domain• multi label conciliation procedure to
specific
generate Gold Standard: lower number oflabels across annotators in Gold Standard.
Object and Experiment: only
Labels are ranked with respect totwo categories that are
popularity and in case of equal popularity
consistently identified withoutwith respect to priority
domain adaptation

Scientific Discourse Characterization

Scientific discourse characterization
Annotation procedure and annotated corpus
Zone Analysis: Argumentative Zoning vs Core Scientific Concepts
Liakata, M., Teufel, S., Siddharthan, A., & Batchelor, C. R. (2010, May). Corpora for the Conceptualisation
and Zoning of Scientific Papers. In LREC.

AZ-IIcharacterize the ownership of theknowledge claims presented in thepaper, thus identifying and motivating
the new contributions of the author

CoreSC
describes the structure of theinvestigation characterizing the highlevel scientific concept presented ineach part of the paper
Category

Description

Hypothesis

An unconfirmed statemen which is a stepping stone of the investigation

Motivation

The reason behind the investigation

Background

Generally expected background knowledge and previous work

Goal

A target state of the investigation where intended discoveries are madeObjectAn entity which is the product or main theme of the investigation (advantage/ disadvantage)

MethodMeans by which the authors seek to achieve the goal of the investigation (old/ new – advantage / disadvantage)

Experiment

An experimental method

Model

A statement about a theoretical model or framework

Observation

The data / phenomena recorded in an investigation

ResultFactual statements about the outputs, interpretation of an observation

Conclusion

Statements inferred from observations and results

Scientific Discourse Characterization

Scientific discourse characterization
Annotation procedure and annotated corpus
Zone Analysis: Argumentative Zoning vs Core Scientific Concepts
Liakata, M., Teufel, S., Siddharthan, A., & Batchelor, C. R. (2010, May). Corpora for the Conceptualisation
and Zoning of Scientific Papers. In LREC.

AZ-IICoreSC36 papers annotated with both schemas
Schemata have complementary roles - it would be beneficial to annotate a text
with respect to both schemata. In particular:• AZ-II identifies knowledge claims that permeates several CoreSC concepts• CoreSC has more granularity when dealing with content-related categories
AIM

Goal

PREV_OWN

Hypothesis

OTHER

Motivation

GAP_WEAK

Object

CO_GRO

BackgroundScientific Discourse Characterization

Scientific discourse characterization
Annotation procedure and annotated corpus
Zone Analysis: IMRAD
Luciana B. Sollaci & Mauricio G. Pereira (July 2004). The introductio, methods, results, and discussion(IMRAD) structure: a fifty-year survey. J Med Libr Assoc. 2004 July; 92(3): 364–371. 92 (3)• Introduction > Methods > Results > Discussion• Structure common to most health science journals• Today more complex derived structures are often used
Random sample of (n = 1,297) articles published in British
Medical Journal, JAMA, The Lancet, and the New England
Journal of Medicine, 1935–1985• first used in 1940s• in 1970s 80% of compliant
papers• since 1980s most of healthscience papers are compliant

Scientific Discourse Characterization

Scientific discourse characterization
Annotation procedure and annotated corpus
Zone Analysis: Dr. Inventor Scientific Discurse Schema
Fisas, B., Ronzano, F., & Saggion, H. (2015). On the Discoursive Structure of Computer Graphics Research
Papers. In The 9th Linguistic Annotation Workshop held in conjuncion with NAACL 2015 (p. 42).
Fisas, B., Ronzano, F., & Saggion, H. (2016). A Multi-Layered Annotated Corpus of Scientific Papers. LREC.

Schema defined by annotating Computer
Graphics papers, starting from AZ and CoreSC
schemas (15 categories, then reduced to 5 top level + 2 second level)
Background
Approach
Challenge
Hypothesis
Goal
OutcomeContribution
Future Work

Dr. Inventor Corpus
10
papers10papersFLUIDSIMULATION
CLOTH
SIMULATION

10papers10papersSKINNING

MOTION• 40 papers / 10,403 sentences• Multi-layered annotations: discursive
structure, citation purpose, summary
sentence relevanceScientific Discourse Characterization

Scientific discourse characterization
Annotation procedure and annotated corpus
Zone Analysis: Dr. Inventor Scientific Discurse Schema
Fisas, B., Ronzano, F., & Saggion, H. (2015). On the Discoursive Structure of Computer Graphics Research
Papers. In The 9th Linguistic Annotation Workshop held in conjuncion with NAACL 2015 (p. 42).
Fisas, B., Ronzano, F., & Saggion, H. (2016). A Multi-Layered Annotated Corpus of Scientific Papers. LREC.

Annotators

Annotation workflow
Training
SessionAvg. annotator agreement K: 0,67

Annotation check after:• 5 papers• 15 papers• 25 papers

Distribution of sentence rethorical class (over papers’ length)4%

Challenge

20%

Background58%

Approach16%

Outcome2%

Future Work
Online at: http://sempub.taln.upf.edu/dricorpus

Scientific Discourse Characterization

Scientific discourse characterization
Automated annotation of scientific texts
Zone Analysis: Dr. Inventor Scientific Discurse Schema
Fisas, B., Ronzano, F., & Saggion, H. (2015). On the Discoursive Structure of Computer Graphics Research
Papers. In The 9th Linguistic Annotation Workshop held in conjuncion with NAACL 2015 (p. 42).
Fisas, B., Ronzano, F., & Saggion, H. (2016). A Multi-Layered Annotated Corpus of Scientific Papers. LREC.

CORPUS: 8,777 sentences that have been
manually associated to one of the 5 high levelclasses
CLASSIFIERS: Logistic regression, SVM (linear)
FEATURES: sentence position (only structuralfeat.), unigrams, bigrams, three-grams, dep. tree
dept, num. and type of edges, dep. tree tokens,num and syntactic role of citations, category ofprevious sentence
RESULTS:• in general the F-score of each category is
proportional to the number of training instances• Future Work has more strongly distinctive
linguistic features than Challenge

58%
20%
4%

2%
16%

Percentage of annotatedsentences by category

Scientific Discourse Characterization

Scientific discourse characterization
Automated annotation of ABSTRACTS of scientific texts
Guo, Y., Korhonen, A., Liakata, M., Karolinska, I. S., Sun, L., & Stenius, U. (2010, July). Identifying theinformation structure of scientific abstracts: an investigation of three different schemes.
In Proceedings of the 2010 Workshop on Biomedical Natural Language Processing (pp. 99-107). ACL.

CORPUS: 1,000 MedLine abstracts concerning Cancer Risk Assessment (7,985 sentences)On-line at: http://www.cl.cam.ac.uk/~yg244/abstract_az.html
3 ANNOTATION SCHEMAS: (k measured over 1/3 of the corpus, three annotators)• Objective, Method, Results and Conclusion (K=0,84) SVM acc: 0.89
• AZ, 7 categories (K=0,85) SVM acc: 0.90
• CoreSC 11 categories (K=0,50) SVM acc: 0.81
CLASSIFIERS: Naïve Bayes, SVM with linear kernel (Weka)FEATURES: location (10 equal parts), unigram, bigram, verb class (60 cluster of frequentverbs), grammatical triples from dependency tree, passive verb
RESULTS:• SVM outperforms Naïve Bayes in all cases (accuracy reported before)• Best features for all schemas: bigrams, verb and unigrams• Worse features for all schemas: history and voice worst (with abstract, the historyof the categories is more varied and has less relevance than in the case in which we
consider the whole text)

Scientific Discourse Characterization

Scientific discourse characterization
Automated annotation of ABSTRACTS of scientific texts
Hirohata, K., Okazaki, N., Ananiadou, S., Ishizuka, M., & Biocentre, M. I. (2008, January). Identifying
Sections in Scientific Abstracts using Conditional Random Fields. In IJCNLP (pp. 381-388).

CORPUS: 51,000 MedLine abstracts with sentences divided in Objective, Method, Resultand Conclusion
CLASSIFIERS: SVM (linear kernel), CRF
FEATURES: unigrams and bigrams also from next and previous sentence features, relative
sentence location
RESULTS:• CRF outperformed the SVM with featuresfrom previous and next sentence showing thatis more adequate to classify sentences ofscientific abstracts•Since features are mainly based on lexical
contents of annotated text (unigrams andbigrams), the accuracy strongly improves
when a greater dataset is consideredScientific Discourse Characterization

Scientific discourse characterization
Automated annotation of scientific texts: ACTIVE LEARNING
Guo, Y., Silins, I., Stenius, U., & Korhonen, A. (2013). Active learning-based information structure analysis
of full scientific articles and two applications for biomedical literature review. Bioinformatics, 29(11)CORPUS: 50 biomedical articles (8,171 sentences) annotated with AZ categories
CLASSIFIER: SVM (linear kernel)
Driven selection of new samples to consider to increase the training set by means of
three strategies:• least confident sampling: instance with more classification uncertainty• margin sampling: instance with the smallest margin between the priors of the two most likely
labelings• query-by-bagging: a committee of models trained on subset of training instances is createdand chosen the instance for which the committees disagree the most (most informative
instance)

FEATURES: unigrams, bigrams, normalized section name, location inside section andparagraph, number of cits and table/figure references, verb class, tense, voice, dep. rels
Fully
RESULTS:supervised• active learning with SVM trained on
accuracy
6% of the corpus performs surprisingly(8,171
instances):
well with the accuracy of 82%, just 2%
0.84
lower than fully supervised learning

Scientific Discourse Characterization

Scientific discourse characterization
Automated annotation of scientific texts
Guo, Y., Korhonen, A., & Poibeau, T. (2011, July). A weakly-supervised approach to argumentative zoningof scientific documents. In Proceedings of the Conference on Empirical Methods in Natural Language
Processing (pp. 273-283). Association for Computational Linguistics.

• Use of Active learning and semi-supervised approaches to improve discursive sentenceclassification• Active SVM outperforms the best supervised SVM with a statistically significant difference
exploiting only a fraction of the training data
Guo, Y., Reichart, R., & Korhonen, A. (2013, June). Improved Information Structure Analysis of ScientificDocuments Through Discourse and Lexical Constraints. In HLT-NAACL (pp. 928-937).

• Adding manually defined constraints to complement the statistical classification ofsentences• Two types of constraints are defined:• lexical: there is one or more reference to figures and tables, there is one or morecitation, there are occurrences of specific word classes• discursive: is the first / last part of the paragraph or section

Scientific Discourse Characterization

Overview of available datasets
AZ Corpus: 80 articles computational linguistics
http://www.cl.cam.ac.uk/~sht25/AZ_corpus.html

ART Corpus: 265 papers from the domains of chemistry andbiochemistry
https://www.aber.ac.uk/en/cs/research/cb/projects/art/art-corpus/

MultiCoreSC CRA Corpus: 50 papers from the domain of Cancer Risk
Assessment
http://www.sapientaproject.com/wpcontent/uploads/2016/05/consensus_annotated.zip

Dr. Inventor Multi-layered Corpus: 40 papers from the domain ofComputer Graphics
http://sempub.taln.upf.edu/dricorpus

MedLine Abstracts Corpus: 1,000 MedLine abstracts onCancer Risk
Assessment
http://www.cl.cam.ac.uk/~yg244/abstract_az.html

Scientific Discourse Characterization

Conclusions• The characterization of scientific discourse provides valuable
information to enhance several scientific text mining tasks like text
quality assessment, information extraction, content retrieval andsummarization• Zone Analysis is the most widespread approach to characterizescientific discourse, often at sentence level• Annotation schemas often offers complementary views by modelingdifferent aspects of scientific discourse• Even if minimal, often annotation schemas need to be adapted to thespecific domain of the scientific textual contents to characterize• Supervised approaches are widely explored: classifiers (Naïve Bayes,
logistic regression, SVM) or sequence labeling approaches (CRF)• A rich set of annotated corpora is freely available for further
experimentation

CITATION ANALYSIS

Outline• Citations in scientific literature• How citations are studied?
– Citation network analysis– Citation function– Citation prediction and recommendation– Citation-based summarization• Citation graphs• Conclusions

Citation analysis

Citations in scientific literature
Citations are the primary device used in scientific literature
to relate a piece of work with other relevant (published) materials

We cite papers to:• ground the arguments and give the work factualbasis• avoid plagiarism (intellectual honesty)• attribute prior or unoriginal work and ideas tothe correct sources• allow the reader to determine independently
whether the referenced material supports theauthor's argument in the claimed way(demonstrate assessors and critics you havecarried out the necessary research)• enable the reader to independently evaluate thestrength and validity of the material the author
has used

CITATION
Body of the paper

In-line citation

Bibliography

Bibliographic entry

Citation analysis

Citations in scientific literature
The elements of a citation

Each citation is a directed link from a citing paper to a cited paper
citing paper

cited paper

Citation context
In-line citation
Cited span

The text of the citing paper surrounding an in-line citation andmotivating the same citation is referred to as citation contex
The excerpt of the cited paper that explains the actual contents cited by
the citing paper surrounding is referred to as cited span

Citation analysis

How citations are studied?
• Citation network analysis• Citation function• Citation prediction and recommendation• Citation-based summarization

Citation analysis

Citation network analysis
Citation networks

Citation networks:• nodes: papers• arks: directed from citing to cited paper
The more often a single paper is cited, the more
important it seems to beCitation analysis

Citation network analysis
Bibliographic coupling and co-citation networks
Bibliographic coupling network:• nodes: papers• arks: undirected, connect pairs of documentsthat share one or more cited documents

coupling strength: 2

Retrospective: is limited to the papers cited by a pairof articles and cannot vary with time
The more often two papers are cited together, the more
likely they are to be part of some research question or
ongoing problem or conversation topic within the discipline

Co-citation network:• nodes: papers• arks: undirected, connect a pair of papers ifthey are cited by the same document(s)Non-retrospective: may vary by new citations received
by the papers in the future
Co-Citation Proximity Index (CPI) can be introduced to account for the placement of citations relative toeach other. Documents co-cited at greater relative distances in the full text receive lower CPI values.

co-citation strength: 2
INTERACTIVE EXAMPLE:
http://jgoodwin.net/network/cites-slider.html

Citation analysis

Citation network analysis
Author coupling and co-citation networks
Author bibliographic coupling:• nodes: authors• arks: undirected, equals to the number ofreferences that the publications of the pairs ofauthors have in common

coupling strength: 2
citepublicationcite
publication

Method to map the research activities of active
authors themselves for a more realistic pictureof the current state of research in a field

Author co-citation:• nodes: authors• arks: undirected, the number of times the pairof connected authors are cited together by thesame article
Method to study the external and internal aswell as recent and historicalintellectual influences on the field

cites
authorcites
authorco-citation strength: 2

Citation analysis

Citation network analysis
How citation networks are exploited?
• identify “hot” areas and key authors (authors that are mostcollaborative or are most highly cited)  centrality, in-degree, outdegree• community detection (meaningful communities of researchers) clustering methods• understand the research habits, trends, and topological patternsof the researchers• spot and characterize productivity, patterns and trends• provide complementary data to enhance the analysis of thecontents of scientific publications

Citation analysis

Citation network analysis
Citation networks

February 2016

• 47,742,000 papers with at least one
reference or one citation (36,8% of total
papers)• 528,682,289 internal citations• each paper in the graph is cited on
average 4.17 times

October 2016

• 384,413 papers• 1,751,463 internal citations• each paper in the graph is cited on
average 4.56 times

Citation analysis

Citation network analysis
Improving detection of scientific topic evolution by citation network(Scientific) Topic detection and evolution
Discover how and what topics change over time since the evolution of a topic in a
specific time period can boost the investigation of other topics in subsequent periods
YEAR 1YEAR 2

YEAR 3

YEAR 4

TOPIC 1

TOPIC 1

TOPIC 1

TOPIC 6

TOPIC 2

TOPIC 3

TOPIC 3

TOPIC 7

TOPIC 4

TOPIC 5same topic
related topic

Citation analysis

Citation network analysis
Improving detection of scientific topic evolution by citation network
Once defined a number k of topics, given a collection of documents,
a topic detection method generates for each topic z a vocabulary distribution
so as to maximize the likelihood of the observed data
Vocabulary distribution

Topic
TOPIC 1

[ Word 1: 0,4

Word 2: 0,2

Word 3: 0,4 ]TOPIC 2

[ Word 1: 0,7

Word 2: 0,1

Word 3: 0,2 ]Given 1 ≥ a > b > 1/k, a pair of topics z(T) and z(T-1)
respectively computed over document collections
at time T and time T-1 is:• equal: p( z(T) | z(T-1) ) > a• similar: b < p( z(T) | z(T-1) ) < a• new: p( z(T) | z(T-1) ) < bp( z(T) | z(T-1) ) equal to sim(z(T), z(T-1) )time T-1

time T

TOPIC 1

TOPIC 4

TOPIC 2

TOPIC 5

TOPIC 2
TOPIC 3

He, Q., Chen, B., Pei, J., Qiu, B., Mitra, P., & Giles, L. (2009, November). Detecting topic evolution in scientific literature: how can citations help?.
In Proceedings of the 18th ACM conference on Information and knowledge management (pp. 957-966). ACM.

Citation analysis

Citation network analysis
Improving detection of scientific topic evolution by citation network
Considering a collection of scientific paper spanning a number of years, in order to track yearby-year topic evolution, we can generate the topic of each year by different approaches:• Time independent topic evolution learning• Accumulative topic evolution learning• Citation-aware topic evolution learning

He, Q., Chen, B., Pei, J., Qiu, B., Mitra, P., & Giles, L. (2009, November). Detecting topic evolution in scientific literature: how can citations help?.
In Proceedings of the 18th ACM conference on Information and knowledge management (pp. 957-966). ACM.

Citation analysis

Citation network analysis
Improving detection of scientific topic evolution by citation network
Citation-aware topic evolution learning
DRAWBACKS• not all citations are equallyimportant (only few can ne related
to the topic of the citing paper)• when historical papers are cited,some out-of-date topic may be
wrongly considered

Inheritance topic model

Cited papers
Citing paper
Inherited part (previous work)
Autonomous part (new ideas)
The autonomous part and the inherited part of a paper (cited papers)
are learned independently
He, Q., Chen, B., Pei, J., Qiu, B., Mitra, P., & Giles, L. (2009, November). Detecting topic evolution in scientific literature: how can citations help?.
In Proceedings of the 18th ACM conference on Information and knowledge management (pp. 957-966). ACM.

Citation analysis

Citation network analysis
Improving detection of scientific topic evolution by citation network

Citation aware

Citation unaware

Evaluation: 650,918 computer and information science papers from CiteSeer from 1993 to 2008
Evolution of 30 topics studied with different approaches
noisy topics
new topics
similar topics
same topics
Large
number of
new topics.
Severalnoisy topics

Time independent topic evolution learning
Produceless new
topics withrespect a
ITM since
old, cited
papers aretreated as
new ones

Citation-aware topic evolution learning

Historical
topics tends
todominate;difficult to
detect newones

Accumulative topic evolution learning

Inheritance topic model

Reach agoodbalancebetweennew andold topics
enabling
detectionof new
ones

He, Q., Chen, B., Pei, J., Qiu, B., Mitra, P., & Giles, L. (2009, November). Detecting topic evolution in scientific literature: how can citations help?.
In Proceedings of the 18th ACM conference on Information and knowledge management (pp. 957-966). ACM.

Citation analysis

Citation network analysis
Co-authorship network
reflects the personal link between scientists
Dataset: 2 co-authorship networks (1991-1998):• maths: 70,975 authors and 70,901 papers• neuroscience: 209,293 authors, 210,750
papersTITLE
Clart, Belman…
Abstract…..

co-authorship strength: 1
Clart

BelmanResults:• degree distribution: power-law, is a scale free network• the average node separation slightly decreases over time: more internal inks areproduced with time (co-authorships) increasing network interconnectivity and decreasing
diameter• the average degree increases with time
• node selection is governed by preferential attachment

Barabâsi, A. L., Jeong, H., Néda, Z., Ravasz, E., Schubert, A., & Vicsek, T. (2002). Evolution of the social network of scientific collaborations. Physica
A: Statistical mechanics and its applications, 311(3), 590-614.

Citation analysis

Citation network analysis
Author co-citation network• 15 journals classified in ISI’s Web of Sciencedealing with Philosophy and History• 12,510 articles dating from 1956
with over 300,000 citations between them

Authors co-citation graph shows intellectual
influences of individual authors, clustering
them by discipline

Source: http://www.scottbot.net/HIAL/wp-content/uploads/2013/06/FullPageMapOfHPS.png

Citation analysis

Citation function
Many research impact and quality indexes are based on citation counts but…

Not all citations are equal!

There are different motivation that could explain why
an author cites other pieces of researchhighlight a positive result

criticize a work express contraryor negative judgments

investigations used as a starting
point for the work describedCitation analysis

Citation function
Many research impact and quality indexes are based on citation counts but…

Not all citations are equal!
The approach presented
by [1] presents several
limitations.
We parse text by meansof the concept extraction
system presented in [1].

We compare our systemwith the concept
extraction performanceof [1].

CRITICISM

USE

[1]

CRITICISM

USE

COMPARISONCRITICISM

Our systems extracts
concepts by implementing
the approach presented
by [1].
The system presented by[1] can’t be easily
adapted to texts fromdifferent domains.
The main drawback of[1] is its impossibility toscale to large systems.

This paper has 6 citations!

CRITICISM
USE

Half of the citations of this paper criticize aspect of the work presented.

Two citations of this paper use the approach / tool presented.

COMPARISON One citation of this paper compares the approach / tool presented.

Citation analysis

Citation function
Many research impact and quality indexes are based on citation counts but…

Not all citations are equal!
In order to understand why a paper is cited
it is fundamental to correctly identify the citation context,that is the text excerpt(s) of the citing paper that explains and motivates the citation
Citation context
In-line citation

The citation context:• may include sentences surrounding the one where the in-line citation occurs• only part of the sentence where the in-line citation occurs can contribute to motivate
the citation

Citation analysis

Citation function
Several annotation schemas have been proposed
to characterize the function of citations
Moravcsik, M. J., & Murugesan, P. (1975). Some results on the function and quality of citations.
Social studies of science, 5(1), 86-92.

4 dimensions for citation characterization
Conceptual
Operational
Organic

Perfunctionary

Evolutionary
Juxtapositional

Confirmative
Negative

If a concept or a theory of the cited paper is used directly or indirectly in the citing
paper in order to lay foundations to build on it or to contribute to the citing paper,then the citation is a conceptual one.
When a concept or theory id referred to as tool… [or] when it borrows mathematicalor physical techniques, results, references, or conclusions from the cited paper.
Those [papers] from which concepts or theories are taken to lay the foundations of
the citing paper, or papers from which certain results (including numerical ones) aretaken to develop the ideas in the citing paper, or papers which help to better
understand certain concepts in the citing paper.
Those [papers] which describe alternative approaches are not utilized in the citing
papers…. references which are used to indicate the fact that a certain method
employed is routine in the literature, and references which merely contribute to thechronological context of the citing paper.
[The paper] provides a concept or theory to build on, or a mathematical techniqueto use, or results of an analysis which is used in the development of the citing paper,or notation used in the citing paper.
[The paper] refers to alternative approaches… [and] refers to other analysis used in
the citing paper only to make comparisons, refers to other works which may help toclarify some ideas but do not contribute to the development of the citing paper, or
refers to a paper only for references given in the latter.
A reference is confirmative if the author of the citing paper considers the paper
referred to as correct.
The author of the citing paper is not certain about the correctness of the cited
paper.

40% of citations
are Perfuctionary(30 Articles in Physicaluse of theory
Review, Published on
use of technical
Theoretical High Energy
Physics from 1968 to 1972)
methodwork is crucially needed forunderstanding of citing article

just a general acknowledgement
own work is an alternative
to cited work
own work is based on the cited work

the work confirm the cited paper
cited paper are criticizedCitation analysis

Citation function
Several annotation schemas have been proposed
to characterize the function of citations
Spiegel-Rösing, I. (1977). Science studies: Bibliometric and content analysis.
Social Studies of Science, 97-113.

13 classes for citation characterization
1

Cited source substantiates a statement or assumption, or points to further information.

2

Cited source is mentioned in the introduction or discussion as part of the history and state of the art of theresearch question under investigation.

3

Cited source contains the data (pertaining to the discipline of the citing article)which are used forcomparative purposes in tables and statistics

4

Cited source contains the data pertaining to the discipline of the citing article) which are used sporadically inthe citing text

5

Cited source is positively evaluated

6

Cited source contains the method used

7

Cited source contains the concepts, definitions, interpretations used (and pertaining to the discipline of theciting article)

8

Cited source is the specific point of departure for the research question investigated.

9

Results of citing article disprove, put into question the data as interpretation of cited source

10

Cited source is negatively evaluated11

Results of citing article prove, verify, substantiate the data or interpretation of cited source.

12

Results of citing article furnish a new interpretation, explanation of the data of the cited source.

13

Cited source contains data and material (from other disciplines than citing article) which is used sporadicallyin the citing text, in tables or statistics.

80% of citations belong
to the first category:
Cited source substantiates astatement or assumption, or
points to further information
0,8% of citations
criticize the cited paper(2,309 citations from Science Studies Vol. 1-4
1971-1974)

Citation analysis

Citation function
Several annotation schemas have been proposed
to characterize the function of citations
Teufel, S., Siddharthan, A., & Tidhar, D. (2009, July). An annotation scheme for citation function.
In Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue (pp. 80-87). ACL

4 top level categories for citation characterization
Weakness
Contrast

Positive

NeutralAuthors point out a weakness in cited work.
Authors make contrast/comparison with cited work (4 categories)
CoCoGM Contrast/Comparison in Goals or Methods (neutral)
CoCoR0 Contrast/Comparison in Results (neutral)
CoCo Unfavourable Contrast/Comparison (current work is better than cited work)
CoCoXY Contrast between 2 cited methods
Authors agree with/make use of/show compatibility or similarity with cited work (6 categories),
PBas author uses cited work as starting point
PUse author uses tools/algorithms/dataPModi author adapts or modi_es tools/algorithms/data
PMot this citation is positive about approach or problem addressed (used to motivate work in current paper)
PSim author's work and cited work are similar
PSup author's work and cited work are compatible/ provide support for each other
Function of citation is either neutral, or weakly signalled, or different from the three functions stated above

Corpus CitRAZ: 26 conference articles – 584 citations from Computation and Language archive
Online at: http://www.cl.cam.ac.uk/~sht25/Project_Index/Citraz_Index.html http://www.cl.cam.ac.uk/~sht25/CFC.html

Citation analysis

Citation function
Several annotation schemas have been proposed
to characterize the function of citations
Teufel, S., Siddharthan, A., & Tidhar, D. (2009, July). An annotation scheme for citation function.
In Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue (pp. 80-87). ACLClassifiers: K-nearest neighbours classifier
Citation features:• grammar (POS-based) with 1762 cue-phrases (Teufel, 1999)• POS-based recognizer for agents and recognizer for actions
that these agents perform (Teufel, 1999)• 892 cue-phrases (about 75 per citation function, identified byannotators)• verb tense and voice• modality (whether or not a main verb is modified by an
auxiliary, and which auxiliary it is)• location of sentence in the whole paper and in the section orparagraph• self citations

Results
4 top classes

All classes

Citation analysis

Citation function
Several annotation schemas have been proposed
to characterize the function of citations
Abu-Jbara, A., Ezra, J., & Radev, D. R. (2013). Purpose and Polarity of Citation: Towards NLP-based
Bibliometrics. In HLT-NAACL (pp. 596-606).

6 classes for citation characterization
Criticism Criticism can be positive or negative. A citing sentence is classified as ”Criticizing” when it
mentions the weakness/strengths of the cited approach, negatively/positively criticizes the cited
approach, negatively/positively evaluates the cited source.
Comparison A citing sentence is classified as ”Comparison” when it compares or contrasts the work in thecited paper to the author’s work. It overlaps with the first category when the citing sentence says
one approach is not as good as the other approach. In this case we use the first category.
Use A citing sentence is classified as ”Use” when the citing paper uses the method, idea or tool of thecited paper.
Substantiation A citing sentence is classified as ”Substantiating” when the results, claims of the citing work
substantiate, verify the cited paper and support each other.
Basis A citing sentence is classified as ”Basis” when the author uses the cited work as starting point or
motivation and extends on the cited work.
Neutral A citing sentence is classified as ”Neutral” when it is a neutral description of the cited work or if it
doesn’t come under any of the above categories.

Corpus: 3,271 citations from ACL Anthology Network Corpus, annotated with respect topolarity and purpose

Citation analysis

Citation function
Several annotation schemas have been proposed
to characterize the function of citations
Abu-Jbara, A., Ezra, J., & Radev, D. R. (2013). Purpose and Polarity of Citation: Towards NLP-based
Bibliometrics. In HLT-NAACL (pp. 596-606).

Classifiers: CRF
Citationcontext
features:(ordered byrelevance)

Results:

Lexical features are more important thanstructural features

Citation analysis

Citation function
Several annotation schemas have been proposed
to characterize the function of citations
Abu-Jbara, A., Ezra, J., & Radev, D. R. (2013). Purpose and Polarity of Citation: Towards NLP-based
Bibliometrics. In HLT-NAACL (pp. 596-606).

Classifiers: SVM with linear kernel
Citationfeatures:(ordered byrelevance)

Results:• Structural features and features characterizing the words
surrounding the citation to classify are the most important• Considering the citation context improves classification ofsubjective categories (exp. Negative)

Citation analysis

Citation function
Several annotation schemas have been proposed
to characterize the function of citations
Athar, A. (2011, June). Sentiment analysis of citations using sentence structure-based features.
In Proceedings of the ACL 2011 student session (pp. 81-87). Association for Computational Linguistics.

Corpus: 8,736 citations from 310 research papers taken from the ACL Anthology, tagged
manually as positive, negative or objective
Online at: http://cl.awaisathar.com/citation-sentiment-corpus/
Features:• unigrams, bigrams and trigrams adding to the lemma also the POS of every token• name of the primary author of the cited paper• science lexicon: 83 polar phrases which have been manually extracted from thedevelopment set of 736 citations• presence of subjectivity clues• number of adjectives, adverbs, pronouns, modals and cardinals• number of negation phrases and valence shifter (Opinion Finder)• dependency tree triples
Dependency tree used to identify the clause of the sentence where the citation occurs
Negated (suffix ‘_neg’) the two lemmas after a negation expression (Opinion Finder)Citation analysis

Citation function
Several annotation schemas have been proposed
to characterize the function of citations
Athar, A. (2011, June). Sentiment analysis of citations using sentence structure-based features.
In Proceedings of the ACL 2011 student session (pp. 81-87). Association for Computational Linguistics.

Corpus: 8,736 citations from 310 research papers taken from the ACL Anthology, tagged
manually as positive, negative or objective
Online at: http://cl.awaisathar.com/citation-sentiment-corpus/
Algorithm: SVM
Result:• n-grams and dependency relations are sufficient to model lexical structure that cancharacterize the polarity of citations• scientific lexicon, word level features, sentence splitting and negation does not help
Athar, A., & Teufel, S. (2012, July). Detection of implicit citations for sentiment detection. In Proceedings ofthe Workshop on Detecting Structure in Scholarly Discourse (pp. 18-26). Association for Computational
Linguistics.  SVM based approach to identify sentences belonging to the citation context. The
information from the citation context improve sentiment analysis performance for citations

Corpus: 852 papers which cite the top 20 target papers. Citation context sentences are
identified and marked as negative, positive, objective/neutral.
Online at: http://cl.awaisathar.com/citation-context-corpus/

Citation analysis

Citation function
Several annotation schemas have been proposed
to characterize the function of citations
Shotton, D. (2010). CiTO, the citation typing ontology. Journal of biomedical semantics, 1(1), 1.

23 properties for citation characterization

Part of the Semantic Publishing and Referencing Ontologies, includes 41 properties forcitation characterization in its most recent version (03/07/2015)

Citation analysis

Citation function
Several annotation schemas have been proposed
to characterize the function of citations
Fisas, B., Ronzano, F., & Saggion, H. (2016). A Multi-Layered Annotated Corpus of Scientific Papers. LREC.

6 top-level purposes and 16 sub-purposes for citation characterization
CRITICISM

WEAKNESSSTRENGTHEVALUATIONOTHERCOMPARISONUSE

SUBSTANTIATIONBASISNEUTRAL

SIMILARITY
DIFFERENCEMETHOD

DATA
TOOL
OTHER

PREVIOUS OWNOTHERS’ WORKFUTURE WORKDESCRIPTIONREFERENCEFOR
MORE INFOCOMMON
PRACTICE
OTHER

A weakness in a cited work may refer to some restriction, its inappropriatenessin the case considered, a requirement, its difficulty, its computational cost, etc.
A strength in a cited work may refer to its easiness of use, its little
computational cost, its speed, its novelty, etc.
Some citations do not only state a strength or a weakness of the cited paper,
but provide the author’s evaluation of the research, by opposing a strengthwith a weakness or by giving his opinion in an explicit (or subtle) way.
If a citation can be considered a CRITICISM, but cannot be included in theprevious sub-purposes, then it should be annotated as CRITICISM_OTHER.
The comparison focuses on the similarities with the author’s work.
The comparison focuses on the differences with the author’s work.
If the author uses the method, technique, or algorithm developed by the cited
paper.
If the author uses the data produced by the cited paper.
If the author uses a tool or software package developed by the cited paper.
A citing sentence is classified as SUBSTANTIATION when the cited paper andthe citing paper support each other.
The author bases the current research on his own previous work.
The author bases the current research on others’ previous work.
Future work can be developed based on the cited work.
If the citation is a neutral description of the cited work.
If the author refers to a work for obtaining more detailed information about aparticular subject.
When other author’s work are cited as common practices in the knowledge
field.
Other reasons for neutral citations.

Dr. Inventor Corpus• 40 Computer Grahics articles• 1,575 citations

Online at:
http://sempub.taln.upf.edu/
dricorpus

Citation analysis

Citation function
Several annotation schemas have been proposed
to characterize the function of citations
Valenzuela, M., Ha, V., & Etzioni, O. (2015, April). Identifying meaningful citations.
In Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence.

Features:Coarse and fine-grained labels forcitation type

Corpus: 465 citations from ACL anthology
Considering direct citations:• direct citations (total and per section)• indirect citations (total and per section)• author overlap• is considered helpful• in table or figure caption (we’re comparing)• number of direct citations over all the direct citations• tf-idf similarity between abstracts (citing / cited)• page rank
• number of citing papers after transitive closure• research field of paper

Algorithms: SVM (RBF k.) and random forest
Evaluation: SVM accuracy: 0,93
and indirect citations:

Most informative feature: direct citations (total and per section)
Followed by: author overlap, is considered helpful, number of direct
citations over all the direct citations, research field of paper

Citation analysis

Citation prediction and recommendation
Given an in-line citation placeholder,predict (recommend) which is the paper that should be cited
In-line citation placeholder[?

]

Search query
It analyses the content
and conceptual structureof scientific articles withan ontology-based
annotation schema theCore Scientific Concept s
scheme (CoreSC).
4/12/2016

Cited paper not retrieved
among the first 10 results
No results retrieved
Cited paper not retrieved
among the first 10 resultsCitation analysis

Citation prediction and recommendation
Given an in-line citation placeholder,predict (recommend) which is the paper that should be cited[

In-line citation placeholder
?
]

[ ? ]

CITATION NETWORKGLOBALCONTEXTLOCAL CONTEXT

Several facets can contribute to identify the best cited paper match:• features local to the citation context (e.g. papers with similar citation contexts)• features global of the whole document (e.g. papers with similar title, abstract,
shared keywords or authors)• user preferences (i.e. publication and citation history of the author, user profile ina bibliography management system)• citation network (i.e. paper-citation matrix)

Citation analysis

Citation prediction and recommendation
Given an in-line citation placeholder,predict (recommend) which is the paper that should be cited
Dealing with citation prediction / recommendation…
• Huge search space: progressive reduction of candidate set (lightweight cited
papers selection methods for a first coarse-grained selection, candidate cited paper
clustering)• Neural models: estimate the probability that, given a word from the citation
context, a document is cited by jointly learning neural representations (embeddings)
of words from citation contexts and cited documents• Citation context identification: models to identify in a paper the candidate
citation contexts and, for each of them, the list of top-n candidate cited papers• Citation motivation: explain why a certain paper should be cited in a given citation
context
Online tool that implements distinct approaches of
document level and context level citation
recommendation - http://refseer.ist.psu.edu/

Citation analysis

Citation-based summarization
The contexts of the citations of a specific paper provide useful• information concerning the core topic of the paper together with
opinions of the research community on the piece of work
Past studies alreadypresented neural-network
approaches to identify
named entities [1].
We parse text by meansof the concept extraction
system presented in [1].

[1]

The approach described in[1] shows several
limitations with respect toscalability.
[1] propose new
approach to spot Named
Entities in legal texts.

“Since citing sentences appear to be somewhat more focused than the abstract and containadditional information not in the abstract, they could be useful as a supplement”
Elkiss, A. et al. 2008). Blind men and elephants: What do citation summaries tell us about a research article?.

“ The inclusion of citation-related information brings to the generation of better summaries.”
Ronzano, F. et a. An Empirical Assessment of Citation Information in Scientific Summarization.

More details and examples of how summarization systems
exploit citation-related information will be presented in the following part of thistutorial dealing with summarization of scientific literature

Citation analysis

Citation graphs
528,682,289 internal citations, each paper in the graph is cited on average 4.17 times
https://academicgraphwe.blob.core.windows.net/graph-2016-02-05/index.html

High-energy physics citation network (2003 KDD cup)
Arxiv HEP-PH (high energy physics phenomenology) citation graph is from the e-print arXiv
January 1993  April 2003 (124 months)
https://academicgraphwe.blob.core.windows.net/graph-2016-02-05/index.html

Patent citation network (2005 KDD cup)
16,522,438 citations, all citations made by patents granted between 1975 and 1999
January 1, 1963  December 30, 1999 (37 years)
https://academicgraphwe.blob.core.windows.net/graph-2016-02-05/index.html

CiteSeer citation network
1,017,457 papers with 10,760,318 citations (Oct. 2013)https://psu.app.box.com/v/refseer (2015 dataset)

ACL Anthology Network
21,212 papers with 110,976 citations (Dec. 2013)
http://clair.eecs.umich.edu/aan/index.php (2013 dataset)

OpenCitations
1,740,050 bibliographic resources with 2,201,568 citations (Dec. 2016)
http://opencitations.net/ (RDF dataset / SPAR ontologies / main crawled source: Europe PMC)

Citation analysis

Conclusions• Citations represent a primary device of scientific literature useful toissue explicit author-created links among publications• Both the network of citations and the textual contents of citation
contexts are exploited in many different tasks including: researchcollaboration analysis, topic analysis and evolution, citation
recommendation, scientific document summarization• Besides citation counts, the (complex task of) characterization of the
purpose of citations can provide deeper insights on the quality ofscientific publications and the feedback of the research community• Citation recommendation system can complement pure scientific
literature search engines in helping to cope with scientific information
overload• A rich collection of citation datasets, including citation networks andcorpora of citations annotated with respect to their sentiment andpurpose, is freely available for further experimentation

SCIENTIFIC DOCUMENTSUMMARIZATION

Outline• Document summarization overview• Summarizing scientific articles– Information extraction and template-based generation– Indicative-informative summaries– Fact-based citation summaries (C-LexRank)– Impact-driven summaries– Non-explicit citations in summaries– Improving summary coherence– Generating state-of-the-art reports• Summarizing patents• Conclusions

Scientific document summarization

Document summarization overview
What is a summary?
A presentation of the substance of a body of material
in a condensed form or by reducing it to its main points; an abstract.
A short text containing the essential information of a document.

What is a summarizer?
An algorithm that selects and presents
the most important content of a document

Scientific document summarization

Document summarization overview

Types of Summaries

Different types of summaries
Summaries should take into account a number of input factors such as
the audience/reader of the summary
one / several
documents

single document summarization

summary
purposeindicative summary (what is text about?)
informative summary (gives information from text)

content

generic summary (main points)
user-focused (answers a query)

input language

monolingual
cross-lingual

formulation

extract (sentences from input documents)
abstract ((quasi) new text )multi-document summarization

Spark Jones (2007)

Scientific document summarization

Document summarization overview
Summarization by sentence extraction
Extract from the input document the subset of sentencesthat contain the most important information• General method to produce “extracts” of size N
a)
b)
c)
d)
e)

S = {}
Associate to each sentence a score and put them in list L
Sort sentences in L by score (in ascending order)
While size of S < N, put next sentence in L in S
Show sentences in S in the order they appear in the original text

• Compression parameter– size in number of words of the summary– compression rate: % of the words or sentences

Scientific document summarization

Document summarization overview
Summarization by sentence extraction: sentence relevance
Function to assess the contribution of a sentence to a summary,developed since the late 50s some still used in the literature
•
•
•
•
•
•
•

Word-distribution measures (Luhn'1958, Nenkova and Vanderwende, 2005)– Term/Word frequency
Document structure (Edmundson, 1969; Lin and Hovy, 1998)– Position of sentence in document– Relation of sentence to title, abstract, keywords, etc.
Presence of specific vocabulary (Paice, 1990)– Formulaic-expressions, key-words, etc.
Centrality information (Barzilay and Elhadad, 1997; Radev et al, 2000; Saggion and Gaizauskas, 2004)– Word-based sentence-sentence relations , centroid– Co-reference
Rhetorical information (Marcu, 1998; Ono et al., 1994)– How argument develops in sentences (more / less central)
Semantic (Saggion and Lapalme, 2002; Jones and Paice, 1993)– Domain/Topic template / Information types to cover in summary
External (Tombros et al, 1998)– Query / User Knowledge

Scientific document summarization

Document summarization overview
Summarization by sentence extraction: superficial techniques
Information about sentence relevance can be provided by:
word/term repetition and document structure
Word/term repetition
Over pre-processed corpora (stemming, stop-words removal), the inverse
document frequency can be used to assess word relevance

relevance (t )  tf (t ) * idf (t )Document structure

NUMDOC
idf (term)  log()
NUMDOC(term)• Position of sentence in document–––

in news give relevance to lead-paragraphin scientific discourse give relevance to sentences under specific section headings
learn optimal positions in a given textual genre• Title / Query sentence relevance––similarity between sentence and document title or user need expressed in a query (cosine, jaccard, etc.)
Information retrieval techniques are useful here

(Edmundson, 1969; Lin and Hovy, 1998)(Tombros et al, 1998)

Scientific document summarization

Document summarization overview
Summarization by sentence extraction: superficial techniquesMore or less fixed vocabulary indicatesthe presence of important information in text
Cue-phrases, indicative expressions, formulaic expressions, etc.
– dictionary with expressions (literal or patterns)• in this {paper | work | article} we....
• {our | my } {results | findings |...} demonstrate ....
– may be organized in categories (results, conclusion, etc.)– expressions may be weighted– expressions might be learnt from corpora(Paice, 1990; Jones and Paice, 1993)

Scientific document summarization

Document summarization overview
Summarization by sentence extraction: feature combinations
No single source of information will produce the best scoring schema:
usually features have to be combined• Features can be combined “a la Edmundson” to score sentences

Weight (S )   .Title (S )   .Cue(S )   .Keyword (S )   .Position (S )• Given training data
– A scoring function can be learnt (regression)– A sentence classification function (extract | non-extract) can be learnt(Edmundson, 1969)(Kupiec et al, 1995)

Scientific document summarization

Document summarization overview
Summarization by sentence extraction: graph-based techniques
Lexical similarity between sentences in the document tell us
about their relevance: text is represented
as a connected structure (unlike other superficial approaches)• Text represented as a graph– vertices are “meaning” units such as words or sentences– edges are connections between units• Inspired by the PageRank algorithm (Page et al. 1998) several
summarization algorithms were proposed– LexRank (Erkan & Radev, 2004)– TextRank (Mihalcea & Tarau, 2004)(Erkan and Radev, 2004; Mihalcea & Tarau, 2004)

Scientific document summarization

Document summarization overview
Summarization by sentence extraction: graph-based techniques
Page Rank
conceived to rank Web pages by relevance• Web pages form a directed graph• PageRank computes the relevance (PageRank score) of each Web page
thanks to the recursive analysis of the connectivity of the complete
network:
PageRank values of page jconnected to i1 d
PR ( j )PR (i)  d *
Number of links
going out of page j
N
C
(
j
)
j 1
N

damping factor

Scientific document summarization

Document summarization overview
Summarization by sentence extraction:
graph-based techniques at document level
Text graphs for summarization seek to associate a weight
to sentences based on an analysis of a text graph
•
•
•
•
•

Sentences are vertex (s1, s2, ..., sn)
There are edges E(sk, sl) connecting sk with slIn(si) is the set of of sentences si such that there is an edge E(si, sj)Out(si) is the set of of sentences si such that there is an edge E(sj, si)
Graph generally undirected but could be directed if text order is taken
into account (si connects with sj only if i < j)

Scientific document summarization

Document summarization overview
Summarization by sentence extraction:
graph-based techniques at document level
LexRank and TextRank score sentences based on
an iterative procedure and weighting mechanisms similar to PageRank
Sentence similarity:• LexRank uses cosine similarity tocompare sentences• TextRank uses a kind of jaccard
coefficient Parameters (d, w(s), etc.) need to be
estimated Scores computed iteratively untilconvergence

LexRankw ( si ) d (1  d ) *N
sjIn ( si )sim( si,sj )* w ( sj ) sim( sk ,sj )skOut ( sj )TextRankw ( si )  (1  d )  d *
sjIn ( si )wj, k wj,i* w ( sj ) wj,kskOut ( sj )|sjsk|log(|sj|) log(|sk|)

Scientific document summarization

Document summarization overview
Summarization by sentence extraction:other Information Retrieval Techniques
The vector space model: paragraph is represented as a vector of terms and weights
and similitude between paragraphs is computed using inner productBsim>thrCAD
isim<=thr
D

B=1

C=2

degreeof node (di1,...,din)sim ( D , D )   d . d
i
j
ik jk

A=3

F
complete
graph

E

D=1
F=2
graph after
pruning

E=3Paragraph selection based ongraph search techniques:
best first, etc.
(Salton et al. 1997)

Scientific document summarization

Document summarization overview
Evaluation of automated summaries
Human assessment of content: check with source document orwith ideal summaries, and text quality: grammaticality, coherence, etc.

• Humans identify units in ideal
summaries and units in automatic
summaries• Units are matched and their overlap
assessed• Text quality assessed by means ofquestionnaires• Human evaluation is very expensive

EVALUATION INTERFACE

DUC evaluations

Scientific document summarization

Document summarization overview
Evaluation of automated summaries
ROUGE: Recall-Oriented Understudy for Gisting Evaluation
Measures content quality of a summary by comparison with ideal(s) summaries
based on n-gram counting
 countmatch(n -gram)
S{Refs} n-gramS
ROUGE-n  count(n-gram)
S{Refs} n-gram S

Other ROUGE metrics:• ROUGE-L: Based on longest common subsequence• ROUGE-W: weighted longest common subsequence, favours consecutive
matches• ROUGE-S: Skip-bigram recall metric• Arbitrary in-sequence bigrams are computed• ROUGE-SU adds unigrams to ROUGE-S(Lin, 2004)

Scientific document summarization

Document summarization overview
Evaluation of automated summaries
Pyramid Score
based on the distribution of content units (Sus) in a set of ideal summaries,
similar content units are grouped together• each SCU in tier Ti in the pyramid has
weight i• the best summary is one which contains
all units of level n, then all units from n1,…
• if Di is the number of SCU in a summary
which appear in Ti for summary is:nD   i  Dii 1

• X is the number of units in the summary

Score  D / Max

w=n
w=n-1
...
w=1
PYRAMID FROM CONTENTUNITS IN IDEAL SUMMARIESnnMax   i | Ti |  j  ( X   | Ti |)i  j 1n

i  j 1j  max ( | Tt | X )it i(Nenkova, Passoneau, McKeown, 2007)

Scientific document summarization

Document summarization overview
Summarization tools
Availability of summarization tools: no reinvent the wheel,
allow comparison, provide baselines, etc.
•

MEAD

– publicly available toolkit for multi-lingual summarization and evaluation
http://www.summarization.com/mead/– implements different algorithms: position-based, centroid-based, it*idf, querybased summarization– implements evaluation methods: co-selection, relative-utility, content-based
metrics•

SUMMA– publicly available: http://www.taln.upf.edu/pages/summa.upf/– JAVA library to implement summarization systems– Statistical analysis of documents– Several relevance features and sentence scoring mechanisms available– Multilingual, Multi-document– Implements ROUGE and BLEU summary evaluation(Radev et al. 2004; Saggion, 2008, 2014)

Scientific document summarization

Summarizing Scientific Articles
Scientific information overload is going to be more and more problematic
Summarization can help scientists and other interested partners
to access text collections by means of automated summaries• Scientist and other interested parties nowadays face the problem of scientificinformation overloado PubMed contains more than 24M papers, Elsevier’ Scopus over 57M, while Thomson
Reuther’s ISIWeb of Knowledge more than 90M….
o Current estimates indicate that a research paper is published every 13 seconds• Scientific text summarization was the first summarization application
domain!
o Summarization of scientific documents has been addressed using traditional
relevance features, classification or generic/domain specific scientific information

Scientific document summarization

Summarizing Scientific Articles
Summarization approaches to the Scientific Document
Summarization has to be adapted to the peculiarities of the scientific discourse:
length, document structure, terminology, citations, rhetorical organization, etc.

• Rhetorical classification of sentences– Extracting sentences likely to contain semantic information on objectives,goal, own contributions, etc.  classification• Extracting scientific specific information– Concept based abstracting  information extraction + template-based
generation• Extraction generic scientific information– Extracting sentences based on generic information types  information
extraction + shallow generation• Relying on the opinion of the scientific community to summarize– Taking advantage of “citation sentences” to summarize a article– Impact-based summarization  uses the source document– Citation-based summarization  uses the “citation sentences”

Scientific document summarization

Summarizing Scientific Articles
Domain Specific Summarization
using Information Extraction and Template-based Generation
Summaries in specific scientific domains report information
on specific and stereotypical domain concepts.
The way the information is presented in the summary is also predictable.
• Example in the are of crop husbandry
SPECIES (what is studied); CULTIVAR (the variety that is studied); HIGH-LEVEL-PROPERTY(the property studied: growth); PEST (a pest that attacks the species); AGENT (thechemical/bio agent used to control the pest); etc.

• Method– Weighted patterns (PEST is a ? pest of SPECIES) are applied to the text to instantiate
concepts– Matched strings are analyzed and weighted to extract final values– Summaries are generated using the strings
This paper studies the effect of [AGENT] on the [HLP] of [SPECIES] OR
this paper studies the effect of [METHOD] on the [HLP] of [SPECIES]
when it is infested by [PEST]…

This paper studies the effect of G. pallida on the yield of potato. An
experiment in 1985 and 1986 at York was undertaken.
(Oakes and Paice, 2000)

Scientific document summarization

Summarizing Scientific Articles
Generating Indicative-Informative Summaries of Technical Articles
Generate a brief indicative summary of the main topics discussed in the paperand expand the topics with useful information about the topics.
Modelling general scientific information
Article Title: Features 3D scanning systems for rapid prototyping (97 sentences)

Indicative Summary: Describes two non-contact scanning systems, REVERSA andModelMaker
Topics: CADAM system; ModelMaker; REVERSA; standard dual view system; system
Expanding Topics: REVERSA and ModelMaker
REVERSA is a dual viewpoint non-contact laser scanner which comes complete withscanning software and data manipulation tools.
The ModelMaker scanning system is a combination of a 3D laser stripe sensor,
6DOF position localizer and a PC...
ModelMaker can simply be retrofitted to existing arms providing the benefits of a
portable CMM with dense depth data sets...
(Saggion and Lapalme, 2002)

Scientific document summarization

Summarizing Scientific Articles
Generating Indicative-Informative Summaries of Technical Articles
Generate a brief indicative summary of the main topics discussed in the paperand expand the topics with useful information about the topics.
Modelling general scientific information– Concepts = author, section, problem, solution, limitations, etc.
– Relations = present topic, define, elaborate, conclude, etc.
– Patterns for interpretation = dictionary elements + syntax + lexical elements– Templates for generation

DICTIONARY• Based on a linguistic & conceptual model of the scientific article• Text analysis: POS tagging + pattern-matching• Sentence scoring: titles (main + section headings) & verb-argument (noun phrase)scoring guide sentence selection• Text generation: order information based on dictionary categories, sentence
fusion, verb transformation (personal (e.g. We describe X)  impersonal (e.g.
Describes X) etc.
• Evaluation: text classification, comparison against author abstract, comparison
against ideal summaries– Improves over all baselinesScientific document summarization

Summarizing Scientific Articles
Citation-based summarization
New forms of scientific summarization are based on citation networks:a paper is summarized taking into account the opinions or viewsof the scientific community has about a paper.
BioSumm 2014 & SciSumm 2016

In 2014 National Institute for Standards & Technology (NIST) proposed the
BioSumm Shared Task to promote the development of methods forsummarizing scientific articles
Writing surveys / overviews of developments in biomedicine (or any otherfield) requires the analysis of considerable number of scientific publications– Author abstracts do not provide information on the lasting influences of a work
– Citations do not provide enough context from the cited paper

https://tac.nist.gov//2014/BiomedSumm/

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: BioSumm 2014 & SciSumm 2016
Structure of the dataset- Ech Collection is made of
1 Reference Paper + 10
Citing Papers- For each Collection, four
‘250-words’ humanwritten summaries of thereference paper

AIM

METHODFor each Collection, three tasks are proposed:• Task 1A: identify text spans being cited• Task 1B: identify citation facet• Task 2: create a community-based summary

BioSumm 2014: 30 Coll. fortraining and 20 Coll. forevaluation
SciSumm 2016: 10 Coll. fortraining and 10 Coll. fordevelopment and 10 Coll. For
evaluation

https://tac.nist.gov//2014/BiomedSumm/
http://wing.comp.nus.edu.sg/birndl-jcdl2016/

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization:fact-based citation summaries (C-LexRank)

DataSet– ACL Anthology Network (ANN)– 5 clusters of documents extracted (each on a given topic,
matched with specific keyword e.g. dependency parsing)– Each cluster 5 different documents, each with citations– For each paper a “citation summary” was created based on
the sentences “citing” the paper– Annotators were asked to extract facts from the citation
summary (keywords or phrases) that represent the content(Qazvinian and Radev, 2008)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization:fact-based citation summaries (C-LexRank)• Some annotators agreed on some facts(Czech DP, lexical rules, etc.)• Some annotators found unique facts like:generative model• A {0,1} matrix can be created which indicates
which facts are covered by which citation
sentences
Facts for paper “A Statistical Parser for Czech”
with 54 citations

The summary of a paper is created by: creating a Citation Summary Network andselecting citing sentences that cover a varied set of relevant facts••

Sentences well connected in the network (high similarity) should represent shared facts
Different sentence similarity measures are compared to decide on the most appropriate(evaluated on paper “A Statistical Parser for Czech”)(Qazvinian and Radev, 2008)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization:fact-based citation summaries (C-LexRank)• Network-based clustering is applied to group sentences which share many
common facts by a hierarchical agglomerative clustering algorithm• Evaluation is carried out computing purity where K are the clusters and C are
the classes (the facts!)
1
purity ( K , C ) 

max | k  c |

N
i

i

j

j

Selection of summary sentence from clusters:
1.
2.

Cluster Round-Robin (C-RR): Sort the clusters by their size and extract one sentencefrom each cluster, then extract more sentences until compression is reached.
Cluster LexRank (C-lexrank): Inside each cluster LexRank is applied to scoresentences. The most salient sentences from each cluster are selected.

Baseline methods:– Random summary– LexRank (without initial clustering)

The best performing system (according to pyramid scores) overall is C-lexrank,followed by Lexrank, and then by C-RR

(Qazvinian and Radev, 2008)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: Impact-driven summaries
Summarizing the impact of a scientific publication: “... the impact of a paper has to be
judged based on the consent of the research community...”Impact-based summary: a set of sentences from the paper thatcan reflect the impact of the paper
Instead of using citation sentences the approach uses sentencesfrom the paper (to avoid including content which is notdirectly related to the paper to summarize)(Mei and Zhai, 2008)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: Impact-driven summaries
Summarizing the impact of a scientific publication: “... the impact of a paper has to be
judged based on the consent of the research community...”• Citation context: widow of sentences around the citation• Approach– Construct a representation of the impact I of a documentd based on d and the citation context C– Develop a scoring function Score(.) to rank sentences of dreflecting I• The approach can be seen as a retrieval problem: sentencesof d are documents and I is a query: retrieve sentences
matching I(Mei and Zhai, 2008)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: Impact-driven summaries
Summarizing the impact of a scientific publication: “... the impact of a paper has to be
judged based on the consent of the research community...”• Impact Language Model: an unigram model for I (the impact),
based on both (i) the document d to summarize and (ii) the
citation context C• probabilities for words in d are estimated using relative
frequencies• probabilities for words in C are estimated from relative
frequencies, citation paper impact (based on pagerank), and position of the sentence with respect to thecitation marker(Mei and Zhai, 2008)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: Impact-driven summaries
Summarizing the impact of a scientific publication: “... the impact of a paper has to be
judged based on the consent of the research community...”The scoring function of paper sentences (score(s)) is based on
Kullback-Leibler (KL) divergence– V is the set of words in the vocabulary–  I is the Impact Language Model–  s is the sentence language modelscore( s)   D(I || s )   p( w | s ) log( p( w | s ))   p( w | I ) log( p( w | I ))
wV

If

wV

 s and  I are very close, the KL-divergence would be small and Score(s) would be high(Mei and Zhai, 2008)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: Impact-driven summaries
Summarizing the impact of a scientific publication: “... the impact of a paper has to be
judged based on the consent of the research community...”• Data– SIGIR papers from 1978 to 2005 (1,303 papers)– Citation contexts extracted (5 sentences): sentence with citation
marker -2,+2– Only papers with at least 20 citations are considered (14 papers)– Experts assessed each sentence in the paper and decided if it
covers “influential” content as indicated in the citation contexts– The influential sentences are considered as the gold standard
summaries for evaluation(Mei and Zhai, 2008)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: Impact-driven summaries
Summarizing the impact of a scientific publication: “... the impact of a paper has to be
judged based on the consent of the research community...”• Evaluation– ROUGE-1 and ROUGE-L are used to compare automatic
summaries with gold summaries– Baselines: LEAD, MEAD, MEAD + Citation Context– KL-divergence summarizer outperforms all baselines– Parameters such as authority and proximity of sentence tocitation have an impact on the results(Mei and Zhai, 2008)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: finding non-explicit citations
Finding sentences that potentially contain useful information about a cited source,
but not explicitly cite it – i.e. expanding explicit citations to citation contexts• A limitation of citation-based approaches to scientificsummarizations is the use of explicit citation information– Explicit citation:
This approach is one of those described in Eisner (1996)
Offers very little information about Eisner’s paper• Implicit or non-explicit citation sentences may contain useful
information on the cited paper– ....the parser searches for the best parse for the sentence.
This approach is one of those described in Eisner (1996)
Non-explicit citation(Qazvinian and Radev, 2010)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: finding non-explicit citations• Method:
Construction of a graphical model based on Markov Random
Fields (MRF) from the sentences in the document• Evaluation with respect to gold-standard (F-measure)• Evaluation with respect to extrinsic citation-basedsummarization (using pyramid method)(Qazvinian and Radev, 2010)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: finding non-explicit citations
HIDDEN NODESOBSERVED NODES

S1

S2

Sk

Sn-1

Sn

• For each sentence Si, ci represents an event of being a non-explicit citation• Observed nodes represent measurable information about sentences (sentence
content)• Hidden nodes represents the state of the sentence (non-explicit citation state) –modelled with a potential function i (ci ) or probability of being at state ci• Relation between neighbouring sentences represented with a weighted edge:
compatibility function ij(cj | ci ) represents i believes about j(Qazvinian and Radev, 2010)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: finding non-explicit citations
HIDDEN NODESOBSERVED NODES

S1

S2• Assumptions about compatibility

Sk

Sn-1

Sn

– if sentence is not “non-explicit citation” can not “say” much about othersentences– If sentence is a “non-explicit citation”, it can say something aboutneighbouring sentences

ij (cj | ci )  0.5
ij (cj | ci )  Sij 11  e  cosine(i, j)(Qazvinian and Radev, 2010)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: finding non-explicit citations• Computation of values of hidden variables(probabilities of being ci ) is carried out with Belief
Propagation (messages are sent from one sentence tothe others)mij(cj )  P(ci )ij(cj | ci )  mki (ci )  P(ci )ij(cj | ci )  mki (ci )kne( i ) \ jkne( i ) \ jmij(cj )  P(ci )ij(cj | ci )  mki (ci )  P(ci )ij(cj | ci )  mki (ci )kne( i ) \ jkne( i ) \ j• ne(i ) indicates the neighbours of sentence i in the
network• Messages mij are initially 0.5 and are updated
through iteration (they are considered probabilities)(Qazvinian and Radev, 2010)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: finding non-explicit citations• Final believe values (i.e. probabilities) are computes
with the final values as (with k a normalization factor):b(ci )  ki (ci )  mij(ci )jne( i )b(ci )  ki(ci )  mij(ci )jne( i )• Choosing a threshold for deciding if the sentence is a
non-explicit citation• The values of i(ci ) are computed with a normalized
linear formula that combines– a binary value for the presence of explicit citation– a binary value for the presence of certain patterns– the “cosine” similarity of the sentence to the cited paper(Qazvinian and Radev, 2010)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: finding non-explicit citations• Evaluation dataset: set of 10 documents from the ACL
anthology + their implicit citation sentences (humanannotated)• Different network configurations explored (BP1: oneneighbour, BP4: 4 neighbours, BPn: all neighbours)• Baseline systems: B1 selects previous/following sentence if
similarity greater than thr. B2 selects any neighbouring
sentences (in a 4-sentence window) matching a pattern, SVM(with 3 features) a trained model using all docs minus one fortraining(Qazvinian and Radev, 2010)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: finding non-explicit citations• Considering 4 sentences as thecontext of influence provides thebest results• Network-based approach better
than sentence classification• Using implicit citations forsummary generation improves
results (pyramid) that use only
explicit citations

F-score for identifying implicit citations(Qazvinian and Radev, 2010)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: improving coherence
Citations may produce incoherent summaries,
so further processing might be needed• Problems– Citations may contain material referring to other articles– Including irrelevant material will waste space– Ordering sentences in a citation-based summary may affectcoherence/cohesion (the order may not be logical)• Approach– Filtering out unsuitable citation sentences and removing irrelevant
parts from citation sentences– Selecting best citation sentences (covering relevant aspects of the cited
paper)– Post-process the sentences to enhance the summary(Abu-Jbara and Radev, 2011)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: improving coherence• Finding the scope of the reference is achieved by parsing the sentence andextracting the smallest sub-tree rooted S (sentence) which contains the reference• Sentences are classified as suitable or unsuitable using supervised learning(SVM)• Sentences are:– classified as Background, Problem, Method, Result and Limitation– in each class, clustered by a hierarchical agglomerative community finding algorithm– in each cluster sentences are weighted using the LexRank algorithm• Sentences are selected based on: their category (B, P, M, R, L), size of the cluster
they belong to, and LexRank values• Finally the sentences are post-processed, the citation marker can be removed ortransformed into a pronominal reference (he/she/they)– A trainable system is used to decide the appropriate transformation(Abu-Jbara and Radev, 2011)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: improving coherence• Dataset– 55 papers from the ANN corpus are used
– Citation sentences are annotated with labels: Background, Problem,
Method, Result, Limitation, Unsuitable– Citation markers are annotated with replace, remove, or keep
• Evaluation with ROUGE-L– (5 sentences long) were created for 30 papers out of citation sentences– Baselines used: MEAD with default settings, LexRank, citation-basedsummaries (QV08 system previous slides)– System outperforms all baselines in ROUGE-L (sentence filtering having ahigh impact in the model)– System has more coherent summaries than QV08
(Abu-Jbara and Radev, 2011)

Scientific document summarization

Summarizing Scientific Articles
Citation-based summarization: generating state-of-the-art reports
A state of the art report or a survey of a scientific topic can be considered
an instance of multi-document summarization• Automatic Related Work Summarization– Combines sentences from target paper and sentences from cited papers– Topic tree of the state of the related work section (manually constructed)– Sentences attached to topic based on how well it reflects topic and a mix ofauthor and reference papers are selected for each topic• Using Keywords– Given an initial query (“Word Sense Disambiguation”) a precise search for paper
based on matching on titles and abstracts is carried out and then expandedwith papers citing/cited by the initial papers– For each paper citing sentences are used to generate the surveys– Sentences are selected based on different methods: Centroid, LexRank, and CLexRank (clustering)– Pyramid scores show that best system is LexRank(Hoang and Kan, 2010) (Jha et al., 2013)

Patent summarization

Summarizing Patents
Legal documents (US Const.)“Art. 1, Sec. 8. The Congress shall have power . . . To promote the progress of science anduseful arts, by securing for limited times to authors and inventors the exclusive right to theirrespective writings and discoveries.”
Objectives: intellectual property protection, secures markets, competitor control , etc.
Once a patent is granted knowledge is disclosed and transferred to society
Characteristics:•
long documents•
long sentences (>500W sentences are common in some sections – claims)•
complicated sentence structure (many embedded clauses and coordination)•complicated terminology (specific classification codes, technical terms, use of peculiar references, e.g. said
device, references to other patents, biblio. references, figures, drawings, measurements, chemical compounds, etc.)•
peculiar document structure (title, field of invention, abstract, prior art, claims, description, drawings, etc.)
Patent Overload!
–

the European Patent Office (EPO) : 90M patents 750 patent applications each day–

Derwent World Patent Index: 33M patents–

Google: 87M patents

Manually creating summaries for patents is unfeasible

CLAIMS SECTION OF PATENT
problem for sentence
extraction methods sincesome sentences would beoverweighed by traditional
methods
Long and complicated sentences

claim structure•

Content peculiarities– claim vocabulary is very vague and abstract toobfuscate the message: [device for recording a
digital information signal in an information
track on a magnetic record carrier] instead oftape recorder– author abstract is also written in vague terms– noun phrases are extremely long: [device forrecording …. on a magnetic record carrier]– a description section elaborates the claims in
more concrete terms

Patent summarization

Summarizing Patents
Trainable patent summarization
Scoring and selection of sub-sentential units and generation of the summary
based on text generation techniques:
use both claims and description for selection of information• Patent processing and text analysis– Segmentation of patents in text segments– Segmentation of each sentence– Mention (noun-phrases) identification based on chunking– Coreference resolution (adaptation of Stanford Coref. Resolution)– Lexical chain computation (coreference, part-whole, setmembership, etc.)– Matching/aligning claim segments with their descriptions(Codina-Filbá et al, 2015)

Patent summarization

Summarizing Patents
Trainable patent summarization
Segments are scored based on a number of classical and patent specific features•••

Mention/Lex. Chains features (aggregated and normalized in sentences)– mention frequency– coreference chain length score– meronym and hyperonym chain score– claim relevance structure
Segment features– best and second best similarities of segment with claims– length– is segment in claim?
– segment mentions the patent invention?
Classical features– similarity to author summary– similarity to patent title– similarity to claims– tf*idf score for segment based on statistics for claims, description, abstract(Codina-Filbá et al, 2015)

Patent summarization

Summarizing Patents
Trainable patent summarization
Scorer is implemented with linear regression
where weights are adjusted with training data

• Data– 26,498 sentences scored based on
their similarity to an ideal abstract• WEKA linear regression (LR) used tolearn optimal weights• SUMMA used to implement features,compute, and select segments
Good predictive power of the LR model
Most features correlate well with relevancePatent summarization

Summarizing Patents
Trainable patent summarization
Generate an abstract based on the content units selected• Complete units and adjust grammar• Remove parts of segments or drop segments• Increase cohesion:• Content evaluation: mention recall, precision, f-measure  systemoutperforms LexRank, Centroid, and LEAD
• Human content evaluation: similar results

Scientific document summarization

Conclusions• The information provided by citations is essential to support andimprove the generation of summaries of scientific documents• Several kinds of information can be included in a summary of ascientific publication: the relevant contents of the paper, which partsof the papers had more impact on the research community, thefeedback of the research community concerning a specific article, etc.
• Multi-document summarization is useful to help the creation ofstate-of-the-art reports• General purpose metrics and techniques have to be adapted inorder to assess scientific content

Scientific document summarization

Cited works (1/3)
Amjad Abu-Jbara, Dragomir R. Radev. Coherent Citation-Based Summarization of Scientific Papers. ACL 2011: 500-509
Anastasios Tombros and Mark Sanderson. 1998. Advantages of query biased summaries in information retrieval. In
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information
retrieval (SIGIR '98).
Ani Nenkova, Rebecca Passonneau, and Kathleen McKeown. 2007. The Pyramid Method: Incorporating human content
selection variation in summarization evaluation. ACM Trans. Speech Lang. Process. 4, 2, Article 4 (May 2007).
Barzilay, R. and Elhadad, M. 1997. Using lexical chains for text summarization. Advances in Automatic Text
Summarization 1999.
Cong Duy Vu Hoang and Min-Yen Kan. 2010. Towards automated related work summarization. In Proceedings of the
23rd International Conference on Computational Linguistics: Posters (COLING '10). Association for Computational
Linguistics, Stroudsburg, PA, USA, 427-435.
Daniel C. Marcu. 1998. The Rhetorical Parsing, Summarization, and Generation of Natural Language Texts. Ph.D.
Dissertation. University of Toronto, Toronto, Ont., Canada, Canada.
Gerard Salton, Amit Singhal, Mandar Mitra, and Chris Buckley. 1997. Automatic text structuring and summarization. Inf.
Process. Manage. 33, 2 (March 1997), 193-207.
Günes Erkan and Dragomir R. Radev. 2004. LexRank: graph-based lexical centrality as salience in text summarization. J.
Artif. Int. Res. 22, 1 (December 2004), 457-479.
H. P. Edmundson. 1969. New Methods in Automatic Extracting. J. ACM 16, 2 (April 1969), 264-285.
H. P. Luhn. 1958. The automatic creation of literature abstracts. IBM J. Res. Dev. 2, 2 (April 1958), 159-165.
Horacio Saggion and Guy Lapalme. 2002. Generating indicative-informative summaries with sumUM. Comput. Linguist.
28, 4 (December 2002), 497-526.
Horacio Saggion. Creating Summarization Systems with SUMMA. LREC 2014: 4157-4163
Horacio Saggion. SUMMA. A Robust and Adaptable Summarization Tool. TAL 49(2): 103-125 (2008)

Scientific document summarization

Cited works (2/3)
Joan Codina-Filbà, Nadjet Bouayad-Agha, Alicia Burga, Gerard Casamayor, Simon Mille, Andreas Müller, Horacio
Saggion, Leo Wanner, Using genre-specific features for patent summaries, Information Processing & Management,
Volume 53, Issue 1, January 2017, Pages 151-174, ISSN 0306-4573
Jones, P.A. and Paice, C.D. A ‘Select and Generate’ Approach to Automatic Abstracting. 14th Information Retrieval
Colloquium. 1993.
Julian Kupiec, Jan Pedersen, and Francine Chen. 1995. A trainable document summarizer. In Proceedings of the 18th
annual international ACM SIGIR conference on Research and development in information retrieval (SIGIR '95).
Karen Spärck Jones. 2007. Automatic summarising: The state of the art. Inf. Process. Manage. 43, 6 (November 2007),1449-1481.
Lin, CY. and Hovy, E.. Identifying Topics by Position. ACL 1997.
Lin, CY. ROUGE: A Package for Automatic Evaluation of summaries. ACL Summarization Workshop 2004
Mead et al. MEAD - a platform for multidocument multilingual text summarization. In LREC 2004.
Nenkova, A. Vanderwende, L. The impact of frequency on summarization. Microsoft Research, Redmond, Washington,
Tech. Rep. MSR-TR-2005-101.
Oakes, M. and Paice, C.D. Term extraction for automatic abstracting.
Page, Lawrence and Brin, Sergey and Motwani, Rajeev and Winograd, Terry (1999) The PageRank Citation Ranking:
Bringing Order to the Web. Technical Report. Stanford InfoLab.
Paice, C.D. Constructing Literarure Abstracts by Computers: Techniques and Prospects. IP&M 1990.
Qiaozhu Mei, ChengXiang Zhai. Generating impact-based summaries for scientific literature. ACL-08: HLT - 46th Annual
Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of theConference.
Rada Mihalcea, Paul Tarau. 2004. TextRank: Bringing Order into Texts. Proceedings of EMNLP 2004, pages 404–411,
Barcelona, Spain. Association for Computational Linguistics.

Scientific document summarization

Cited works (3/3)
Rahul Jha, Reed Coke, and Dragomir Radev. 2015. Surveyor: a system for generating coherent survey articles forscientific topics. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI'15). AAAI Press
2167-2173.
Seiji Miike, Etsuo Itoh, Kenji Ono, and Kazuo Sumita. 1994. A full-text retrieval system with a dynamic abstract
generation function. In Proceedings of the 17th annual international ACM SIGIR conference on Research anddevelopment in information retrieval (SIGIR '94)
Vahed Qazvinian and Dragomir R. Radev. 2008. Scientific paper summarization using citation summary networks. In
Proceedings of the 22nd International Conference on Computational Linguistics - Volume 1 (COLING '08), Vol. 1.
Association for Computational Linguistics, Stroudsburg, PA, USA, 689-696.
Vahed Qazvinian and Dragomir R. Radev. 2010. Identifying non-explicit citing sentences for citation-basedsummarization. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL '10).
Association for Computational Linguistics, Stroudsburg, PA, USA, 555-564.

CHALLENGES, DATASETSAND ARCHITECTURES

Outline• Scientific Literature Mining Challenges• Datasets and tools• Structured / semantic publication formats• Scholary literature architectures

Challenges, datasets and architectures

Scientific Literature Mining Challenges
Several challenges have been organized
to explore how we can take advantage of scientific literature
to automatically carry out specific text analysis tasks

SemEval-2010 Task 5 : Automatic Keyphrase
Extraction from Scientific Articles
SemEval-2017 Task 10: ScientceIE - Extracting Keyphrases
and Relations from Scientific Publications

Challenges, datasets and architectures

Scientific Literature Mining Challenges
Issue: author-name ambiguity (authors that publish with several name variations,different authors sharing the same name)

Dataset (from Microsoft Academic Graph):•••

250k (authors + affiliation)
2,5M (papers + conference / journal info)
Author/paper pairs (to evaluate if correct or not) ground truth on manual corrections of Microsoft
Academic Graph

Two Tracks:
1. Author-Paper identification: for each author papers that she has written
None of these approaches
2. Author disambiguation challenge: group duplicated author
directly scales sufficiently
names referring to the same author
well for use on the entire
Microsoft Academic Searchauthor and publication data

Track 1: extensive feature engineering on the MAG and binary classifier of paper-author pairs
Track 2: multi step approach for string name processing and matching
Roy, S. B., De Cock, M., Mandava, V., Savanna, S., Dalessandro, B., Perlich, C., ... & Hamner, B. (2013, August). The microsoft academic searchdataset and kdd cup 2013. In Proceedings of the 2013 KDD cup 2013 workshop (p. 1). ACM.

Challenges, datasets and architectures

Scientific Literature Mining Challenges

Issue: given a research field, rank the relevance of institutions
Dataset: any dataset publicly available online together with the Microsoft
Academic Graph can be used
Track:
Rank a set of institutions with respect to the number of full research papers
they get accepted in 2016 conferences: SIGIR, SIGMOD, SIGCOMM, KDD, ICML,
FSE, MobiCom, MM
Great predictive power of the participation of the institution
in the past editions of the conference

Roy, S. B., De Cock, M., Mandava, V., Savanna, S., Dalessandro, B., Perlich, C., ... & Hamner, B. (2013, August). The microsoft academic searchdataset and kdd cup 2013. In Proceedings of the 2013 KDD cup 2013 workshop (p. 1). ACM.

Challenges, datasets and architectures

Scientific Literature Mining Challenges

Issue: assess the query-independent importance of scholarly articles
Dataset: any dataset publicly available online together with the Microsoft
Academic Graph can be used
Track:
Generate static ranking of papers with respect to their relevance

Iterative solution that refine citation-graph paper ranking measures by means of the information
concerning paper authors and venue of publication

Wade, A. D., Wang, K., Sun, Y., Gulli, A. 2016. WSDM Cup 2016 – Entity Ranking Challenge. Proceedings of the 9th ACM Conference on Web
Search and Data Mining, San Francisco, CA.

Challenges, datasets and architectures

Scientific Literature Mining Challenges
2014 / 2015 / 2016 – in conjunction with Extended Semantic Web Conference

Issue: automatically generate semantic publishing RDF datasets from bothconference proceedings and papers
Dataset:CEUR-WS Web proceedings (task 1), CEUR-WS papers in PDF format (task 2), RDF
semantic publishing datasets (task 3)
Tasks (2016):
1. Extract information from CEUR-WS online proceeding (HTML) (what workshop series
a workshop is part of, affiliations of editors, exact date of workshop and of proceedings publication, distinction between invitedand contributed papers)

2.

Extract information from PDF files of papers from CEUR-WS (author, affiliations andcountries, captions of tables and figures, funding agencies, EU projects, sections)

3.

Interlink semantic publishing RDF datasets

Dimou, A., Di Iorio, A., Lange, C., & Vahdati, S. (2016, May). Semantic Publishing Challenge–Assessing the Quality of Scientific Output in Its
Ecosystem. In Semantic Web Evaluation Challenge (pp. 243-254). Springer International Publishing.

Challenges, datasets and architectures

Scientific Literature Mining Challenges

Task 1

Linguisti &structural
analyzer

Semantic
annotator

Annotation
sanitizer

Task 2

2014 / 2015 / 2016 – in conjunction with Extended Semantic Web Conference

PDF-to-Text
converter

Linguistic &structural
analyzer

BiblioparserRDF data model

External
resources
linker

RDF
generator

Spot ontology /founding bodymentions

RDF
generator

SPARQL evaluation query

Number and avg. page numberof papers in each proceeding

Ronzano, F., Fisas, B., del Bosque, G. C., & Saggion, H. (2015, May). On the automated generation of scholarly publishing linked datasets: the caseof CEUR-WS proceedings. In Semantic Web Evaluation Challenge (pp. 177-188). Springer International Publishing.

Scientific Literature
Mining Challenges

COLLECTION 1
COLLECTION 2
COLLECTION 20

Reference Paper
Reference Paper
Citing Paper 1 CITATION CONTEXTTITLE: A Genetic Screen Implicates miRNA-372 and miRNA-373
Citing Paper 1 CITATION CONTEXTTITLE:A
Genetic
ScreenImplicatesmiRNA-372
andmiRNA-373
AUTH.: Voorhoeve, M., le Sage,Reference
C., Schrier, M.,Paper
et al. – YEAR: 2006
However, the contributions of miRAUTH.: Voorhoeve, M., le Sage, C., Schrier, M., et al. – YEAR: 2006…identified.
CitingPaper the1 contributions…identified.
However,miR10b, miR-21,
and miR-373/520carenot easilyof
TITLE: A Genetic Screen Implicates miRNA-372 and miRNA-373
CITATION
CONTEXTABSTRACTABSTRACTABSTRACT

Abstractand etmiR-373/520c
are(Voorhoeveal., 2006;
et notal., easilyof miRAUTH.:
Voorhoeve,M.,
le Sage, regulateC., Schrier,M., et al. – YEAR: discerned
200610b, miR-21,
Abstract…identified.
However,theMacontributions
Endogenous
small RNAs
(miRNAs)
gene…

(Voorhoeve
et al.,Ma et al.,2007;discerned
Si et10b,al., 2007).
Similarly,
the2006;
miRNAs…
Endogenous
small RNAs
(miRNAs)acrossregulategene
miR-21,
andmiR-373/520c
are not easily
expression
bymechanisms
conserved
metazoans.
2007; Si et al., 2007). Similarly, the miRNAs…
Abstract
expression
bymechanisms
conserved
acrossmetazoans.
discerned(Voorhoeve
et
al.,
2006;
Ma et al.,While the Endogenous
number of verified
human
miRNAsregulate
is still gene
small
RNAs
(miRNAs)
While
thenumberofverified
human
miRNAsisstill2007; Si et al., 2007). Similarly, the miRNAs…
expanding,expression
only few havebeen functionallyannotated.
bymechanisms
conserved
acrossmetazoans.
CitingPaper
10 CITATION CONTEXT
expanding,
only
few
have
been
functionallyannotated.
To performWhile
genetic
screens…
CitingPaper
10 CITATION CONTEXT
thenumberofverified
human
miRNAsisstill
To perform genetic screens…
…LATS2 expression. To directly measure theexpanding, only few have been functionally annotated.
Introduction
expression.
To on
directly
measurePaper
10effect…LATS2
of Citingendogenous
Dnd1
theCITATION
activityof theCONTEXT
IntroductionTo
perform
genetic
screens…
effect ofmiR-372
endogenous
Dnd1
on theactivity ofSince their discovery, the functions of only a handful of
endogenous
family,weusedsensor…LATS2 miR-372
expression.
To directly
measureSince theirdiscovery,
functionsof onlyaSPAN
handful of
CITIED
endogenous
family,weusedsensor the
microRNAs(miRNAs)
have thebeen
determined(recently
Introduction
molecules
containing
theluciferase
gene
under
CITING
SPAN
effectcontainingof endogenous
Dnd1 on gene
the activityofmicroRNAs(miRNAs)
have2005).
been determined(recently
molecules
the LATS2-3′UTR
luciferase
underreviewedinZamoreandHaley,
Relevant
tothecontrolofeitherwild-type
or
a
Since
their
discovery,
thefunctionsof
onlyahandfulofendogenous
miR-372
family,we used sensor
reviewed init Zamore
andthatHaley,
2005).
Relevant
totheincontrol
eitherwild-type
LATS2-3′UTRor acarcinogenesis,
was (miRNAs)
found
let-7
inhibits
RASCITING
SPANmutant
the 372oftarget
sites(letheSage
et al., genemicroRNAs
have
been
determined(recently
molecules
containing
luciferase
undercarcinogenesis,
it
was
found
that
let-7
inhibits
RAS
mutantin
the372
target
sites
(le
Sage
et
al.,
expressionreviewed
and in lungtumors negatively
correlateswith to2007
andVoorhoeveetal.,2006).
Figure
1H…
inZamoreandHaley,
2005).
Relevantof eitherwild-type
LATS2-3′UTR
expression
andet
in al.,
lung2005).
tumorsnegatively correlates
with
2007theandcontrol
Voorhoeve
et al.,
2006). Figure
1H… or a
RAS levels(Johnson
Furthermore,the…
carcinogenesis,
it
was
found
that
let-7
inhibits
RAS
mutantin
the372
target
sites
(le
Sage
et al.,RAS levels (Johnson et al., 2005). Furthermore, the…
expression
and
in
lung
tumorsnegatively
correlates
with
CITED
SPAN
Results
2007 and Voorhoeve et al., 2006). Figure 1H…
CITINGSPAN the…
Results
RASzebrafish
levels (Johnson
al.,To2005).
Furthermore,
HUMAN SUMMARY…mouse,and(Figureet
5D).
further
HUMAN SUMMARY…mouse,andzebrafish(Figure5D).
TofurtherIn the article. Voorhoeve et al., performed
substantiate
LATS2 as a direct target of miR372&3,
we SPANCITING
Results
In thearticle.
Voorhoeve
et al., performed
substantiate
LATS2 as a directtargetofluciferase
miR372&3, we
genetic
screensof spansmiRNA
toof
investigate
itsclonedits 3′UTR
downstreamof thefirefly5D).
HUMAN
SUMMARY…mouse,andzebrafish(FigureTofurther
Task
1:
For
each
citationcontext,
identify
thetext(citedgenetic
screens
ofhas
miRNA
to
investigate
its text spans)its 3′UTR
downstreamofthe firefly
luciferase
novel
functions;which
implicatedtwogene cloned(pGL3-LATS2).
Wetransfected
eitherpGL3-LATS2
or
In
the
article.
Voorhoeveetal.,performed
LATS2
as a direct target
miR372&3,orwe
novelfunctions;
which
has
implicated twogene substantiate(pGL3-LATS2).
Wetransfected
eitherof
pGL3-LATS2
in3′UTRtheRP
that(containing
most
accuratelyreflectthecitationcontextof themasgenetic
oncogenes.
Theydemonstrated
the controls
pGL3-372
andpGL3-373
a miRscreensof miRNA
to investigate its
clonedits
downstream
ofthefirefly
luciferase
of
them
as
oncogenes.
Theydemonstrated
thecontrols
pGL3-372
andpGL3-373(containingamiRthatmiRNA-372&3
participate
incomplementary
sequence
intheir
3′UTR)
orpGL3
intonovelspanfunctions;which hasgene (pGL3-LATS2).
We
pGL3-LATS2
or cited
Task 2: identify
theeitherfacetof eachtext
among:
Hypothesis,
thatmiRNA-372&3
participate
inimplicated twoin transfected
their
3′UTR)
pGL3
intoproliferationand tumorigenesisofTheyprimary
Tera1complementary
and theMCF-7
cellssequence(respectively
positive
andor
negativeofthemas
oncogenes.
demonstrated
controls
pGL3-372
andpGL3-373(containingamiRproliferation
andtumorigenesis
ofandprimary
Tera1 and (MCF-7
cells(respectively
positiveandnegative Discussion
Method,Results,
Implication,
human
cells
along
with
oncogenic
RAS
for miR-371-3)
Figures
4D
and
S6).
As
predicted,
thethatmiRNA-372&3
participate
in andcomplementary
sequence
intheir
3′UTR)
or
pGL3
into
human
cellsp53
alongwith oncogenic
RAS
for miR-371-3)( Figures
4D andmediated…
S6). As predicted, theactivewild-typebynumbingthep53
372/373
complementary
sequencesproliferation
andtumorigenesisof primary
Tera1
and
MCF-7
(respectively
and negativeTask
3:cells
generatea positive
max 250
wordspathway.
summary
considering
thethecommunity
activewild-typep53
by numbing
p53
372/373
complementary
sequencesmediated…
The
authors
created…
References
human
alongcreated…
with oncogenic RAS andfor miR-371-3) ( Figures 4D and S6). As predicted, thepathway.
Thecells
authors
discussionofthereference
paper
representedbythecitation contexts
References
active wild-type p53 by numbing the p53
372/373 complementary sequences mediated…
pathway. The
authors created…
Dataset: 20 training collections + 30 evaluationcollections
References

BODY
BODY
BODY

……

http://wing.comp.nus.edu.sg/cl-scisumm2016/…

Challenges, datasets and architectures

Scientific Literature Mining Challenges
SemEval-2010 Task 5 : Automatic Keyphrase Extraction from Scientific Articles
Kim, S. N., Medelyan, O., Kan, M. Y., & Baldwin, T. Semeval-2010 task 5: Automatic keyphrase extraction from scientific articles.
In Proceedings of the 5th International Workshop on Semantic Evaluation (pp. 21-26). Association for Computational Linguistics.

• 100 articles for training and 144 for testing (from ACM Digital Library)• converted by pdftotext• keyphrases present in the text of the papers identified by authors and students
SemEval-2017 Task 10: ScientceIE - Extracting Keyphrases and Relations from Scientific
Publications
http://alt.qcri.org/semeval2017/task10/ & https://scienceie.github.io/• Corpus: Science Direct, 500 journal articles evenly distributed among the domains
Computer Science, Material Sciences and Physics• training: 350 documents, development: 50 documents, test: 100 documents• task 1: Identification of keyphrases• task 2: Classification of identified keyphrases (PROCESS, TASK and MATERIAL)• task 3: identification of relations among keyphrases: HYPONYM-OF, SYNONYM-OF, NONE

Challenges, datasets and architectures

Datasets and tools
The ACL anthology network corpus
Radev, D. R., Muthukrishnan, P., Qazvinian, V., & Abu-Jbara, A. (2013). The ACL anthology network corpus. Language Resources and
Evaluation, 47(4), 919-944.

• last release: December 2013
• PDFbox to convert PDF papers• semi-automated manual editing

ACL Anthology SearchBench
Schäfer, U., Kiefer, B., Spurk, C., Steffen, J., & Wang, R. (2011, June). The ACL anthology searchbench. In Proceedings of the 49th Annual
Meeting of the Association for Computational Linguistics: Human Language Technologies: Systems Demonstrations (pp. 7-13).
Association for Computational Linguistics.

• last update: November 2013, 28,000 papers• commercial OCR to parse PDF• integrates CiBRO to visualize citation
networkChallenges, datasets and architectures

Datasets and tools

https://core.ac.uk/

https://aminer.org/• open access content aggregator• 37,634,579 papers with bibliographic record + PDF• 6000 journals, collected from over 2300 Open
Access repositories around the world (OAI-PMH)• Web API• metadata: authors, abstract, topics, year, provided
by OAI• more than 130,000,000 researcher profiles and100,000,000 papers from multiple publication
databases• Services: Researcher profile extraction (connection
with social networks like LinkedIn andVideoLectures), expert finding, social network
search, , topic browser , conference analysis•Web APIChallenges, datasets and architectures

Datasets and tools

http://citeseerx.ist.psu.edu/• open access digital library search engine (all docs
with full text)• extract and index both metadata and full text
• provides access to metadata by means of OAI• index also tables and figures• 20,000 to 40,000 new crawled PDF per day – 10
PDF downloaded per second• Computer science and Neuroscience papers
from: ArXiv, DBLP, CiteSeer, OdySci Academic,
Aminer
https://www.semanticscholar.org/• cits. count estimated (statistical model)• keyphrases• citation velocity and acceleration• influential authors

Challenges, datasets and architectures

Structured / semantic publication formats
Even if 80% of scientific literature is accessed as PDF documents,
structured textual formats to model the contents of scientific publications
are increasingly spreading• JATS XML: an de facto standard for archiving and interchange of scientific open-accessjournals and its contents with XML• Major publishers have their own XML schemas: Elsevier, Springer

Semantic Web and Scholarly data
Set of ontologies that support the creation of comprehensive machinereadable RDF metadata for every aspect of semantic publishing andreferencing

SPAR Ontologies• FRBR-aligned Bibliographic Ontology (FaBiO)• Document Components Ontology (DoCO)• Citation Typing Ontology (CiTO)• Publishing Status Ontology (PSO)• Bibliographic Reference Ontology (BiRO)• Publishing Roles Ontology (PRO)• Citation Counting and Context Characterisation • Publishing Workflow Ontology (PWO)• Ontology (C4O)• Discourse Elements Ontology (DEO)Challenges, datasets and architectures

Scholarly literature architectures
Mendeley Suggest
Provide users with articles that help them to keep up-to-date with research in their fieldand explore relevant research that is, as of yet, unknown to them
Consolidate theconsistency of thelist ofrecommendation

Multi-model
recommendation:
collaborative filtering(Mahout), popularitybased model,
trending-basedmodel, content-based
model

Filter out data nod
needed by therecommender model
Online models to adapt
dynamically recommendations
to user interactions

Event service: Track user interactions
Article service: crowdsurced collection of
Mendeley artlcles
Profile service: user profile information(academic discipline, etc.)

Challenges, datasets and architectures

Scholarly literature architectures
CiteSeer
1. Academic and non-academic classification: SVM – features: document length,
inclusion of bibliography, etc.
2. Paper de-duplication:• Exact PDF match: SHA1 digest• Near-duplicate match: based on document signature strings
3. Metadata extraction:• Header: cascade of SVM classifiers• Body• Citations: ParsCit (CRF-based)
4. Author name disambiguation: author names grouped into blocks of similar names.
Names are matched by comparing features like titles of edited papers, co-authors, etc.

40,000 lines of codes
10 person-years for the development
https://github.com/SeerLabs/CiteSeerX
Crawler:Name disambiguation:

DR. INVENTOR SCIENTIFICTEXT MINING FRAMEWORK

http://drinventor.eu/
FP7 ICT 2013.8.1, Grant no.: 611383

Outline• Dealing with scientific articles in Dr. Inventor• Dr. Inventor Text Mining Framework– Architectural overview– Hands-on Dr. Inventor Framework

Dealing with scientific articles in Dr. Inventor
The (bootstrap of) textual analyses of scientific publications
often still constitutes a time-consuming activity due to: hetherogeneous input formats (PDF, XML schemas, etc.)
 lack of explicit structural and semantic information
 need to enrich contents by leveraging on external data sources lack of convenient facilities to easily access and process contents

Dealing with scientific articles in Dr. Inventora scientific information mining infrastructure useful to:• analyze publications and track research topics• assess the novelty of ideas• stimulate researchers creativity by suggesting
analogies between scientific outcomes

Text mining

Scientificpublications

Big data

Semantic
analysis
Visualanalytics

Graph mining

http://drinventor.eu/
FP7 ICT 2013.8.1, Grant no.: 611383

Dr. Inventor Text Mining Framework• Integrate and customize text mining tools and on-line services
to enable and ease a wide range of scientific publication analyses• Papers are enriched with structural, linguistic and semantic
information

http://driframework.readthedocs.io/• Self-contained

library managed by• Focused on textual content• Relying on a shared data model (java classes) to represent a paper• Exposing a convenient API to access the mined information• Based on

to manage textual annotations

Extractive summarizer

Coreference res., causalityspotter & graph builder

Rhetorical annotator

Citation-aware dep. parser

Web based reference parser

Sentence splitter

Inline citation
spotter

PDF to text
converter

Dr. Inventor Text Mining
Framework

Babelfy WSD and Entity Linker

Architectural overview

Ronzano, F., & Saggion, H. (2015, October). Dr. Inventor Framework: Extracting Structured Informationfrom Scientific Publications. In International Conference on Discovery Science (pp. 209-220). Springer
International Publishing.

Ronzano, F., & Saggion, H. (2016, April). Knowledge Extraction and Modeling from Scientific
Publications. In The Semantics, Analytics, Visualization: Enhancing Scholarly Data Workshop, co-locatedwith the 25th International World Wide Web Conference.

Dr. Inventor
paper data modelTITLE

CAPTION(SUB)SECTION

BIBLIOGRAPHIC ENTRY
ABSTRACT

Dr. Inventor Text Mining
Framework

Extractive summarizer

Coreference res., causalityspotter & graph builder

Babelfy WSD and Entity Linker

Rhetorical annotator

Citation-aware dep. parser

Web based reference parser

Sentence splitter

Inline citation
spotter

PDF to text
converter

Architectural overview

Supported by:Bibliography

Extractive summarizer

Coreference res., causalityspotter & graph builder

Rhetorical annotator

Citation-aware dep. parser

Web based reference parser

Sentence splitter

Inline citation
spotter

PDF to text
converter

Dr. Inventor Text Mining
Framework

Babelfy WSD and Entity Linker

Architectural overview

1. identification of inline citation markers and spans 
JAPE rules
2. linking of inline citation markers to bibliographic entries
3. identification of syntactic / non-syntactic role of inline
citation spansSome alternative phrase alignment approaches have beendeveloped, which do not rely on the Viterbi word alignment.
Both (Marcu, 2002) and (Zhang, 2003) consider a sentence pair
as different realizations of a sequence of concepts. Thesealignment approaches segment the sentences into a sequenceof phrases.

Extractive summarizer

Coreference res., causalityspotter & graph builder

Rhetorical annotator

Citation-aware dep. parser

Web based reference parser

Sentence splitter

Inline citation
spotter

PDF to text
converter

Dr. Inventor Text Mining
Framework

Babelfy WSD and Entity Linker

Architectural overview

Customization of ANNIE sentence splitter
Rule set adapted to peculiarities of scientific papers
by analyzing the most frequent sentence split
patterns / errors in a set of 40 Computer
Graphics papers

YEAR

TITLE

CONFERENCEAUTHORS

Bibliography

Dr. Inventor Text Mining
Framework

Extractive summarizer

Coreference res., causalityspotter & graph builder

Babelfy WSD and Entity Linker

Rhetorical annotator

Citation-aware dep. parser

Web based reference parser

Sentence splitter

Inline citation
spotter

PDF to text
converter

Architectural overview

Dr. Inventor Text Mining
Framework

Extractive summarizer

Coreference res., causalityspotter & graph builder

Babelfy WSD and Entity Linker

Rhetorical annotator

Citation-aware dep. parser

Web based reference parser

Sentence splitter

Inline citation
spotter

PDF to text
converter

Architectural overview

Based on MATE dependency parser

Inline citation spans should be considered as a word-token by the parser if they have a syntactic role:Extractive summarizer

Coreference res., causalityspotter & graph builder

Rhetorical annotator

Citation-aware dep. parser

Web based reference parser

Sentence splitter

Inline citation
spotter

PDF to text
converter

Dr. Inventor Text Mining
Framework

Babelfy WSD and Entity Linker

Architectural overview• Rhetorical categories: Background, Challenge,
Approach, Outcome, Future Work• Linguistic and syntactic sentence features exploitedto train Logistic Regression classifier on Dr. Inventor
Corpus (40 Computer Graphics papers including 8,777
sentences)bn:00074060n
Statisticalmethod, statistic
al procedure

Dr. Inventor Text Mining
Framework

Extractive summarizer

Coreference res., causalityspotter & graph builder

Babelfy WSD and Entity Linker

Rhetorical annotator

Citation-aware dep. parser

Web based reference parser

Sentence splitter

Inline citation
spotter

PDF to text
converter

Architectural overview

bn:00488805n
Feature (machinelearning, patternrecognition)
bn:00075149n
Summarisation,summarization

These modules undergo
deformation caused by force
fields. They also implement
attractive and repulsive force
fields.

These modules

coreferent

undergodeformation

Extractive summarizer

Coreference res., causalityspotter & graph builder

Rhetorical annotator

Citation-aware dep. parser

Web based reference parser

Sentence splitter

Inline citation
spotter

PDF to text
converter

Dr. Inventor Text Mining
Framework

Babelfy WSD and Entity Linker

Architectural overview

They

implement

cause

force fields

Extractive summarizer

Coreference res., causalityspotter & graph builder

Rhetorical annotator

Citation-aware dep. parser

Web based reference parser

Sentence splitter

Inline citation
spotter

PDF to text
converter

Dr. Inventor Text Mining
Framework

Babelfy WSD and Entity Linker

Architectural overview• Based on SUMMA text summarization toolkit• Sentence relevance ranking approaches:• TF-IDF centroid of each section• TF-IDF similarity with title• LexRank / TextRank (soon)• LR of sentence features (soon)

Hands-on Dr. Inventor Framework
Full documentation and examples at: http://driframework.readthedocs.io/•••

Lazy loading
Object caching
Factory Design Pattern to manage resource allocations
Importing the library

MAVEN:<repositories><repository><id>backingdata-repo</id><name>Backingdata repository</name><url>http://backingdata.org/dri/library/mavenRepo/</url></repository></repositories><dependency><groupId>edu.upf.taln.dri</groupId><artifactId>lib</artifactId><version>1.0</version></dependency>
JAVA:
Download ZIP file with JAR and dependencies

Hands-on Dr. Inventor Framework
Full documentation and examples at: http://driframework.readthedocs.io/
Configure programmatically the library
Which PDF-to-text converter?
// To use PDFX:
Factory.setPDFtoTextConverter(PDFtoTextConvMethod.PDFX);// To use GROBID:
Factory.setPDFtoTextConverter(PDFtoTextConvMethod.GROBID);

Which modules are enabled?
// Instantiate the ModuleConfig class - the constructor sets all modules// enabled by default
ModuleConfig modConfigurationObj = new ModuleConfig();// Disable the parsing of bibliographic entries by means of online// services (Bibsonomy, CrossRef, FreeCite)
modConfigurationObj.setEnableBibEntryParsing(false);// Disable the association of a rhetorical category to the sentences of// the paper
modConfigurationObj.setEnableRhetoricalClassification(false);// Improt the configuration parameters set in the ModuleConfig instance
Factory.setModuleConfig(modConfigurationObj);Hands-on Dr. Inventor Framework
Full documentation and examples at: http://driframework.readthedocs.io/
Import PDF / JATS XML from file / URL// From File (substitute parsePDF for parseJATS to import JATS file):
Document doc_PDFfile =Factory.getPDFloader().parsePDF("/my/file/path/PDF_file_name.pdf");// From URL (substitute parsePDF for parseJATS to import JATS file):
Document doc_PDFURL = Factory.getPDFloader().parsePDF(newURL("http://www2007.org/workshops/paper_45.pdf"));Get ordered lists of sentences// Only abstract sentences
List<Sentence> abstract_SentList =doc_PDFfile.extractSentences(SentenceSelectorENUM.ONLY_ABSTRACT);// Only body sentences
List<Sentence> abstract_SentList =doc_PDFfile.extractSentences(SentenceSelectorENUM.ALL_EXCEPT_ABSTRACT);// Only abstract sentences
List<Sentence> abstract_SentList =doc_PDFfile.extractSentences(SentenceSelectorENUM.ALL);Hands-on Dr. Inventor Framework
Full documentation and examples at: http://driframework.readthedocs.io/
Print the content of the first sentence of the abstract(by the asString method of the Sentence object instance)// Get ordered list of abstract sentences
List<Sentence> abstract_SentList =doc_PDFfile.extractSentences(SentenceSelectorENUM.ONLY_ABSTRACT);// Get the first sentence of the abstract
Sentence firstAbstractSentence = abstract_SentList.get(0);// Print all the data associated to the first sentence of the abstract
System.out.println(firstAbstractSentence.asString(true));[SENTENCE] ID: '22047',
Text: 'Puppetry has been a popular art form for many centuries in different cultures, which becomes a valuable and fascinating heritage assert.',Rhetorical class: 'DRI_Background'23 TOKENS ASSOCIATED[BABELNET SYNSET] ID: '22047', Text: 'Puppetry', In-sentence ID: '22047', Babel URL: 'http://babelnet.org/rdf/s00065258n', Synset ID:'bn:00065258n', DBpedia URL: 'http://dbpedia.org/resource/Puppetry', Global score: '2.958149116792834E-4', Coherence score:'0.09968354430379747', Score: '0.8340262582056893', Source: 'bn:00065258n', Num tokens: '1'[BABELNET SYNSET] ID: '22047', Text: 'art', In-sentence ID: '22047', Babel URL: 'http://babelnet.org/rdf/s00005927n', Synset ID: 'bn:00005927n',
DBpedia URL: 'http://dbpedia.org/resource/Art', Global score: '0.023262002991754745', …

Hands-on Dr. Inventor Framework
Full documentation and examples at: http://driframework.readthedocs.io/
Print the content of the paper sections in document order// Get ordered list of document sections
List<Section> sectionList = doc_PDFfile.extractSections(false);
for(Section sec : rootSectionList) {// Print all the data associated to the section
System.out.println(sec.asString(true));// Get the list of sub-sections
List<Section> subSection = sec.getSubsections();// Get the list of sentences inside the section
List<Sentence> sentencesOfSection = sec.getSentences();}[SECTION] ID: '21452', Name: '1. INTRODUCTION', Level: '1', Children sections IDs: '[]', Sentences IDs: '[22053, 22054, 22055, 22056, 22057,
22058, 22059, 22060, 22061, 22062, 22063]'[SECTION] ID: '21453', Name: ' 2. RELATED WORK', Level: '1', Children sections IDs: '[21454, 21455]', Sentences IDs: '[]‘
[SECTION] ID: '21454', Name: ' 2.1. Head Modelling', Level: '2', Children sections IDs: '[]', Sentences IDs: '[22075, 22064, 22065, 22066, 22067,
22068, 22069, 22070, 22071, 22072, 22073, 22074]‘[SECTION] ID: '21455', Name: ' 2.2 Swept Surface Modelling', Level: '2', Children sections IDs: '[]', Sentences IDs: '[22082, 22083, 22084, 22085,
22086, 22076, 22077, 22078, 22079, 22080, 22081]’ …

Hands-on Dr. Inventor Framework
Full documentation and examples at: http://driframework.readthedocs.io/
Print the content of the bibliographic entries / citations// Get ordered list of bibliographic entries
List<Citation> citations = doc_PDFfile.extractCitations();for(Citation citation : citations) {// Print all the data associated to the citation
System.out.println(citation.asString(true));}[CITATION] ID: '21440', Source: '[Bibsonomy]', Title: 'Realtime performance-based facial animation.', Year: '2011', Pages: '77',
Bibsonomy URL: 'http://dblp.uni-trier.de/db/journals/tog/tog30.html#WeiseBLP11', Volume: '30', Journal: 'ACM Trans. Graph.',Text: 'Realtime performance-based facial animation T Weise S Bouaziz H Li Pauly M ACM Transactions on Graphics (TOG) 30 4 77'[AUTHOR] Full name: ', Thibaut Weise', First name: ', Thibaut', Surname: 'Weise'[AUTHOR] Full name: ', Sofien Bouaziz', First name: ', Sofien', Surname: 'Bouaziz'[AUTHOR] Full name: ', Hao Li', First name: ', Hao', Surname: 'Li'
[AUTHOR] Full name: ', Mark Pauly', First name: ', Mark', Surname: 'Pauly'
PUB ID TYPE: DOI - VALUE: http://dx.doi.org/10.1145/2010324.1964972[CIT MARKER] ID: '40228', Citation ID: '21440', Sentence ID: '22067', Reference text: '10'
[CIT MARKER] ID: '40241', Citation ID: '21440', Sentence ID: '22071', Reference text: '10'

Hands-on Dr. Inventor Framework
Full documentation and examples at: http://driframework.readthedocs.io/
Get an print the list of sentences of the 10-sentences extractive summary
generated by the section title tf-idf similarity method// Get ordered list of summary sentences
List<Sentence> summarySentences_TITLE_10 =doc_PDFfile.extractSummary(20, SummaryTypeENUM.TITLE_SIM);// Print the text of each sentencefor(Sentence sent : summarySentences_CENTROID_20){
System.out.println(sent.getText());}

Several high level tasks look for either one-way rewriting between single sentences, like recognizing textual entailment (RTE) ( Dagan et al., 2006 ), or two-way rewritings likeparaphrase identification ( Dolan et al., 2004 ) and semantic textual similarity ( Agirre et al., 2012 ).
Our system based on type-enriched string rewriting kernels obtains state-of-the-art results on paraphrase identification and answer sentence selection and outperforms
comparable methods on RTE.
String rewriting kernels ( Bu et al., 2012 ) count the number of common rewritings between two pairs of sentences seen as sequences of words.
Following the terminology of string kernels, we use the term string and character instead of sentence and word.
A type-enriched string rewriting kernel (TESRK) is simply a string rewriting kernel as defined in Equation 1 but with R a set of typed rewriting rules.
However, it cannot match the pair of sentences (C) in the original kb-SRK.
We experimented on three tasks: paraphrase identification, recognizing textual entailment and answer sentence selection.
Recognizing Textual Entailment asks whether the meaning of a sentence hypothesis can be inferred by reading a sentence text.
A SVM classifier with this kernel yields state-of-the-art results in paraphrase identification and answer sentence selection and outperforms comparable systems in recognizing
textual entailment.

Hands-on Dr. Inventor Framework
Full documentation and examples at: http://driframework.readthedocs.io/
Save and reload a processed paper - serialized as XML// Get the raw text contents of the paper
String rawText = doc_PDFfile.getRawText();// Get the XML serialization of the contents of the paper, including// all the metadata already extracter
String XMLText = doc_PDFfile.getXMLString();// Save the XML serialization of the contents of the paper to// the file: /my_path/stored_paper.xml…

// Reload the contents of the paper from// the file: /my_path/stored_paper.xml
Document doc_PDFfile_Loaded =Factory.createNewDocument("/my_path/stored_paper.xml");GLOBAL CONCLUSIONSAND DISCUSSION• There is considerable room for improvement in the next future withrespect to the automation of the analysis, aggregation andsummarization of scientific literature• The Natural Language Processing community plays a key role inproviding better automated techniques to mine scientific literature• The investigation of scientific text mining approaches should take into
account both their effectiveness and the possibility to scale overlarge, heterogeneous and dynamic collections of papers

Natural Language Processing
for Intelligent Accessto Scientific Information

Many thanks for your attention!
Francesco Ronzano and Horacio Saggion
Natural Language Processing Group (TALN)
Universitat Pompeu Fabra, Barcelona, Spain
Tutorial @ COLING 2016
11th December 2016