WORKSHOP REPORT

Report on the 8th International Workshop on
Bibliometric-enhanced Information Retrieval(BIR 2019)
Guillaume Cabanac
University of Toulouse, Franceguillaume.cabanac@univ-tlse3.fr

Ingo Frommholz
University of Bedfordshire, UK
ifrommholz@acm.org

Philipp MayrGESIS – Leibniz Institute for the Social Sciences, Germany
philipp.mayr@gesis.org

Abstract
The Bibliometric-enhanced Information Retrieval workshop series (BIR) at ECIR tackledissues related to academic search, at the crossroads between Information Retrieval and Bibliometrics. BIR is a hot topic investigated by both academia (e.g., ArnetMiner, CiteSeerχ , DocEar) and the industry (e.g., Google Scholar, Microsoft Academic Search, Semantic Scholar).
This report presents the 8th iteration of the one-day BIR workshop held at ECIR 2019 inCologne, Germany.

1Introduction

Searching for scientific information is a long-lived information need. In the early 1960s, Salton
was already striving to enhance information retrieval by including clues inferred from bibliographic
citations [1]. The development of citation indexes pioneered by Garfield [2] proved determinant
for such a research endeavour at the crossroads between the nascent fields of Information Retrieval(IR) and Bibliometrics1 . The pioneers who established these fields in Information Science—suchas Salton and Garfield—were followed by scientists who specialised in one of these [5], leading tothe two loosely connected fields we know of today.
The purpose of the BIR workshop series founded in 2014 is to tighten up the link between IRand Bibliometrics. We strive to get the ‘retrievalists’ and ‘citationists’ [5] active in both academia
and the industry together, who are developing search engines and recommender systems such asArnetMiner [6], CiteSeerχ [7], DocEar [8], Google Scholar [9], Microsoft Academic Search [10],and Semantic Scholar [11], just to name a few.
1Bibliometrics refers to the statistical analysis of the academic literature [3] and plays a key role in scientometrics:
the quantitative study of science and innovation [4].

ACM SIGIR Forum

21

Vol. 53 No. 1 June 2019

Bibliometric-enhanced IR systems must deal with the multifaceted nature of scientific information by searching for or recommending academic papers, patents [12], venues (i.e., conferencesor journals), authors, experts (e.g., peer reviewers), references (to be cited to support an argument), and datasets. The underlying models harness relevance signals from keywords provided
by authors, topics extracted from the full-texts, coauthorship networks, citation networks, andvarious classifications schemes of science.
Bibliometric-enhanced IR is a hot topic whose recent developments made the news—see forinstance the Initiative for Open Citations [13] and the Google Dataset Search [14] launched on
September 4, 2018, which give an impression of arising challenges subject to both communities.
We believe that BIR@ECIR is a much needed scientific event for the ‘retrievalists’ and ‘citationists’to meet and join forces pushing the knowledge boundaries of IR applied to literature search andrecommendation.

2Past Related Activities

The BIR workshop series was launched at ECIR in 2014 [15] and it was held at ECIR each year
since then [16, 17, 18, 19]. As our workshop has been lying at the crossroads between IR andNLP, we also ran it as a joint workshop called BIRNDL (for Bibliometric-enhanced IR and NLPfor Digital Libraries) at the JCDL [20] and SIGIR [21, 22] conferences. All workshops had alarge number of participants, demonstrating the relevance of the workshop’s topics. The BIR andBIRNDL workshop series gave the community the opportunity to discuss latest developments andshared tasks such as the CL-SciSumm [23], which was introduced at the BIRNDL joint workshop.
The authors of the most promising workshop papers were offered the opportunity to submit anextended version for a Special Issue for the Scientometrics journal [24, 25] and of the International
Journal on Digital Libraries [26].
The target audience of our workshop are researchers and practitioners, junior and senior,from Scientometrics as well as Information Retrieval. These could be IR researchers interested
in potential new application areas for their work as well as researchers and practitioners working
with, for instance, bibliometric data and interested in how IR methods can make use of such data.

3Objectives and Topics for BIR@ECIR 2019

We called for original research at the crossroads of IR and Bibliometrics. Thirteen peer-reviewed
papers were accepted2 [27]: 9 long papers, 3 short papers and 1 demo paper. These report on
new approaches using bibliometric clues to enhance the search or recommendation of scientificinformation or significant improvements of existing techniques. Thorough quantitative studies ofthe various corpora to be indexed (papers, patents, networks or else) were also contributed. The
papers are as follows:• Long papers:– An interactive visual tool for scientific literature search: Proposal and algorithmic
specification [28]2

See workshop proceedings: http://ceur-ws.org/Vol-2345/.

ACM SIGIR Forum22Vol. 53 No. 1 June 2019– A searchable space with routes for querying scientific information [29]– Discovering seminal works with marker papers [30]
– How do computer scientists use Google Scholar?: A survey of user interest in elements
on SERPs and author profile pages [31]– Feature selection and graph representation for an analysis of science fields evolution:
An application to the digital library ISTEX [32]– Optimal citation con-text window sizes for biomedical retrieval [33]– Bibliometric-enhanced arXiv: A data set for paper-based and citation-based tasks [34]– Mining intellectual influence associations [35]– Citation metrics for legal information retrieval systems [36]• Short papers:– Finding temporal trends of scientific concepts [37]– A preliminary study to compare deep learning with rule-based approaches for citation
classification [38]– Improving scientific article visibility by neural title simplification [39]• Demo:– Recommending multimedia educational resources on the MOVING platform [40].
The topics of the workshop are in line with those of the past BIR and BIRNDL workshops(Fig. 1): a mixture of IR and Bibliometric concepts and techniques. More specifically, the callfor papers featured current research issues regarding three aspects of the search/recommendation
process:
1. User needs and behaviour regarding scientific information, such as:• Finding relevant papers/authors for a literature review;• Measuring the degree of plagiarism in a paper;• Identifying expert reviewers for a given submission;• Flagging predatory conferences and journals.
2. The characteristics of scientific information:• Measuring the reliability of bibliographic libraries;• Spotting research trends and research fronts.
3. Academic search/recommendation systems:• Modelling the multifaceted nature of scientific information;• Building test collections for reproducible BIR.

ACM SIGIR Forum

23Vol. 53 No. 1 June 2019

Figure 1: Main topics of the BIR and BIRNDL workshop series (2014–2018) as extracted fromthe titles of the papers published in the proceedings, see https://dblp.org/search?q=BIR.ECIRand https://dblp.org/search?q=BIRNDL.

4Peer Review Process and Organization

The 8th BIR edition ran as a one-day workshop, as it was the case for the previous editions.
Dr. Iana Atanassova delivered a keynote entitled “Beyond Metadata: the New Challenges inMining Scientific Papers” [41] to kick off the day.
Two types of papers were presented: long papers (15-minute talks) and short papers (5-minutetalks). As the interactive session introduced last year was generally acclaimed, we decided toorganize a interactive session to close the workshop. Two weeks earlier, we invited all registeredattendees to demonstrate their prototypes or pitch a poster during flash presentations (5 minutes).
This was an opportunity for our speakers to further discuss their work and for the public toshowcase their work too.
We ran the workshop with peer review supported by EasyChair3 . Each submission was assigned
to 2 to 3 reviewers, preferably at least one expert in IR and one expert in Bibliometrics. The
stronger submissions were accepted as long papers while weaker ones were accepted as short papers,and demo. All authors were instructed to revise their submission according to the reviewers’
3

https://easychair.org

ACM SIGIR Forum24Vol. 53 No. 1 June 2019

reports. All accepted papers were included in the workshop proceedings [27] hosted at ceur-ws.
org, an established open access repository with no author-processing charges.
As a follow-up of the workshop, all authors are encouraged to submit an extended version of
their papers to the Special Issue of the Scientometrics journal launched in Spring 2019.

References[1] Salton, G.: Associative document retrieval techniques using bibliographic information. Journal of the ACM 10(4) (1963) 440–457[2] Garfield, E.: Citation indexes for science: A new dimension in documentation throughassociation of ideas. Science 122(3159) (1955) 108–111[3] Pritchard, A.: Statistical bibliography or bibliometrics? [Documentation notes]. Journal ofDocumentation 25(4) (1969) 348–349[4] Leydesdorff, L., Milojević, S.: Scientometrics. In Wright, J.D., ed.: International Encyclopedia of the Social & Behavioral Sciences. Volume 21. 2nd edn. Elsevier (2015) 322–327[5] White, H.D., McCain, K.W.: Visualizing a discipline: An author co-citation analysis ofInformation Science, 1972–1995. Journal of the American Society for Information Science49(4) (1998) 327–355[6] Tang, J., Zhang, J., Yao, L., Li, J., Zhang, L., Su, Z.: ArnetMiner: Extraction and mining ofacademic social networks. In: KDD’08: Proceeding of the 14th ACM SIGKDD international
conference on Knowledge discovery and data mining, New York, NY, USA, ACM (2008)
990–998[7] Williams, K., Wu, J., Choudhury, S.R., Khabsa, M., Giles, C.L.: Scholarly big data information extraction and integration in the CiteSeerχ digital library. In: ICDE’14: Proceedings ofthe 30th IEEE International Conference on Data Engineering Workshops, IEEE (2014) 68–73[8] Beel, J., Langer, S., Gipp, B., Nürnberger, A.: The architecture and datasets of docear’s
research paper recommender system. D-Lib Magazine 20(11/12) (2014)[9] Van Noorden, R.: Google Scholar pioneer on search engine’s future. Nature (2014)[10] Sinha, A., Shen, Z., Song, Y., Ma, H., Eide, D., Hsu, B.J.P., Wang, K.: An overview of Microsoft Academic Service (MAS) and applications. In Gangemi, A., Leonardi, S., Panconesi,A., eds.: WWW’15: Proceedings of the 24th International Conference on World Wide Web,New York, NY, USA, ACM (2015) 243–246[11] Bohannon, J.: A computer program just ranked the most influential brain scientists of themodern era. Science (2016)[12] Garfield, E.: Patent citation indexing and the notions of novelty, similarity, and relevance.
Journal of Chemical Documentation 6(2) (1966) 63–65

ACM SIGIR Forum25

Vol. 53 No. 1 June 2019[13] Shotton, D.: Funders should mandate open citations. Nature 553(7687) (2018) 129[14] Castelvecchi, D.: Google unveils search engine for open data [News & Comment]. Nature(2018)[15] Mayr, P., Schaer, P., Scharnhorst, A., Larsen, B., Mutschke, P., eds.: BIR’16 Proceedings ofthe 1st Workshop on Bibliometric-enhanced Information Retrieval co-located with the 36th
European Conference on Information Retrieval. Volume 1143., Aachen, CEUR-WS (2014)[16] Mayr, P., Frommholz, I., Mutschke, P., eds.: BIR’15 Proceedings of the 2nd Workshop on
Bibliometric-enhanced Information Retrieval co-located with the 37th European Conferenceon Information Retrieval. Volume 1344., Aachen, CEUR-WS (2015)[17] Mayr, P., Frommholz, I., Cabanac, G., eds.: BIR’16 Proceedings of the 3rd Workshop on
Bibliometric-enhanced Information Retrieval co-located with the 38th European Conferenceon Information Retrieval. Volume 1567., Aachen, CEUR-WS (2016)[18] Mayr, P., Frommholz, I., Cabanac, G., eds.: BIR’17 Proceedings of the 5th Workshop on
Bibliometric-enhanced Information Retrieval co-located with the 39th European Conferenceon Information Retrieval. Volume 1823., Aachen, CEUR-WS (2017)[19] Mayr, P., Frommholz, I., Cabanac, G., eds.: BIR’18 Proceedings of the 7th Workshop on
Bibliometric-enhanced Information Retrieval co-located with the 40th European Conferenceon Information Retrieval. Volume 2080., CEUR-WS (2018)[20] Cabanac, G., Chandrasekaran, M.K., Frommholz, I., Jaidka, K., Kan, M.Y., Mayr, P., Wolfram, D., eds.: BIRNDL’16: Proceedings of the Joint Workshop on Bibliometric-enhanced
Information Retrieval and Natural Language Processing for Digital Libraries co-located withthe Joint Conference on Digital Libraries. Volume 1610., Aachen, CEUR-WS (2016)[21] Mayr, P., Chandrasekaran, M.K., Jaidka, K., eds.: BIRNDL’17: Proceedings of the 2nd Joint
Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processingfor Digital Libraries co-located with the Joint Conference on Digital Libraries. Volume 1888.,Aachen, CEUR-WS (2017)[22] Mayr, P., Chandrasekaran, M.K., Jaidka, K., eds.: BIRNDL’17: Proceedings of the 3rd JointWorkshop on Bibliometric-enhanced Information Retrieval and Natural Language Processingfor Digital Libraries co-located with the Joint Conference on Digital Libraries. Volume 2132.,
Aachen, CEUR-WS (2018)[23] Jaidka, K., Chandrasekaran, M.K., Rustagi, S., Kan, M.Y.: Insights from CL-SciSumm 2016:
The faceted scientific document summarization shared task. International Journal on DigitalLibraries 19(2–3) (2018) 163–171[24] Mayr, P., Scharnhorst, A.: Scientometrics and information retrieval: weak-links revitalized.
Scientometrics 102(3) (2015) 2193–2199[25] Cabanac, G., Mayr, P., Frommholz, I.: Bibliometric-enhanced information retrieval: Preface.
Scientometrics 116(2) (2018) 1225–1227

ACM SIGIR Forum

26

Vol. 53 No. 1 June 2019[26] Mayr, P., Frommholz, I., Cabanac, G., Chandrasekaran, M.K., Jaidka, K., Kan, M.Y., Wolfram, D.: Special issue on bibliometric-enhanced information retrieval and natural language
processing for digital libraries. International Journal on Digital Libraries 19(2–3) (2018)
107–111[27] Cabanac, G., Frommholz, I., Mayr, P., eds.: BIR’19 Proceedings of the 8th Workshop on
Bibliometric-enhanced Information Retrieval co-located with the 41th European Conferenceon Information Retrieval. Volume 2345., Aachen, CEUR-WS (2019)[28] Bascur, J.P., van Eck, N.J., Waltman, L.: An interactive visual tool for scientific literature search: Proposal and algorithmic specification. In: Proc. of the 8th Workshop on
Bibliometric-enhanced Information Retrieval, CEUR-WS.org (2019) 76–87[29] Fabre, R.: A “searchable” space with routes for querying scientific information. In: Proc.
of the 8th Workshop on Bibliometric-enhanced Information Retrieval, CEUR-WS.org (2019)
112–124[30] Haunschild, R., Marx, W.: Discovering seminal works with marker papers. In: Proc. of the
8th Workshop on Bibliometric-enhanced Information Retrieval, CEUR-WS.org (2019) 27–38[31] Kim, J., Trippas, J.R., Sanderson, M., Bao, Z., Croft, W.B.: How do computer scientists use
Google Scholar?: A survey of user interest in elements on SERPs and author profile pages. In:Proc. of the 8th Workshop on Bibliometric-enhanced Information Retrieval, CEUR-WS.org(2019) 64–75[32] Lamirel, J.C., Cuxac, P.: Feature selection and graph representation for an analysis of sciencefields evolution: An application to the digital library ISTEX. In: Proc. of the 8th Workshop
on Bibliometric-enhanced Information Retrieval, CEUR-WS.org (2019) 88–99[33] Lykke Nielsen, B., Lavlund Skau, S., Meier, F., Larsen, B.: Optimal citation context window sizes for biomedical retrieval. In: Proc. of the 8th Workshop on Bibliometric-enhanced
Information Retrieval, CEUR-WS.org (2019) 51–63[34] Saier, T., Färber, M.: Bibliometric-enhanced arXiv: A data set for paper-based and citationbased tasks. In: Proc. of the 8th Workshop on Bibliometric-enhanced Information Retrieval,CEUR-WS.org (2019) 14–26[35] Shah, T., Pudi, V.: Mining intellectual influence associations. In: Proc. of the 8th Workshop
on Bibliometric-enhanced Information Retrieval, CEUR-WS.org (2019) 100–111[36] Wiggers, G., Verberne, S.: Citation metrics for legal information retrieval systems. In: Proc.
of the 8th Workshop on Bibliometric-enhanced Information Retrieval, CEUR-WS.org (2019)39–50[37] Färber, M., Jatowt, A.: Finding temporal trends of scientific concepts. In: Proc. of the 8thWorkshop on Bibliometric-enhanced Information Retrieval, CEUR-WS.org (2019) 132–139

ACM SIGIR Forum27

Vol. 53 No. 1 June 2019[38] Perier-Camby, J., Bertin, M., Atanassova, I., Armetta, F.: A preliminary study to compare
deep learning with rule-based approaches for citation classification. In: Proc. of the 8thWorkshop on Bibliometric-enhanced Information Retrieval, CEUR-WS.org (2019) 125–131[39] Shvets, A.: Improving scientific article visibility by neural title simplification. In: Proc.
of the 8th Workshop on Bibliometric-enhanced Information Retrieval, CEUR-WS.org (2019)
140–147[40] Vagliano, I., Nazir, S.: Recommending multimedia educational resources on the MOVING
platform. In: Proc. of the 8th Workshop on Bibliometric-enhanced Information Retrieval,
CEUR-WS.org (2019) 148–158[41] Atanassova, I.: Beyond metadata: the new challenges in mining scientific papers. In: Proc.
of the 8th Workshop on Bibliometric-enhanced Information Retrieval, CEUR-WS.org (2019)
8–13

ACM SIGIR Forum28

Vol. 53 No. 1 June 2019