Large expert-curated database for benchmarking document similarity detection in biomedical literature search
Authors: Brown Peter ,   RELISH Consortium, Tan Aik-Choon , El-Esawi Mohamed A , Liehr Thomas , Blanck Oliver , Gladue Douglas P , Almeida Gabriel M F , Cernava Tomislav , Sorzano Carlos O , Yeung Andy W K , Engel Michael S , Chandrasekaran Arun Richard , Muth Thilo , Staege Martin S , Daulatabad Swapna V , Widera Darius , Zhang Junpeng , Meule Adrian , Honjo Ken , Pourret Olivier , Yin Cong-Cong , Zhang Zhongheng , Cascella Marco , Flegel Willy A , Goodyear Carl S , van Raaij Mark J , Bukowy-Bieryllo Zuzanna , Campana Luca G , Kurniawan Nicholas A , Lalaouna David , Hüttner Felix J , Ammerman Brooke A , Ehret Felix , Cobine Paul A , Tan Ene-Choo , Han Hyemin , Xia Wenfeng , McCrum Christopher , Dings Ruud P M , Marinello Francesco , Nilsson Henrik , Nixon Brett , Voskarides Konstantinos , Yang Long , Costa Vincent D , Bengtsson-Palme Johan , Bradshaw William , Grimm Dominik G , Kumar Nitin , Martis Elvis , Prieto Daniel , Sabnis Sandeep C , Amer Said E D R , Liew Alan W C , Perco Paul , Rahimi Farid , Riva Giuseppe , Zhang Chongxing , Devkota Hari P , Ogami Koichi , Basharat Zarrin , Fierz Walter , Siebers Robert , Tan Kok-Hian , Boehme Karen A , Brenneisen Peter , Brown James A L , Dalrymple Brian P , Harvey David J , Ng Grace , Werten Sebastiaan , Bleackley Mark , Dai Zhanwu , Dhariwal Raman , Gelfer Yael , Hartmann Marcus D , Miotla Pawel , Tamaian Radu , Govender Pragashnie , Gurney-Champion Oliver J , Kauppila Joonas H , Zhang Xiaolei , Echeverría Natalia , Subhash Santhilal , Sallmon Hannes , Tofani Marco , Bae Taeok , Bosch Oliver , Cuív Páraic O , Danchin Antoine , Diouf Barthelemy , Eerola Tuomas , Evangelou Evangelos , Filipp Fabian V , Klump Hannes , Kurgan Lukasz , Smith Simon S , Terrier Olivier , Tuttle Neil , Ascher David B , Janga Sarath C , Schulte Leon N , Becker Daniel , Browngardt Christopher , Bush Stephen J , Gaullier Guillaume , Ide Kazuki , Meseko Clement , Werner Gijsbert D A , Zaucha Jan , Al-Farha Abd A , Greenwald Noah F , Popoola Segun I , Rahman Md Shaifur , Xu Jialin , Yang Sunny Y , Hiroi Noboru , Alper Ozgul M , Baker Chris I , Bitzer Michael , Chacko George , Debrabant Birgit , Dixon Ray , Forano Evelyne , Gilliham Matthew , Kelly Sarah , Klempnauer Karl-Heinz , Lidbury Brett A , Lin Michael Z , Lynch Iseult , Ma Wujun , Maibach Edward W , Mather Diane E , Nandakumar Kutty S , Ohgami Robert S , Parchi Piero , Tressoldi Patrizio , Xue Yu , Armitage Charles , Barraud Pierre , Chatzitheochari Stella , Coelho Luis P , Diao Jiajie , Doxey Andrew C , Gobet Angélique , Hu Pingzhao , Kaiser Stefan , Mitchell Kate M , Salama Mohamed F , Shabalin Ivan G , Song Haijun , Stevanovic Dejan , Yadollahpour Ali , Zeng Erliang , Zinke Katharina , Alimba C G , Beyene Tariku J , Cao Zehong , Chan Sherwin S , Gatchell Michael , Kleppe Andreas , Piotrowski Marcin , Torga Gonzalo , Woldesemayat Adugna A , Cosacak Mehmet I , Haston Scott , Ross Stephanie A , Williams Richard , Wong Alvin , Abramowitz Matthew K , Effiong Andem , Lee Senhong , Abid Muhammad Bilal , Agarabi Cyrus , Alaux Cedric , Albrecht Dirk R , Atkins Gerald J , Beck Charles R , Bonvin A M J J , Bourke Emer , Brand Thomas , Braun Ralf J , Bull James A , Cardoso Pedro , Carter Dee , Delahay Robin M , Ducommun Bernard , Duijf Pascal H G , Epp Trevor , Eskelinen Eeva-Liisa , Fallah Mazyar , Farber Debora B , Fernandez-Triana Jose , Feyerabend Frank , Florio Tullio , Friebe Michael , Furuta Saori , Gabrielsen Mads , Gruber Jens , Grybos Malgorzata , Han Qian , Heinrich Michael , Helanterä Heikki , Huber Michael , Jeltsch Albert , Jiang Fan , Josse Claire , Jurman Giuseppe , Kamiya Haruyuki , de Keersmaecker Kim , Kristiansson Erik , de Leeuw Frank-Erik , Li Jiuyong , Liang Shide , Lopez-Escamez Jose A , Lopez-Ruiz Francisco J , Marchbank Kevin J , Marschalek Rolf , Martín Carmen S , Miele Adriana E , Montagutelli Xavier , Morcillo Esteban , Nicoletti Rosario , Niehof Monika , O’Toole Ronan , Ohtomo Toshihiko , Oster Henrik , Palma Jose-Alberto , Paterson Russell , Peifer Mark , Portilla Maribel , Portillo M C , Pritchard Antonia L , Pusch Stefan , Raghava Gajendra P S , Roberts Nicola J , Ross Kehinde , Schuele Birgitt , Sergeant Kjell , Shen Jun , Stella Alessandro , Sukocheva Olga , Uversky Vladimir N , Vanneste Sven , Villet Martin H , Viveiros Miguel , Vorholt Julia A , Weinstock Christof , Yamato Masayuki , Zabetakis Ioannis , Zhao Xin , Ziegler Andreas , Aizat Wan M , Atlas Lauren , Bridges Kristina M , Chakraborty Sayan , Deschodt Mieke , Domingues Helena S , Esfahlani Shabnam S , Falk Sebastian , Guisado J L , Kane Nolan C , Kueberuwa Gray , Lau Colleen L , Liang Dai , Liu Enwu , Luu Andreas M , Ma Chuang , Ma Lisong , Moyer Robert , Norris Adam D , Panthee Suresh , Parsons Jerod R , Peng Yousong , Pinto Inês Mendes , Reschke Cristina R , Sillanpää Elina , Stewart Christopher J , Uhle Florian , Yang Hui , Zhou Kai , Zhu Shu , Ashry Mohamed , Bergsland Niels , Berthold Maximilian , Chen Chang-Er , Colella Vito , Cuypers Maarten , Eskew Evan A , Fan Xiao , Gajda Maksymilian , Gonzálezlez-Prendes Rayner , Goodin Amie , Graham Emily B , Groen Ewout J N , Gutiérrez-Sacristán Alba , Habes Mohamad , Heffler Enrico , Higginbottom Daniel B , Janzen Thijs , Jayaraman Jayakumar , Jibb Lindsay A , Jongen Stefan , Kinyanjui Timothy , Koleva-Kolarova Rositsa G , Li Zhixiu , Liu Yu-Peng , Lund Bjarte A , Lussier Alexandre A , Ma Liping , Mier Pablo , Moore Matthew D , Nagler Katja , Orme Mark W , Pearson James A , Prajapati Anilkumar S , Saito Yu , Tröder Simon E , Uchendu Florence , Verloh Niklas , Voutchkova Denitza D , Abu-Zaid Ahmed , Bakkach Joaira , Baumert Philipp , Dono Marcos , Hanson Jack , Herbelet Sandrine , Hobbs Emma , Kulkarni Ameya , Kumar Narendra , Liu Siqi , Loft Nikolai D , Reddan Tristan , Senghore Thomas , Vindin Howard , Xu Haotian , Bannon Ross , Chen Branson , Cheung Johnny T K , Cooper Jeffrey , Esnakula Ashwini K , Feghali Karine A , Ghelardi Emilia , Gnasso Agostino , Horbar Jeffrey , Lai Hei M , Li Jian , Ma Lan , Ma Ruiyan , Pan Zihang , Peres Marco A , Pranata Raymond , Seow Esmond , Sydes Matthew , Testoni Ines , Westermair Anna L , Yang Yongliang , Afnan Masoud , Albiol Joan , Albuquerque Lucia G , Amiya Eisuke , Amorim Rogerio M , An Qianli , Andersen Stig U , Aplin John D , Argyropoulos Christos , Asmann Yan W , Assaeed Abdulaziz M , Atanasov Atanas G , Atchison David A , Avery Simon V , Avillach Paul , Baade Peter D , Backman Lars , Badie Christophe , Baldi Alfonso , Ball Elizabeth , Bardot Olivier , Barnett Adrian G , Basner Mathias , Batra Jyotsna , Bazanova O M , Beale Andrew , Beddoe Travis , Bell Melanie L , Berezikov Eugene , Berners-Price Sue , Bernhardt Peter , Berry Edward , Bessa Theolis B , Billington Craig , Birch John , Blakely Randy D , Blaskovich Mark A T , Blum Robert , Boelaert Marleen , Bogdanos Dimitrios , Bosch Carles , Bourgoin Thierry , Bouvard Daniel , Boykin Laura M , Bradley Graeme , Braun Daniel , Brownlie Jeremy , Brühl Albert , Burt Austin , Butler Lisa M , Byrareddy Siddappa N , Byrne Hugh J , Cabantous Stephanie , Calatayud Sara , Candal Eva , Carlson Kimberly , Casillas Sònia , Castelvetro Valter , Caswell Patrick T , Cavalli Giacomo , Cerovsky Vaclav , Chagoyen Monica , Chen Chang-Shi , Chen Dong F , Chen Hao , Chen Hui , Chen Jui-Tung , Chen Yinglong , Cheng Changxiu , Cheng Jianlin , Chinapaw Mai , Chinopoulos Christos , Cho William C S , Chong Lillian , Chowdhury Debashish , Chwalibog Andre , Ciresi A , Cockcroft Shamshad , Conesa Ana , Cook Penny A , Cooper David N , Coqueret Olivier , Corea Enoka M , Costa Elisio , Coupland Carol , Crawford Stephanie Y , Cruz Aparecido D , Cui Huijuan , Cui Qiang , Culver David C , D’Angiulli Amedeo , Dahms Tanya E S , Daigle France , Dalgleish Raymond , Danielsen Håvard E , Darras Sébastien , Davidson Sean M , Day David A , Degirmenci Volkan , Demaison Luc , Devriendt Koenraad , Ding Jiandong , Dogan Yunus , Dong X C , Donner Claudio F , Dressick Walter , Drevon Christian A , Duan Huiling , Ducho Christian , Dumaz Nicolas , Dwarakanath Bilikere S , Ebell Mark H , Eisenhardt Steffen , Elkum Naser , Engel Nadja , Erickson Timothy B , Fairhead Michael , Faville Marty J , Fejzo Marlena S , Festa Fernanda , Feteira Antonio , Flood-Page Patrick , Forsayeth John , Fox Simon A , Franks Steven J , Frentiu Francesca D , Frilander Mikko J , Fu Xinmiao , Fujita Satoshi , Galea Ian , Galluzzi Luca , Gani Federica , Ganpule Arvind P , García-Alix Antonio , Gedye Kristene , Giordano Maurizio , Giunta Cecilia , Gleeson Paul A , Goarant Cyrille , Gong Haipeng , Gora Diop , Gough Michael J , Goyal Ravinder , Graham Kathryn E , Grande-Pérez Ana , Graves Patricia M , Greidanus Harm , Grice Darren , Grunau Christoph , Gumulya Yosephine , Guo Yabin , Gurevich Vsevolod V , Gusev Oleg , Hacker Elke , Hage Steffen R , Hagen Guy , Hahn Steven , Haller Dagmar M , Hammerschmidt Sven , Han Jianwei , Han Renzhi , Handfield Martin , Hapuarachchi Hapuarachchige C , Harder Timm , Hardingham Jennifer E , Heck Michelle , Heers Marcel , Hew Khe F , Higuchi Yohei , Hilaire Cynthia St , Hilton Rachel , Hodzic Enisa , Hone Andrew , Hongoh Yuichi , Hu Guoku , Huber Heinz P , Hueso Luis E , Huirne Judith , Hurt Lisa , Idborg Helena , Ikeo Kazuho , Ingley Evan , Jakeman Philip M , Jensen Arne , Jia Hong , Jia Husen , Jia Shuqin , Jiang Jianping , Jiang Xingyu , Jin Yi , Jo Daehyun , Johnson Andrew M , Johnston Marie , Jonscher Karen R , Jorens Philippe G , Jorgensen Jens O L , Joubert Johan W , Jung Sin-Ho , Junior Antonio M , Kahan Thomas , Kamboj Sunjeev K , Kang Yong-Kook , Karamanos Yannis , Karp Natasha A , Kelly Ryan , Kenna Ralph , Kennedy Jonathan , Kersten Birgit , Khalaf Roy A , Khalid Javaria M , Khatlani T , Khider Tarig , Kijanka Gregor S , King Sarah R B , Kluz Tomasz , Knox Paul , Kobayashi Tatsuya , Koch Karl-Wilhelm , Kohonen-Corish Maija R J , Kong Xiangpeng , Konkle-Parker Deborah , Korpela Kalevi M , Kostrikis Leondios G , Kraiczy Peter , Kratz Harald , Krause Günter , Krebsbach Paul H , Kristensen Søren R , Kumari Prerna , Kunimatsu Akira , Kurdak Hatice , Kwon Young D , Lachat Carl , Lagisz Malgorzata , Laky Brenda , Lammerding Jan , Lange Matthias , Larrosa Mar , Laslett Andrew L , LeClair Elizabeth E , Lee Kyung-Woo , Lee Ming-Yih , Lee Moon-Soo , Li Genyuan , Li Jiansheng , Lieb Klaus , Lim Yau Y , Lindsey Merry L , Line Paul-Dag , Liu Dengcai , Liu Fengbin , Liu Haiyan , Liu Hongde , Lloyd Vett K , Lo Te-Wen , Locci Emanuela , Loidl Josef , Lorenzen Johan , Lorkowski Stefan , Lovell Nigel H , Lu Hua , Lu Wei , Lu Zhiyong , Luengo Gustavo S , Lundh Lars-Gunnar , Lysy Philippe A , Mabb Angela , Mack Heather G , Mackey David A , Mahdavi S R , Maher Pamela , Maher Toby , Maity Sankar N , Malgrange Brigitte , Mamoulakis Charalampos , Mangoni Arduino A , Manke Thomas , Manstead Antony S R , Mantalaris Athanasios , Marsal Jan , Marschall Hanns-Ulrich , Martin Francis L , Martinez-Raga Jose , Martinez-Salas Encarnacion , Mathieu Daniel , Matsui Yoichi , Maza Elie , McCutcheon James E , McKay Gareth J , McMillan Brian , McMillan Nigel , Meads Catherine , Medina Loreta , Merrick B Alex , Metzger Dennis W , Meunier Frederic A , Michaelis Martin , Micheau Olivier , Mihara Hisaaki , Mintz Eric M , Mizukami Takuo , Moalic Yann , Mohapatra D P , Monteiro Antonia , Montes Matthieu , Moran John V , Morozov Sergey Y , Mort Matthew , Murai Noriyuki , Murphy Denis J , Murphy Susan K , Murray Shauna A , Naganawa Shinji , Nammi Srinivas , Nasios Grigorios , Natoli Roman M , Nguyen Frederique , Nicol Christine , van Nieuwerburgh Filip , Nilsen Erlend B , Nobile Clarissa J , O’Mahony Margaret , Ohlsson Sophie , Olatunbosun Oluremi , Olofsson Per , Ortiz Alberto , Ostrikov Kostya , Otto Siegmar , Outeiro Tiago F , Ouyang Songying , Paganoni Sabrina , Page Andrew , Palm Christoph , Paradies Yin , Parsons Michael H , Parsons Nick , Pascal Pigny , Paul Elisabeth , Peckham Michelle , Pedemonte Nicoletta , Pellizzon Michael A , Petrelli M , Pichugin Alexander , Pinto Carlos J C , Plevris John N , Pollesello Piero , Polz Martin , Ponti Giovanna , Porcelli Piero , Prince Martin , Quinn Gwendolyn P , Quinn Terence J , Ramula Satu , Rappsilber Juri , Rehfeldt Florian , Reiling Jan H , Remacle Claire , Rezaei Mohsen , Riddick Eric W , Ritter Uwe , Roach Neil W , Roberts David D , Robles Guillermo , Rodrigues Tiago , Rodriguez Cesar , Roislien Jo , Roobol Monique J , Rowe J Alexandra , Ruepp Andreas , van Ruitenbeek Jan , Rust Petra , Saad Sonia , Sack George H , Santos Manuela , Saudemont Aurore , Sava Gianni , Schrading Simone , Schramm Alexander , Schreiber Martin , Schuler Sidney , Schymkowitz Joost , Sczyrba Alexander , Seib Kate L , Shi Han-Ping , Shimada Tomohiro , Shin Jeon-Soo , Shortt Colette , Silveyra Patricia , Skinner Debra , Small Ian , Smeets Paul A M , So Po-Wah , Solano Francisco , Sonenshine Daniel E , Song Jiangning , Southall Tony , Speakman John R , Srinivasan Mandyam V , Stabile Laura P , Stasiak Andrzej , Steadman Kathryn J , Stein Nils , Stephens Andrew W , Stewart Douglas I , Stine Keith , Storlazzi Curt , Stoynova Nataliya V , Strzalka Wojciech , Suarez Oscar M , Sultana Taranum , Sumant Anirudha V , Summers Mathew J , Sun Gang , Tacon Paul , Tanaka Kozo , Tang Haixu , Tanino Yoshinori , Targett-Adams Paul , Tayebi Mourad , Tayyem Reema , Tebbe Christoph C , Telfer Evelyn E , Tempel Wolfram , Teodorczyk-Injeyan Julita A , Thijs Gert , Thorne Sally , Thrift Amanda G , Tiffon Celine , Tinnefeld Philip , Tjahjono Daryono H , Tolle Fabrice , Toth Ervin , del Tredici Andria L , Tsapas Apostolos , Tsirigotis Konstantinos , Turak Ayse , Tzotzos George , Udo Edet E , Utsumi Toshiaki , Vaidyanathan Subramanian , Vaillant Michel , Valsesia Armand , Vandenbroucke Roosmarijn E , Veiga Feliciano H , Vendrell Marc , Vesk Peter A , Vickers Paul , Victor Victor M , Villemur Richard , Vohl Marie-Claude , Voolstra Christian R , Vuillemin Anne , Wakelin Steven , Waldron Levi , Walsh Laurence J , Wang Amanda Y , Wang Fuan , Wang Yun , Watanabe Yoichi , Weigert Andreas , Wen Jet-Chau , Wham Carol , White Ethan P , Wiener Jan , Wilharm Gottfried , Wilkinson Simon , Willmann Raffaella , Wilson Coralie , Wirth Brunhilde , Wojan Timothy R , Wolff Mathieu , Wong Bryan M , Wu Tzu-Wei , Wuerbel Hanno , Xiao Xiangshu , Xu Dong , Xu J W , Xu Jianping , Xue Bin , Yalcin Suayib , Yan Hong , Yang En-Cheng , Yang Shiqi , Yang Wei , Ye Yuzhen , Ye Zhi-Qiang , Yli-Kauhaluoma Jari , Yoneyama Hiroshi , Yu Ying , Yuan Guo-Cheng , Yuh Chiou-Hwa , Zaccolo Manuela , Zeng Chen , Zevnik Branko , Zhang Chi , Zhang Li , Zhang Li , Zhang Yingkai , Zhang Yusen , Zhang Zhiyong , Zhang Zhong-Yin , Zhao Yuan , Zhou Min , Zuberbier Torsten , Aanei Carmen M , Ahmad Rafi , Al-Lawama Manar , Alanio Alexandre , Allardyce Judith , Alonso-Caneiro David , Atack John M , Baier Dirk , Bansal Abhisheka , Benezeth Yannick , Berbesque Colette , Berrevoet Frederik , Biedermann Peter H W , Bijleveld Erik , Bittner Florian , Blombach Fabian , van den Bos Wouter , Boudreau Shellie A , Bramoweth Adam D , Braubach Oliver , Cai Yufeng , Campbell Matthew , Cao Zanxia , Catry Thibault , Chen Xin , Cheng Shuiqin , Chung Hee-Jung , Chávez-Fumagalli Miguel A , Conway Aaron , Costa Bruno M , Cyr Normand , Dean Lorraine T , Denzel Martin S , Dlamini S V , Dudley Kevin J , Dufies Maeva , Ecke Thorsten , Eckweiler Denitsa , Eixarch Elisenda , El-Adawy Hosny , Emmrich Julius V , Eustace Alex J , Falter-Wagner Christine M , Fuss Johannes , Gao Jianzhao , Gill Martin R , Gloyn Liz , Goggs Robert , Govinden Usha , Greene Garrett , Greiff Victor , Grundle D S , Grüneberg Patrick , Gumede Nicksy , Haore Gbaguidi , Harrison Pille , Hoenner Xavier , Hojsgaard Diego , Hori Hikaru , Ikonomopoulou Maria P , Jeurissen Patrick , Johnson Daniel M , Kabra Dhiraj , Kamagata Koji , Karmakar Chandan , Kasian Olga , Kaye Linda K , Khan Murad M , Kim Yong-Min , Kish J K , Kobold Sebastian , Kohanbash Gary , Kohls Gregor , Kugler Jan-Michael , Kumar Gyanendra , Lacy-Colson Jon , Latif Asam , Lauschke Volker M , Li Bingling , Lim Chinten J , Liu Fang , Liu Xiaodong , Lu Jin-Jian , Lu Qiang , Mahavadi Poornima , Marzocchi Ugo , McGarrigle Christine A , van Meerten Tom , Min Rogier , Moal Iain , Molari Massimiliano , Molleman Lucas , Mondal Saiful R , van de Mortel Thea , Moss W N , Moultos Othonas A , Mukherjee Maheswari , Nakayama Kazuhiko , Narayan Edward , Navaratnarajah  , Neumann Philipp-Alexander , Nie Jiyun , Nie Yingjiu , Niemeyer Frank , Nolan Fiona , Nwaiwu Ogueri , Oldenmenger Wendy H , Olumayede Emmanuel , Ou Jianhong , Pallebage-Gamarallage Menuka , Pearce Simon P , Pelkonen Tuula , Pelleri Maria C , Pereira Joana L , Pheko Mpho , Pinto Karina A , Piovesan Allison , Pluess Michael , Podolsky Illya M , Prescott Julie , Qi Dongchen , Qi Xingshun , Raikou Vaia D , Ranft Andreas , Rhodes Johanna , Rotge Jean-Yves , Rowe Anna D , Saggar Manish , Schuon Robert A , Shahid Shaouli , Shalchyan Vahid , Shirvalkar Prasad , Shiryayev Oleg , Singh Jugpreet , Smout Michael J , Soares António , Song Chunjiao , Srivastava Kshitij , Srivastava Rupesh K , Sun Jim , Szabo Attila , Szymanski Wiktor , Tai Chan N P , Takeuchi Hisashi , Tanadini-Lang S , Tang Fei , Tao Wanyin , Theron G , Tian Chang F , Tian Yu-Shi , Tuttle Lisa M , Valenti Anna , Verlot Pierre , Walker Mirella , Wang Jun , Welter Danielle , Winslade Matthew , Wu Dalei , Wu Yi-Rui , Xiao Han , Xu Beisi , Xu Juan , Xu Ziyue , Yang Dongdong , Yang Mingjun , Yankilevich Patricio , You Yuyi , Yu Chenglong , Zhan Jian , Zhang Gong , Zhang Kai , Zhang Tuo , Zhang Yi , Zhao Guoyan , Zhao Jing , Zhou Xiaofan , Zhu Zhenxing , Ajani Penelope A , Anazodo Udunna C , Bagloee Saeed A , Bail Kasia , Bar Ido , Bathelt Joe , Benkeser David , Bernier Meghan L , Blanchard Adam M , Boakye Dominic W , Bonatsos Vasileios , Boon Michele H , Bouboulis George , Bromfield Elizabeth , Brown Joshua , Bul Kim C M , Burton Kathryn J , Butkowski Eugene G , Carroll Grace , Chao Fengqing , Charrier Elisabeth E , Chen Xiaoyin , Chen Yu-Chih , Chenguang  , Choi Jane R , Christoffersen Tore , Comel João C , Cosse Cyril , Cui Yanru , van Dessel Pieter , Dhaval  , Diodato Daria , Duffey Maelle , Dutt Avik , Egea Luis G , El-Said Mohammed , Faye Martin , Fernandez-Fernandez Beatriz , Foley Kieran G , Founou Luria L , Fu Fan , Gadelkareem Rabea A , Galimov Evgeny , Garip Gulcan , Gemmill Alison , Gouil Quentin , Grey James , Gridneva Zoya , Grothe Michel J , Grébert Théophile , Guerrero Fabricio , Guignard Léo , Haenssgen Marco J , Hasler David , Holgate Joan Y , Huang Ancheng , Hulse-Kemp Amanda M , Jean-Quartier Claire , Jeon Sang-Min , Jia Yangyang , Jutzeler Catherine , Kalatzis Panagiotis , Karim Masud , Karsay Kathrin , Keitel Anne , Kempe Andreas , Keown Jeremy R , Khoo Chin M , Khwaja Nyil , Kievit Rogier A , Kosanic Aleksandra , Koutoukidis Dimitrios A , Kramer Paul , Kumar Dilip , Kırağ Nükhet , Lanza Giuseppe , Le Thuc D , Leem Jung W , Leightley Daniel , Leite Andreia , Lercher Lukas , Li Ying , Lim Renly , Lima Luiz R A , Lin Li , Ling Tong , Liu Yuchen , Liu Zhonghua , Lu Yao , Lum Fok M , Luo Hang , Machhi Jatin , Macleod Angus , Macwan Isaac , Madala Hanumantha R , Madani Nima , de Maio Nicola , Makowiecki Kalina , Mallinson Daniel J , Margelyte Ruta , Maria Caracausi , Markonis Y , Marsili Luca , Mavoa Suzanne , McWilliams Lorna , Megersa Moa , Mendes Caetano S M , Menichetti Julia , Mercieca-Bebber Rebecca , Miller Jack J , Minde David-Paul M , Minges Alexander , Mishra Eleanor , Mishra Virendra R , Moores Carly , Morrice Nicola , Moskalensky Alexander E , Navarin Nicolò , Negera Edessa , Nolet Philippe , Nordberg Ana , Nordén Rickard , Nowicki Jessica P , Olova Nelly , Olszewski Paweł , Onzima Robert , Pan Chih-Long , Park Charny , Park Dong Ik , Park Seyoung , Patil Chandrashekhar D , Pedro Sansoa A , Perry Samuel R , Peter Jessica , Peterson Brent M , Pezzuolo Andrea , Pozdnyakov Ilya , Qian Siyu , Qin Lei , Rafe Ali , Raote Ishier , Raza Ali , Rebl Henrike , Refai Osama , Regan Tim , Richa Tambi , Richardson Mark F , Robinson K R , Rossoni Luca , Rouet Romain , Safaei Soroush , Schneeberger Pierre H H , Schwotzer Daniela , Sebastian Agata , Selinski Jennifer , Seltmann Stefanie , Sha Feng , Shalev Nir , Shang Jin-Long , Singer Josef , Singh Mandeep , Smith Taylor , Solomon-Moore Emma , Song Lijuan , Soraggi Samuele , Stanley Ryan , Steckhan Nico , Strobl Frederic , Subissi Lorenzo , Supriyanto Irwan , Surve Chinmay R , Suzuki Tomo , Syme Caitlin , Sörelius Karl , Tang Young , Tantawy Marwa , Tennakoon Sumudu , Teseo Serafino , Toelzer Christine , Tomov Nikola , Tovar Miguel , Tran Linh , Tripathi Sushil , Tuladhar Anil M , Ukubuiwe Azubuike C , Ung Carolina O L , Valgepea Kaspar , Vatanparast Hamid , Vidal Arnau , Wang Fang , Wang Qing , Watari Ricky , Webster Rebecca , Webster Ruth , Wei Junnian , Wibowo David , Wingenbach Tanja S H , Xavier Rose M , Xiao Shumin , Xiong Peng , Xu Shicai , Xu Shilin , Yao Ruifeng , Yao Wen , Yin Qinan , Yu Yongbo , Zaitsu Masayoshi , Zeineb Zian , Zhan Xiao-Yong , Zhang Jilei , Zhang Rongqiang , Zhang Wei , Zhang Xianglilan , Zheng Shan , Zhou Bailing , Zhou Xiaoyan , Ahmad Haroon , Akinwumi Sayo A , Albery Gregory F , Alhowimel Ahmed , Ali Junaid , Alshehri Mansour , Alsuhaibani Mohammed , Anikin Andrey , Azubuike Samuel O , Bach-Mortensen Anders , Baltiansky Lior , Bartas Martin , Belachew Kiflemariam Y , Bhardwaj Vivek , Binder Karin , Bland Nicholas S , Boah Michael , Bullen Benjamin , Calabrò Giovanna E , Callahan Tiffany J , Cao Bing , Chalmers Kelsey , Chang Wei , Che Zhengping , Chen Andrew T Y , Chen Haimin , Chen Huaming , Chen Youning , Chen Zhao , Choi YoungRok , Chowdhury Mohiuddin A K , Christensen Martin R , Cooke Robert S C , Cottini Marzia , Covington Natalie V , Cunningham Catriona , Delarocque Julien , Devos Lucie , Dhar Aurup R , Ding Ke-Feng , Dong Kexian , Dong Zheng , Dreyer Niklas , Ekstrand Chelsea , Fardet Tanguy , Feleke Berhanu E , Feurer Thomas , Freitas Angela , Gao Tian , Asefa N G , Giganti Francesco , Grabowski Piotr , Guerra-Mora José R , Guo Chengying , Guo Xinyi , Gupta Himanshu , He Shuonan , Heijne Marloes , Heinemann Stephanie , Hogrebe Alexander , Huang Zhengping , Iskander-Rizk Sophinese , Iyer Lavanya M , Jahan Yasmin , James Ameh S , Joel Emmanuel , Joffroy Bastian , Jégousse Clara , Kambondo George , Karnati Priyanka , Kaya Cihan , Ke An , Kelly Daniel , Kickert Rob , Kidibule Peter E , Kieselmann Jennifer P , Kim Hyeon J , Kitazawa Takeshi , Lamberts Aniek , Li You , Liang Huakang , Linn Sabrina N , Litfin Thomas , Liusuo Wang , Lygirou Vasiliki , Mahato Ajay K , Mai Zhi-Ming , Major Rupert W , Mali Samira , Mallis Panagiotis , Mao Wenzhi , Mao Wenzhi , Marvin-Dowle Katie , Mason Leanda D , Merideth Ben , Merino-Plaza Maria J , Merlaen Britt , Messina Rossella , Mishra Anand K , Muhammad Junaid , Musinguzi Conrad , Nanou Afroditi , Naqash Amreen , Nguyen Joe T , Nguyen Thi T H , Ni Duan , Nida  , Notcovich Shirli , Ohst Barnabas , Ollivier Quinn R , Osses Daniël F , Peng Xiangda , Plantinga Arnoud , Pulia Michael , Rafiq Muhammad , Raman Ayush , Raucher-Chéné Delphine , Rawski Rafał , Ray Asit , Razak Lubna A , Rudolf Kevin , Rusch Peter , Sadoine Margaux L , Schmidt Axel , Schurr Roey , Searles Stephen , Sharma Saurab , Sheehan Barry , Shi Chunhu , Shohayeb Belal , Sommerlad Andrew , Strehlow Jan , Sun Xianbao , Sundar Raghav , Taherzadeh Ghazaleh , Tahir Nur D M , Tang Jun , Testa Jean , Tian Zhiqi , Tingting Qian , Verheijen Geert P , Vickstrom Casey , Wang Teng , Wang Xiaomin , Wang Zhenxing , Wei Pan , Wilson Alex , Wyart  , Yassine Abdul-Amir , Yousefzadeh Abbas , Zare Asma , Zeng Zhen , Zhang Chengrong , Zhang Haowen , Zhang Linxing , Zhang Tongchuan , Zhang Weijia , Zhang Zhe , Zhou Jianyu , Zhu Dongjie , Adamo Vincenzo , Adeyemo Adebolajo A , Aggelidou Maria , Al-Owaifeer Adi M , Al-Riyami Arwa Z , Alzghari Saeed K , Andersen Vibeke , Angus Kathryn , Asaduzzaman Muhammad , Asady Hadi , Ato Dai , Bai Xiaoyong , Baines Rebecca L , Ballantyne Maghan , Ban Bo , Beck Jill , Ben-Nafa Walid , Black Emma , Blancher Antoine , Blankstein Ron , Bodagh Neil , Borges Paulo A V , Brooks Anastasia , Brox-Ponce Josue , Brunetti Arturo , Canham Colin D , Carninci Piero , Carvajal Richard , Chang Shun C , Chao Jie , Chatterjee Pranab , Chen He , Chen Yi-Chun , Chhatriwalla Adnan K , Chikowe Ibrahim , Chuang Trees-Juen , Collevatti Rosane G , Valera-Cornejo Diego A , Cuenda Ana , Dao Myriam , Dauga Delphine , Deng Zaian , Devkota Kiran , Doan Lisa V , Elewa Yaser H A , Fan Dongsheng , Faruk Mohammed , Feifei Shi , Ferguson Trevor S , Fleres Francesco , Foster Emma J , Foster C Stephen , Furer Tzvi , Gao Yibo , Garcia-Rivera Enid J , Gazdar Adi , George Ronald B , Ghosh Sayantan , Gianchecchi Elena , Gleason Joshua M , Hackshaw Allan , Hall Adam , Hall Richard , Harper Paul , Hogg William E , Huang Guangqun , Hunter Kylie E , IJzerman Adriaan P , Jesus Carlos , Jian Gao , Jr James S Lewis , Kanj Souha S , Kaur Harsheen , Kelly Shona , Kheir Fayez , Kichatova V S , Kiyani Musa , Klein Reinhild , Kovesi Tom , Kraschnewski Jennifer L , Kumar Addanki P , Labutin Dmitry , Lazo-Langner Alejandro , Leclercq Guy , Li Maoteng , Li Qingchun , Li Tangliang , Li Yongzhe , Liao Wei-Ting , Liao Zheng-yin , Lin Jessica , Lizer J , Lobreglio Giambattista , Lowies Cher , Lu Cheng , Majeed Haroon , Martin Adam , Martinez-Sobrido Luis , Meresh Edwin , Middelveen Marianne , Mohebbi Alireza , Mota Jorge , Mozaheb Zahra , Muyaya Ley , Nandhakumar Amar , Ng Sheryl H X , Obeidat Monther , Oh Deog-Hwan , Owais Mohammed , Pace-Asciak Pia , Panwar Ajay , Park Caroline , Patterson Chris , Penagos-Tabaree Felipe , Pianosi Paolo T , Pinzi Valentina , Pridans Clare , Psaroulaki Anna , Pujala Ravi Kumar , Pulido-Arjona Leonardo , Qi Peng-Fei , Rahman Proton , Rai Nayanjot K , Rassaf Tienush , Refardt Julie , Ricciardi Walter , Riess Olaf , Rovas Alexandros , Sacks Frank M , Saleh Sherif , Sampson Christopher , Schmutz Axel , Sepanski Robert , Sharma Neeraj , Singh Manisha , Spearman Paul , Subramaniapillai Mehala , Swali Ritu , Tan Cher M , Tellechea Juan I , Thomas Lisa-Marie , Tong Xin , Vavvas Demetrios G , Veys Ralf , Vitriol Veronica , Wang Horng-Dar , Wang Jinhui , Wang Jiucun , Waugh Jason , Webb S A , Williams Brendan A , Workman Alan D , Xiang Tingxiu , Xie Li-Xin , Xu Jun , Xu Taosheng , Yang Chongjun , Yoon Jihoon G , Yuan Christina M , Zaritsky Arno , Zhang Yao , Zhao Haochen , Zuckerman Hannah , Lyu Ran , Pullan Wayne , Zhou Yaoqi 
Published: June 2019

Abstract
Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency–Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical research.