CRAWLDoc: A Dataset for Robust Ranking of Bibliographic Documents
Authors: name: Fabian Karl; name: Ansgar Scherp
Published: 2025-06-04 10:52:55

Abstract: Publication databases rely on accurate metadata extraction from diverse web
sources, yet variations in web layouts and data formats present challenges for
metadata providers. This paper introduces CRAWLDoc, a new method for contextual
ranking of linked web documents. Starting with a publication's URL, such as a
digital object identifier, CRAWLDoc retrieves the landing page and all linked
web resources, including PDFs, ORCID profiles, and supplementary materials. It
embeds these resources, along with anchor texts and the URLs, into a unified
representation. For evaluating CRAWLDoc, we have created a new, manually
labeled dataset of 600 publications from six top publishers in computer
science. Our method CRAWLDoc demonstrates a robust and layout-independent
ranking of relevant documents across publishers and data formats. It lays the
foundation for improved metadata extraction from web documents with various
layouts and formats. Our source code and dataset can be accessed at
https://github.com/FKarl/CRAWLDoc.