MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMs
Authors: name: Zaid Alyafeai, name: Maged S. Al-Shaibani, name: Bernard Ghanem

Metadata extraction is essential for cataloging and preserving datasets,
enabling effective research discovery and reproducibility, especially given the
current exponential growth in scientific research. While Masader (Alyafeai et
al.,2021) laid the groundwork for extracting a wide range of metadata
attributes from Arabic NLP datasets' scholarly articles, it relies heavily on
manual annotation. In this paper, we present MOLE, a framework that leverages
Large Language Models (LLMs) to automatically extract metadata attributes from
scientific papers covering datasets of languages other than Arabic. Our
schema-driven methodology processes entire documents across multiple input
formats and incorporates robust validation mechanisms for consistent output.
Additionally, we introduce a new benchmark to evaluate the research progress on
this task. Through systematic analysis of context length, few-shot learning,
and web browsing integration, we demonstrate that modern LLMs show promising
results in automating this task, highlighting the need for further future work
improvements to ensure consistent and reliable performance. We release the
code: https://github.com/IVUL-KAUST/MOLE and dataset:
https://huggingface.co/datasets/IVUL-KAUST/MOLE for the research community.