Understanding High-complexity Technical and Regulatory Documents with State-of-the-Art Models: A Pilot Study
Authors: Bernardo Magnini, Alessandro Dal Pozzo, Roberto Zanoli (Fondazione Bruno Kessler, Rete Ferroviaria Italiana S.p.A, 2024)
Topic: In-domain IR and IE

Abstract
We explore the potential of state-of-the-art Large Language Models (LLMs) to reason on the content of high-complexity documents written in Italian. We focus on both technical documents (e.g., describing civil engineering works) and regulatory documents (e.g., describing procedures). While civil engineering documents contain crucial information that supports critical decision-making in construction, transportation and infrastructure projects, procedural documents outline essential guidelines and protocols that ensure efficient operations, adherence to safety standards and effective incident management. Although LLMs offer a promising solution for automating the extraction and comprehension of high-complexity documents, potentially transforming our interaction with technical information, LLMs may encounter significant challenges when processing such documents due to their complex structure, specialized terminology and strong reliance on graphical and visual elements. Moreover, LLMs are known to sometimes produce unexpected or incorrect analyses, a phenomenon referred to as hallucination. The goal of the paper is to conduct an assessment of LLM capacities along several dimensions, including the format of the document (i.e., selectable text PDFs versus scanned OCR PDFs), the structure of the documents (e.g., number of pages, date of the document), the graphical elements (e.g., tables, graphs, photos), the interpretation of text portions (e.g., make a summary), and the need of external knowledge (e.g., to interpret a mathematical expressions). To run the assessment, we took advantage of GPT-4omni, a large multi-modal model pre-trained on a variety of different data. Our findings suggest that there is great potential for real-world applications for high-complexity documents, although LLMs may still be susceptible to produce misleading information.

Introduction
Technical documents employed in civil engineering contain information essential for planning, designing and constructing structures that need to ensure safety and compliance with regulations. As an example, such high-complexity documents provide technical guidelines for managing the development of roads, bridges and other transport networks. Additionally, these documents are fundamental for public infrastructure projects, ensuring they serve the community effectively and safely. These documents are highly complex, particularly due to their multi-modal nature, where textual content is mixed with several graphical content. The written content can vary from simple explanations to very detailed technical instructions, often referring to specialized regulations. The visual elements typically include tables with numbers, math formulas and detailed drawings of engineering stuff, as well as photos from natural environments and rendering of a construction once realized. In addition, documents are available either in PDF format as scanned documents, or as PDFs processed with Optical Character Recognition (OCR) software, introducing an additional layer of complexity due to potential variations in text recognition quality. Finally, civil engineering technical documents are typically long, easily reaching hundreds of pages. Figure~fig:figure shows one of the many visual elements occurring in the technical documents (civil engineering projects in Italian) considered in this study. [htbp] \centering \includegraphics[width=\linewidth]{images/figure.png Similarly to technical documents, regulatory documents play an equally important role across the same sectors, as they outline the steps for managing incidents, supervising safety procedures and ensuring regulatory compliance. For example, railway procedural documents contain comprehensive instructions on handling incidents and supervising safety measures, introducing additional complexity through procedural frameworks. Although procedural documents lack the visual complexity typical of technical projects, such as the presence of figures, tables and graphs, they are dense with text, focusing on legal and procedural details. The paper investigates how state-of-the-art generative models are able to reason on the content of high-complexity technical and regulatory documents written in Italian. As generative models, both LLMs and Large Multimodal Models (LMMs), are rapidly becoming more and more powerful, our research questions aim at assessing their ability to extract and interpret key information, this way reducing the need for manual reviews by human experts. To this end, we have defined a simple question-answer evaluation framework tailored to technical and regulatory documents. As an example, we ask the model questions such as Provide a general summary of the technical specifications in the document and then we manually check the model answer. We also consider the potential for LLMs/LMMs to generate content that is not grounded to the document, an issue often referred to as model confabulations or hallucinations AUTHOR. To assess confabulations we included ``trap" questions mentioning non-existing objects in the document. Finally, the assessment considers both selectable text PDFs, which are extractable and editable, and scanned OCR PDFs, where text is derived from scanning or from OCR. A state-of-the-art survey on articles published between 2000 and 2021, focusing on the applications of Text Mining in the construction industry was presented in AUTHOR. AUTHOR and AUTHOR explored NLP application and development in construction. Various machine learning and deep learning-based NLP techniques, and their applications in construction research, are documented in AUTHOR. There are several potential real-world applications of LLMs in supporting and enhancing various sectors. Construction firms can exploit LLMs to assist in reviewing technical documents for safety regulations and building codes, helping simplifying compliance checks. Additionally, organizations with large document archives can leverage LLMs to identify potential inconsistencies or conflicts in procedures, providing valuable insights for further human review and ensuring adherence to unified operational protocols.

Conclusion
We showed that GPT-4omni has a high potential for analyzing technical and regulatory documents. However, the model tends to make factual errors, to generate inaccurate details and to provide misleading answers supported by technical explanations. These observations highlight potential limitations when handling long and complex documents, and further research is needed to better understand and address these challenges. Our study has some limitations that should be considered. Limited Sample Size. The evaluation was based on a dataset of four documents, which may not be representative of the broader range of technical documents. Query Format. We employed a multi-question prompt format, grouping multiple questions within a single prompt. We plan to explore an approach where each question is presented as an individual prompt. Examining Positional Bias. There is a possibility that the answer location within the document (beginning, middle, or end) might affect the model's performance. Contextual Sensitivity Analysis. The amount of context provided could influence GPT in answering questions related to specific document elements. We plan to systematically compare the model accuracy when presented with the entire document versus just the relevant page containing the answer. Playground vs. API Analysis. We primarily used the OpenAI API for evaluation. It would be valuable to explore whether analyzing documents through OpenAI's Playground interface yields similar results.