# Structure-Aware and Format-Enhanced Transformer for Accident Report Modeling
## 1 Introduction
Firstly, report lengths frequently surpass the maximum input length constraints of pre-trained language models, prompting researchers to utilize truncation or chunking strategies [17; 26; 27], potentially resulting in the loss of critical semantic elements such as operational conditions, human factors, and risk factors. Secondly, traditional hierarchical models such as HAN [28], Hi-Transformer [29] and ERNIE-SPARSE [30] typically rely on implicit hierarchical structures (\"word \(\rightarrow\) sentence \(\rightarrow\) document\"), limiting their capacity to explicitly capture the unique chapter-level relationships characteristic of accident reports, thereby failing to provide sufficient domain-specific semantic anchors necessary for global semantic modeling. Lastly, downstream tasks involving accident risk evolution and causal reasoning often span multiple paragraphs, and current general-purpose sparse attention mechanisms such as BigBird [31] and Longformer [32] lack domain-specific structural priors. This limitation leads to semantic drift when balancing long-range dependency capture with computational efficiency, potentially omitting critical details and adversely impacting performance on downstream tasks. Notably, the inherent hierarchical directory structure in accident investigation reports, such as chapter numbering like \"5.1.2\" and heading texts, provides natural semantic anchors that facilitate structured navigation for global information retrieval and relationship mining [33]; yet, this feature has not been fully utilized. Accordingly, the aim of this study is to develop a Transformer-based model that explicitly incorporates section numbers and heading semantics as structural priors, thereby capturing the hierarchical and semantic structures intrinsic to accident investigation reports and enhancing the extraction of meaningful accident intelligence from long and complex documents. Toward this end, we propose a novel Structure-Aware and Format-Enhanced Transformer (SAFE-Transformer). This model employs a dual-stream encoding architecture that decouples symbolic features of the hierarchical structure and heading text. The structural stream explicitly models the symbolic features of chapter numbering, while the semantic stream extracts heading textual features. Both streams dynamically and adaptively integrate with paragraph content through a gating mechanism. Unlike existing hierarchical models, SAFE-Transformer enhances semantic associations through symbolic-aware encoding of chapter numbering and utilizes hierarchical structural constraints to guide sparse attention computations, effectively reducing complexity while preserving essential structural information. To validate SAFE-Transformer, we use a multi-label classification task for accident intelligence, aiming to assign multiple safety intelligence labels, such as \"unsafe human behaviors\", \"unsafe material conditions\", and \"management deficiencies\", to accident reports. Experimental results demonstrate that the structural anchors constructed via symbolic-aware encoding, combined with semantic streams from headings and the dynamic gating fusion mechanism, effectively achieve adaptive integration of global and local contextual semantics, significantly enhancing multi-label classification performance. The principal objectives and contributions of this work are threefold:  
1. Proposing SAFE-Transformer as the first model to inject section numbers and heading semantics as explicit structural priors into a Transformer;
2. Validating the performance of SAFE-Transformer using a national-level accident dataset to demonstrate its practical effectiveness;
3. Showing how SAFE-Transformer alleviates key-information loss, context fragmentation, and long-tail effects through evaluation of a multi-label accident-intelligence benchmark.