# Structure-Aware and Format-Enhanced Transformer for Accident Report Modeling
## 2 Literature Review
### Long Text Modeling Based on Transformer Architecture  
The traditional Transformer employs a self-attention mechanism, calculating associations between each token and all other tokens in a sequence, causing a notable decrease in inference speed and performance when processing lengthy texts [53; 54]. Especially in specialized domains with lengthy documents, information decay weakens semantic associations between distant tokens, often leading to neglected crucial long-range dependencies [54]. Current long-text modeling methods primarily include sparse attention mechanisms, hierarchical positional encodings, and chunk-based training strategies [32]. Balancing computational efficiency and semantic integrity remains the core challenge in long-text modeling. Early exploration of sparse attention strategies led to [PERSON] et al. [55] proposing the Sparse Transformer, which systematically reconstructed the dense self-attention matrix of conventional Transformers into sparse operations of lower computational complexity through predefined sparse patterns and sparse factorization frameworks (from \(O(n^{2})\) to \(O(n\sqrt{n})\)). However, this fixed sparsity pattern can limit flexible modeling of multi-granularity contexts. Subsequently, [PERSON] et al. [32] introduced the Longformer, which incorporates Dilated Sliding Window and Global Attention mechanisms to maintain linear computational complexity while supporting modeling sequences up to 32K characters (for language modeling tasks) or over 4K tokens (for downstream tasks). The sliding window in Longformer expands its receptive field through dilated sampling, capturing distant dependencies, while global attention highlights task-critical information alongside local context preservation. [PERSON] et al. [56] proposed the ETC (Extended Transformer Construction) model, dividing inputs into global sequences (such as paragraph summaries) and long sequences (detailed text) through a global-local attention mechanism. It restricts attention within local windows (e.g., 84 tokens), reducing computational complexity from quadratic to linear. Additionally, ETC introduced relative positional encodings, enhancing dynamic positional relationship capturing and explicitly encoding structured input, exhibiting strong performance with long texts and structured data. However, ETC faced efficiency issues and limitations in capturing local contexts for extremely long sequences. Addressing these shortcomings, [PERSON] et al. [31] introduced the BigBIRD model, extending ETC with a comprehensive sparse attention scheme, combining global tokens for global context, local windows for neighboring dependencies, and random attention for enhanced long-rangeinformation interactions, effectively reducing complexity to linear while preserving full attention representation capabilities. Contrasting sparse attention methods, hierarchical approaches adopt hierarchical iterative modeling architectures to aggregate semantics from local to global contexts, achieving multi-granularity representations [28; 53; 57]. [PERSON] et al. [28] proposed the Hierarchical Attention Network (HAN) for document classification, exploiting the natural hierarchical structure of documents (word \(\rightarrow\) sentence \(\rightarrow\) document). HAN leverages bidirectional GRUs to encode contextual information layer by layer, dynamically highlighting important content through attention mechanisms. Despite integrating implicit document structures, HAN suffers from one-way information flow and limited parallelization due to sequential dependencies inherent in RNNs, impairing efficiency in processing long documents. To overcome these limitations, [PERSON] et al. [29] developed the Hi-Transformer model, employing a bidirectional context information transmission mechanism. Initially, sentence-level Transformers independently encode local semantics. Subsequently, document-level Transformers compute global self-attention across sentences, integrating document themes and logic into new sentence representations.