# Structure-Aware and Format-Enhanced Transformer for Accident Report Modeling
## 6 Result
### Overall Performance Analysis
(**d**) Macro-F1 scores versus document length, revealing performance degradation on long-tail categories in very short (<512 tokens) and very long (>4K tokens) documents. reports are sourced from accident reporting and investigation systems maintained by local emergency management departments. In long-form reports (3K-4K words), the frequency of certain labels breaks the long-tail constraint, forming local peaks--for instance, label L6 (legal clauses). Further analysis reveals that such extended reports usually correspond to major accidents characterized by complex contributing factors, multiple violations, and challenges in assigning liability. These documents often contain repeated references to incident descriptions, unlawful conduct, and legal provisions, resulting in dense cross-references and posing significant learning challenges for models. Attention Mechanism Patterns: The baseline models evaluated in this study adopt differing attention mechanisms. RoBERTa [65] employs full attention (quadratic attention), allowing every token to interact with all others in the sequence. In contrast, long-text models reduce computational complexity through sparse attention mechanisms. To directly observe the distribution of attention weights across long-text models, we selected 200 short accident reports from the test set and visualized their attention distributions. Figure 7 presents attention heatmaps derived from this analysis. As shown in Figure 7, BigBird [31] combines global attention (focusing on special tokens such as [CLS]), sliding window attention (for local context), and random attention (random token interactions) [31, 68], resulting in several long-range focus regions within the heatmap. Longformer [32] applies a sliding window (w = 3) and task-relevant global attention; its fixed window yields broader attention coverage than BigBird [31]. During pretraining and fine-tuning, both BigBird and Longformer [32] prioritize long-range dependencies, but this may become redundant in short-text tasks and introduce instability through random attention. ERNIE-SPARSE [30] incorporates hierarchical sparse attention (Hierarchical Sparse Transformer) and self-attention regularization. In the heatmap, ERNIE-SPARSE exhibits a distinct hierarchical structure. However, in short texts, many parameters may remain inactive, leading to overparameterization. Figure 6: Label distribution across accident reports of varying lengths. (**a**) Short documents (L \(\leq\) 512): L1–L3 labels dominate (>95% occurrence), characteristic of government portal briefings. (**b**) Medium-short (512 \(<\) L \(\leq\) 1024): Long-tail distribution eases with emerging L7/L8 causal analysis. (**c**) Medium-long (1024 \(<\) L \(\leq\) 2048): Balanced labels form a complete narrative chain, predominant in local emergency reports. (**d**) Long documents (2048 \(<\) L \(\leq\) 4096): L6 (Legal Provisions) peaks due to complex liability cases. (**e**) Gini coefficient: Quantifies a U-shaped imbalance shift—extreme in shorts (L1–L3), balanced in mediums, and renewed in longs (L6-dominated). SAFE-Transformer adopts a hierarchical sparse attention mechanism with block-wise dynamic sparsity, constraining information flow within the same level (e.g., between words in a sentence) or across parent-child levels (e.g., between section anchors and sentence anchors), resulting in multi-level contextual aggregation. Its heatmap displays both wide attention coverage and clearly layered regions. However, this integration of local and global attention may become redundant in short-text processing [69; 70]. In summary, the suboptimal performance of long-text-optimized models in short-document classification tasks may be attributed not only to the long-tail label distribution, but also to fundamental mismatches between attention architecture design and the semantic  
Figure 7: Attention heatmaps of long-text optimized models on short accident reports. Color scale: dark blue represents low attention, bright yellow indicates high attention.