# Structure-Aware and Format-Enhanced Transformer for Accident Report Modeling
## 6 Result
### Overall Performance Analysis
Longformer [32] showed moderate performance in our experiments, suggesting that merely expanding attention span is insufficient to accommodate the structural characteristics of vertical-domain texts like accident reports. Table 5 compares the P@5 performance of different models across accident reports of varying lengths. The data reveal a non-linear relationship between classification performance and document length for the long-text optimized models. When the sequence length is L \(\leq\) 512, RoBERTa [65] achieves the highest P@5 score of 73.13. However, as the sequence length increases to 4096, performance across all models declines, forming an inverted U-shaped trend. It is noteworthy that long-text-optimized models, including SAFE-Transformer, underperform the standard self-attention model RoBERTa [65] in short-text scenarios. This finding aligns with the conclusions of [PERSON] et al. [67]. However, in medium-length documents, long-text models consistently exhibit stronger performance. For instance, when the sequence length ranges from 1K to 2K tokens, models such as SAFE-Transformer, ERNIE-SPARSE [30], Longformer [32], and BigBird [31] all maintain P@5 scores above 70%, indicating strong robustness. We further examined the distribution of Micro/Macro AUC and F1 scores across different document lengths, as visualized in Figure 5. The inverted U-shaped performance fluctuation is consistently observed among all long-text-optimized models. To explore the underlying causes of this phenomenon, we conducted a detailed analysis from the perspectives of label distribution and attention mechanism design. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **L = [0, 512]** & **L = [512, 1024]** & **L = [1024, 2048]** & **L = [2048, 4096]** \\ \hline RoBERTa & 73.13 & 68.41 & 66.75 & 59.12 \\ Longformer & 69.43 & 70.83 & 71.32 & 69.16 \\ BigBird & 69.15 & 73.82 & 70.22 & 71.25 \\ ERNIE-SPARSE & 70.36 & 75.03 & 71.29 & 70.25 \\ SAFE-Transformer & 71.48 & 75.87 & 72.21 & 71.35 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: P@5 Performance on reports of varying lengths. Label Distribution Analysis: We analyzed the distribution of data labels in the test set across different document lengths, as shown in Figure 6. A pronounced long-tail distribution is observed in shorter documents, where high-frequency labels such as accident overview, emergency response, and liability attribution account for over 95% of the label occurrences (see Figure 6a). Further investigation into the sources of these short documents revealed that most of them originate from government information portals. These public-facing websites are often constrained in length and typically disclose only basic accident details, including the time, location, type, severity, casualty figures, emergency response process, and a brief statement on responsibility. Such texts tend to adopt a journalistic reporting style similar to that of open-domain news articles. In medium-length reports (1K-2K words), the long-tail effect remains evident but is noticeably mitigated (see Figure 6b,c). Labels describing causes of accidents--such as unsafe behaviors, hazardous environments, and organizational management deficiencies--appear with greater frequency. These documents generally cover a complete narrative chain: \"accident overview \(\rightarrow\) cause analysis \(\rightarrow\) responsibility determination \(\rightarrow\) corrective actions\", and are rich in informational content. Statistical analysis shows that 86% of these  
Figure 5: AUC and F1 performance curves across documents of varying lengths. (**a**) Micro-AUC scores versus document length, showing the inverted U-shaped trend with peak performance in medium-length documents (1K-2K tokens). (**b**) Macro-AUC scores versus document length, highlighting the impact on rare labels (e.g., L6 Legal Provisions) that improves in medium-length reports. (**c**) Micro-F1 scores versus document length, demonstrating similar non-linear patterns as AUC metrics.