# Structure-Aware and Format-Enhanced Transformer for Accident Report Modeling
## 4 SAFE-Transformer Model
### Encoder  
The hierarchical encoder comprises a stack of \(N\) improved SAFE blocks, each consisting of three core sublayers: a hierarchical multi-head self-attention sublayer, a gated semantic fusion sublayer, and a feedforward sublayer.  
Input preprocessing: The encoder integrates word embeddings with Hierarchical Symbolic Format-aware Positional Encoding (HSFPE), which encodes both hierarchical position indices and symbolic format types (see Section 4.2) to provide structured positional priors for each token.  
Hierarchical multi-head self-attention sublayer: Using the dynamically generated sparse attention mask from Section 4.4, this sublayer computes interaction weights between tokens, allowing information to flow only among same-level sibling nodes and direct  
Figure 3: Sparse attention mask based on hierarchical structures. Rule 1: Tokens at the same hierarchical level are allowed to interact only if they share the same parent node across all upper levels. As shown in (**a**), tokens B, C, and H share the same parent node A at level 1, thus satisfying the intra-level interaction condition and should be permitted to attend to each other. Similarly, D and E, as well as F and G, are allowed to interact. However, interactions between D and F, D and G, E and F, or E and G are prohibited since their respective parent nodes differ (B vs. C). Rule 2: Bidirectional attention is allowed between parent and child nodes across adjacent hierarchical levels. For example, A \(\to\) B, A \(\to\) C, B \(\to\) A, and C \(\to\) A are all permitted. The final sparse attention mask is illustrated in (**b**).  
parent-child pairs. For instance, the attention mask for the heading \"5.1 Direct Cause\" permits access to its parent node \"5. Accident Causes\" (level 1), the heading itself (level 2), and its associated paragraph content (level 3).  
Gated semantic fusion sublayer: Inspired by the document context feedback mechanism proposed in [29], we implement a three-way semantic fusion via dynamic gating. Using the hierarchical index \(p\_id\), we retrieve from the encoder cache the parent node representation \(\mathbf{h}_{\text{parent}}\), the current heading representation \(\mathbf{h}_{\text{title}}\), and the weighted-pooled content representation \(\mathbf{h}_{\text{content}}\). The fusion is controlled by a gate vector \(\mathbf{g}=\big{[}g_{P},g_{t},g_{c}\big{]}\in[0,1]^{3}\), computed via a sigmoid activation:  
\[\mathbf{g}=\sigma\big{(}\mathbf{W}_{g}\cdot\big{[}\mathbf{h}_{\text{parent}} \ ;\mathbf{h}_{\text{title}}\ ;\mathbf{h}_{\text{content}}\ \big{]}+\mathbf{b}_{g}\big{)} \tag{6}\]  
Here, \(\mathbf{W}_{g}\in\mathbb{R}^{3d_{h}\times 3}\) is a learnable parameter matrix.  
The final fused representation is computed as:  
\[\mathbf{h}^{{}^{\prime}}_{\text{fused}}\ =g_{P}\cdot\mathbf{h}_{\text{parent}} \ +g_{t}\cdot\mathbf{h}_{\text{title}}\ +g_{c}\cdot\mathbf{h}_{\text{content}} \tag{7}\]  
This mechanism enables dynamic aggregation of hierarchical semantics based on learnable weights.  
Feedforward sublayer: The fused representation is further refined through a two-layer feedforward network with ReLU activation:  
\[\mathbf{h}_{\text{fused}}\ =\text{FFN}\Big{(}\mathbf{h}^{{}^{\prime}}_{\text{fused }}\Big{)}=\mathbf{W}_{2}\cdot\text{ReLU}\Big{(}\mathbf{W}_{1}\cdot\mathbf{h}^{ {}^{\prime}}_{\text{fused}}+\mathbf{b}_{1}\Big{)}+\mathbf{b}_{2} \tag{8}\]  
Residual connections and layer normalization are applied throughout to ensure training stability and gradient flow.