## 3 CGR-NER Architecture
### Fusion Layer  
After obtaining the contextual representation sequence and glyph representation sequence, we further propose an interactive fusion method with a crossmodal attention and gate mechanism to obtain a semantic enhanced fusion representation sequence, as shown in Figure 3. We first use two crossmodal Transformer encoders (CMTE) to complete the feature complementarity between the two modal representations, and then use a gate mechanism to dynamically adjust the contributions of different modal representations in different contexts. At the heart of CMTE to realize the interaction between different modal features is that the source modal feature can adaptively learn the complementary information from the target modal feature through the crossmodal attention mechanism. In the following, we take the contextual representation adaptively learning potential complementary information from the glyph representation as an example, which is denoted as \(g\to c\) (see \(CMTE_{g\to c}\) in Figure 3). In this example, the contextual representation sequence \(E_{c}\) is used as the source modal feature sequence, and the glyph representation sequence \(E_{g}\) is the target modal feature sequence. Formally, \(CMTE_{g\to c}\) computes feed-forwardly for \(i=1,\dots,D\) layers of crossmodal attention blocks. In the crossmodal attention block of each layer, the crossmodal attention is calculated first:  
\[U_{g\to c}^{[0]}=E_{c}W_{c} \tag{3}\]  
\[Q_{c}^{[i-1]}=LN(U_{g\to c}^{[i-1]}) \tag{4}\]  
\[K_{g}=V_{g}=LN(E_{g}W_{g}) \tag{5}\]  
Figure 2: Architecture of CBHNN. “g_s” represents the tensor size of output from each network. “Conv3D”, “MaxPool3D”, “k” and “s”’’ stand for 3 DCNN, 3D max-pooling, kernel size and stride. \[S_{g\to c}^{[i-1]}=MHCMA(Q_{c}^{[i-1]},K_{g},V_{g}) \tag{6}\]  
where \(W_{c}\in\mathbb{R}^{d_{c}\times d}\), \(W_{g}\in\mathbb{R}^{d_{x}\times d}\) are transformation matrices, and \(LN\) means layer normalization. \(MHCMA\) represents multi-head crossmodal attention, which is the key to achieving crossmodal interaction. To enhance the direction and distance perception ability of the model in the interaction process, we adopt the relative positional encoding scheme proposed by TENER [36], so \(MHCMA\) can be calculated as follows:  
\[MHCMA(Q_{c}^{[i-1]},K_{g},V_{g})=[head^{(1)},\ldots;head^{(n)}]W_{o} \tag{7}\]  
\[Q^{(h)},K^{(h)},V^{(h)}=Q_{c}^{[i-1]}W_{q}^{(h)},K_{g}^{d_{k}},V_{g}W_{v}^{(h)} \tag{8}\]  
\[R_{t-j}=[\ldots sin(\frac{t-j}{10000^{2m/d_{k}}})cos(\frac{t-j}{10000^{2m/d_{k }}})\ldots]^{T} \tag{9}\]  
\[A_{t,j}^{rel}=Q_{t}^{(h)}K_{j}^{T}+Q_{t}^{(h)}R_{t-j}^{T}+uK_{j}^{T}+vR_{t-j}^ {T} \tag{10}\]  
\[U_{g\to c}^{[0]}=E_{c}W_{c} \tag{3}\]0  
where \(n\) is the number of heads, \(h\) is the index of heads, \(d_{k}\times n=d\). \(W_{0}\in\mathbb{R}^{d\times d}\), \(W_{q}^{(h)}\in\mathbb{R}^{d\times d_{k}}\), \(W_{v}^{(h)}\in\mathbb{R}^{d\times d_{k}}\), \(u\in\mathbb{R}^{d_{k}}\), \(v\in\mathbb{R}^{d_{k}}\) are learnable parameters. \(K_{g}^{d_{k}}\) is the part of \(K_{g}\) allocated to each head. \(R_{t-j}\in\mathbb{R}^{d_{k}}\) is the relative positional encoding, \(m\in[0,\frac{d_{k}}{2}]\) in Equation (9). \(t\), \(j\) is the index of the target token and context token, respectively. \(Q_{t}^{(h)}\) is the query vector of token \(t\), \(K_{j}\) is the key vector of token \(j\). \(Q_{t}^{(h)}K_{j}^{T}\) is the attention score between token \(t\), \(j\); \(Q_{t}^{(h)}R_{t-j}^{T}\) is the bias of the token \(t\) in the relative position; \(uK_{j}^{T}\) is the bias of the token \(j\); \(vR_{t-j}^{T}\) is the bias term for distance and direction between tokens \(t\), \(j\). Then, a reset function \(r_{a}\) is used to dynamically control the residual connection of \(Q_{c}^{[i-1]}\) and \(S_{g\to c}^{[i-1]}\):  
\[U_{g\to c}^{[0]}=E_{c}W_{c} \tag{3}\]1  
\[U_{g\to c}^{[0]}=E_{c}W_{c} \tag{3}\]2  
where \(\sigma\) is the sigmoid function, \(W_{l_{1}}\), \(W_{l_{2}}\) are trainable parameters and \(b_{r}\) is the bias term.