## 3 Integrated Model for MA and NER
In Figure 2, \(E_{f}(C_{i})\) is a feature embedding of \(C_{i}\) based on looking up a predefined NE dictionary. Subsequently, the character, alphabet, and feature embeddings are concatenated into the input embedding \(E(C_{i})\), as shown in Figure 1.  
In the MA layer, the input embeddings \(E(C)=\{E(C_{1}),\;E(C_{2}),\;\ldots,\;E(C_{n})\}\) of the \(n\) characters in a sentence are fed into a Bi-LSTM to yield a sequence of forward-hidden and backward-hidden states, respectively. Subsequently, these two states are concatenated to reflect bidirectional contextual information, as shown in the following equation:  
\[ \begin{array}{l}\overrightarrow{h}_{i}=LSTM\Big{\{}E(C_{i}),\; \overrightarrow{h}_{i-1}\Big{\}},\\ \overleftarrow{h}_{i}=LSTM\Big{\{}E(C_{i}),\;\overleftarrow{h}_{i-1}\Big{\}},\\ \overleftarrow{h}_{i}=\Big{[}\overrightarrow{h_{i}};\overleftarrow{h}_{i} \Big{]},\\ \overleftarrow{H}=\Big{\{}\overleftarrow{h}_{1},\;h_{2},\;\ldots,\; \overrightarrow{h}_{n}\Big{\}},\end{array}  \tag{3}\]  
where \(\Big{[}\overrightarrow{h}_{i};\overleftarrow{h}_{i}\Big{]}\) is the concatenation of forward hidden state \(\overrightarrow{h}_{i}\) and backward hidden state \(\overleftarrow{h}_{i}\) of the \(i\)-th character in a sentence. Next, the degrees of association between the contextualized input embeddings \(\overleftarrow{H}=\Big{(}\overleftarrow{h}_{1},\;\overleftarrow{h}_{2},\; \ldots\;\;\overleftarrow{h}_{n}\Big{)}\) and the morpheme tag embeddings \(E(M)=\{E(M_{1}),E(M_{2}),\ldots,E(M_{m})\}\) are calculated based on a multihead attention mechanism [14], as shown in the following equation:  
\[ \begin{array}{l}head_{j}=attention\Big{\{}QW_{j}^{Q},\;KW_{j}^{K},\;VW_{j}^{ V}\Big{\}}=\alpha_{j}*VW_{j}^{V},\;\text{where}\;\text{Q}=\overleftarrow{H},\;K=V=E(M), \\ \alpha_{j}=softmax\Bigg{\{}\frac{QW_{j}^{Q}\big{(}KW_{j}^{K}\big{)}^{T}}{\sqrt{ d_{h}}}\Bigg{\}},\\ A(C_{i})=head_{1}\bigoplus head_{2}\bigoplus\cdots\bigoplus head_{k},\end{array}  \tag{4}\]  
Figure 2: Input unit of MANE. where \(W_{j}^{Q}\), \(W_{j}^{K}\), and \(W_{j}^{V}\) are the weighting parameters of the \(j\)-th parameter among \(k\) heads to be learned during training. The morpheme tag embeddings \(E(M)\) represent the embedding vectors of the \(m\) morpheme tags that are randomly initialized and fine-tuned during training. The attention score \(\alpha_{j}\) is calculated using a scaled-dot product, where \(d_{h}\) is a normalization factor and denotes that the hidden size of Bi-LSTM is the same as the dimension of the morpheme tag embeddings. The attention score vector \(A(C_{i})\) represents the degrees of association between the contextualized input embedding \(\overset{\leftrightarrow}{h}_{i}\) of the \(i\)-th input character and each morpheme tag. In other words, it can be considered as a potential distribution of morpheme tags associated with an input character. In the prediction phase, the MA layer outputs the morpheme tags, as shown in the following equation:  
\[M_{i}=argmax\Big{(}\hat{A}_{i}^{1},\ \hat{A}_{i}^{2},\ \ldots,\hat{A}_{i}^{m} \Big{)}, \tag{5}\]  
where \(\hat{A}_{i}^{j}\) denotes the \(j\)-th one among \(m\) attention scores in the trained attention vector \(\hat{A}_{i}\). In the NER layer, the \(i\)-th input embedding \(E(C_{i})\) is concatenated to the embedding of the morpheme tag with a maximum attention score, \(E(M_{i})\). Subsequently, the concatenated vectors are fed into a Bi-LSTM in the same manner that is used for the MA layer, as shown in the following equation:  
\[ \begin{array}{l}\overset{\rightarrow}{h}_{i}=LSTM\Big{\{}[E(C_{i});E(M_{i})], \ \overset{\rightarrow}{h}_{i-1}\Big{\}},\\ \overset{\leftarrow}{h}_{i}=LSTM\Big{\{}[E(C_{i});E(M_{i})],\ \overset{ \leftarrow}{h}_{i-1}\Big{\}},\\ \overset{\leftarrow}{h}_{i}=\Big{[}\overset{\rightarrow}{h}_{i}\overset{ \leftarrow}{h}_{i}\Big{]},\\ \overset{\leftrightarrow}{H}=\Big{\{}\overset{\leftrightarrow}{h}_{1},\ \overset{ \leftrightarrow}{h}_{2},\ \ldots\ \overset{\leftrightarrow}{h}_{n}\Big{\}}\end{array}