## 3 Methodology
### Neural Network Architecture for NER Task Fine-Tuning
#### 3.3.2 NER Task Network Architecture  
We construct a Bi-LSTM-CRF network architecture shown in Figure 4, for fine-tuning the downstream MenuNER task, which consists of two layers: Bi-LSTM layer--captures the long-term dependencies and semantic information from input sequence; CRF layer--labels the tokens based on output probabilities. We also construct a variant of Bi-LSTM-CRF model by applying multi-head self-attention layer on the top of stacked Bi-LSTM. In the input embeddings, we extract the feature vector of each word \(w_{k}\) from the sequence by combining word- and character-level features and auxiliary POS tag features as described previously and fed into the stacked Bi-LSTM layer. Bidirectional Long short-term memory (Bi-LSTM) Layer  
LSTM-based networks are capable of learning long-range dependencies and prevent gradient vanishing problem. Bi-LSTMs enable the hidden states to make full use of the context information of the input sequence. A single LSTM memory unit consists of an input gate \((i_{t})\), output gate \((o_{t})\), forget gate \((f_{t})\) and a memory cell \((c_{t})\), and produces a hidden vector of LSTM cell unit using the following mathematical formulation:  
\[i_{t}=\sigma(W_{xi}x_{t}+W_{hi}h_{t-1}+b_{i}) \tag{4}\]  
\[f_{t}=\sigma\Big{(}W_{xf}x_{t}+W_{hf}h_{t-1}+b_{f}\Big{)} \tag{5}\]  
\[g_{t}=tanh(W_{xc}x_{t}+W_{hc}h_{t-1}+b_{c}) \tag{6}\]  
\[c_{t}=f_{t}\ \odot c_{t-1}+i_{t}\ \odot\ g_{t} \tag{7}\]  
\[o_{t}=\sigma(W_{xo}x_{t}+W_{ho}h_{t-1}+b_{o}) \tag{8}\]  
\[h_{t}=o_{t}\ \odot\ \tan\text{h}(c_{t}) \tag{9}\]  
where \(\sigma\) and \(\tan\text{h}\) denotes the logistic function and a hyperbolic tangent function, respectively, and \(\odot\) is a Hadamard product. \(W_{i},\ W_{f},\ W_{o}\ \in\ \mathbb{R}^{\text{d}\times 2\text{d}}\) are the weighted matrices. Our network uses the stacked Bi-LSTM by concatenating the forward \(\overset{\rightarrow}{h_{t}}\) and backward \(\overset{\leftarrow}{h_{t}}\) states. The forward LSTM layer extracts the past information while the backward layer captures the future information of the sequence and produces the final output \(y_{t}=\left[\overset{\rightarrow}{h_{t}},\ \overset{\leftarrow}{h_{t}}\right]\). In the stacked Bi-LSTM network, the output \(y_{t}\) from the lower layer feeds into the upper-level Bi-LSTM layer. We process Bi-LSTM with 512 hidden units so that the output representation dim is 1024 for each token in each stacked layer of the network with dropout 0.3. This recurrent layer learns the dependencies between tokens in an input sequence. Conditional Random Field (CRF) Layer  
We adopt the Conditional Random Field (CRF) model for our prediction task. CRF is based on conditional probability model that is used for labeling sequential data. For an input sequence \(X=\{x_{1},\ x_{2},\dots,\ x_{n}\}\), where \(x_{n}\) is the input vector of the \(n\)th word and \(y=\{y_{1},\ y_{2},\dots,\ y_{n}\}\) is the corresponding label sequences for \(X\). We have \(n\times l\) matrix \(P\), obtained from the previous stacked Bi-LSTM layer, where \(n\) is the number of input word and \(l\) is the number of distinct tags, \(P_{i,j}\) is the probability of label \(i\) of the word \(j\) in the sentence. The score is computed as:  
\[s(X,\ y)=\sum_{i=0}^{n}A_{y_{i},y_{i+1}}+\sum_{i=0}^{n}P_{i,y_{i}} \tag{10}\]  
where \(A_{i,j}\) is the likelihood of transition from the tag \(y_{i}\) to tag \(y_{j}\). The \(y_{0}\) and \(y_{n}\) are the first and last tags of a sentence, respectively, that we add to a list of possible tags. We then obtain the normalized probability of the label sequence \(y\) and output the label sequence \(\overline{y}\) by using softmax function:  
\[p(y|X)=\frac{e^{s(X,\ y)}}{\Sigma_{\overline{y}\in Y_{X}}e^{s(X,\overline{y})}} \tag{11}\]  
Figure 4: The proposed Bi-LSTM-CRF neural network with a variation of multi-head self-attention for MenuNER task, fine-tuned on domain specific BERT. where for a given sentence \(X\),..., \(Y_{X}\) represents all conceivable tag sequences.