# Vpn: Variation on Prompt Tuning for Named-Entity Recognition
## 4 Modelling VPN  
We let the LM predict several label words in the vocabulary and obtain the overall tag-related logits. These label words are more relevant to tags rather than classes. In this way, we can also model the logits of positions labelled 0 and use the BIO scheme rather than the IO scheme, which can use the CRF layer to boost the model's performance.  
In this work, we consider an NER task as a sequence-to-sequence task. Figure 3 shows the overall architecture of our proposed model. Given an input sequence \(\mathbf{x}=\{x_{1},x_{2},...,x_{T}\}\) and the corresponding label sequence \(\mathbf{y}=\{y_{1},y_{2},...,y_{T}\}\), we embed each word using a pre-trained LM to obtain an embedded sequence \(E_{emb}\in\mathbb{R}^{T\times d_{H}}\):  
\[\begin{split} E_{emb}&=\texttt{Encoder}(\mathbf{x})\\ &=[\mathbf{e}(x_{1}),\mathbf{e}(x_{2}),...,\mathbf{e}(x_{T})],\end{split} \tag{1}\]  
where \(\mathbf{e}(x_{t})\in\mathbb{R}^{d_{H}}\) is the last layer of the hidden state of word \(x_{t}\), and \(T\) and \(d_{H}\) denote the sequence length and hidden dimension of the transformer model, respectively.  
\begin{table}
\begin{tabular}{c c c c c c} \hline
**Dataset** & **\#ENT** & **Type** & **Train** & **Dev** & **Test** \\ \hline \multirow{2}{*}{Weibo NER} & 4 & S & 1.4k & 0.3k & 0.3k \\  & & T & 73.5k & 14.4k & 14.8k \\ \multirow{2}{*}{MSRA} & \multirow{2}{*}{3} & S & 46.4k & - & 4.4k \\  & & T & 979.2k & - & 172.6k \\ \multirow{2}{*}{OntoNotes 4.0} & \multirow{2}{*}{4} & S & 15.7k & 4.3k & 4.3k \\  & & T & 491.9k & 200.5k & 208.1k \\ \hline \end{tabular}
\end{table}
Table 1: Details of the three datasets. #ENT: number of entities; S: number of sentences; T: number of tokens.  
In order to take full advantage of the pre-trained model, along with BERT's pre-training stage, we calculate the word prediction logits using the masked language model head as follows:  
\[E_{1}=\texttt{Dense}_{1}(E_{emb}),\]  
\[E_{2}=\texttt{LayerNorm}(E_{1}), \tag{2}\]  
\[logit^{vocab}=\texttt{Dense}_{2}(E_{2}),\]  
where \(\texttt{Dense}_{1}\in\mathbb{R}^{T\times d_{H}}\), \(\texttt{Dense}_{2}\in\mathbb{R}^{d_{H}\times|\mathcal{V}|}\), \(logit^{vocab}\in\mathbb{R}^{T\times|\mathcal{V}|}\), and \(|\mathcal{V}|\) represents the cardinal number of the vocabulary.  
For each word \(x_{t}\), we obtain the label logit through mean pooling corresponding to the top \(K\) representative words of the entity tags, that is,  
\[logit^{label}_{t,j}=\sum_{k=1}^{K}w_{j,k}logit^{vocab}_{t,d_{j,k}}, \tag{3}\]  
where \(logit^{label}\in\mathbb{R}^{T\times m}\).  
Then, we feed the \(logit^{label}\) to the conditional random field (CRF) [26] layer. Implementation-wise, CRF computes an energy given a candidate output \(\mathbf{y}\) and a context \(\mathbf{x}\) (i.e., input sequence), followed by a softmax operator to obtain the conditional likelihood, i.e.,  
\[\mathbb{P}(\mathbf{y}|\mathbf{x})=\frac{e^{s(\mathbf{x},\mathbf{y})}}{\sum_{\mathbf{\bar{y}}\in Y _{\text{all}}}e^{s(\mathbf{x},\mathbf{\bar{y}})}}, \tag{4}\]  
\[s(\mathbf{x},\mathbf{y})=\sum_{t=1}^{T-1}logit^{label}_{t,\mathbf{y}_{t}}(\mathbf{x})+A_{\mathbf{y }_{t},\mathbf{y}_{t+1}}. \tag{5}\]  
Here, \(Y_{\text{all}}\) is the set of all possible tag sequences, and the transition matrix \(A\in\mathbb{R}^{m\times m}\) characterizes the smoothness of the label sequence (probability of switching between consequent labels).  
Figure 3: Architecture of our proposed model. The left side illustrates the overall process of inferring a specific sentence, while the right side illustrates the calculation of the \(logit^{label}\). Chinese label words are the same as those in Figure 2.