## 3 Methodology
### Neural Network Architecture for NER Task Fine-Tuning
#### 3.3.1 Embedding Layer  
The embedding layer transforms words, characters and POS representation from an input sentence into dense feature vectors where a vector represents the projection of the word into a continuous vector space. The position of a word in the learned vector space is referred to as its embedding [29]. In our work, we utilize POS tag, word- and character-level embedding features. For a given sentence of word sequence \(S=\{w_{1},\ w_{2},\ldots,\ w_{k}\}\), the dense embedding feature vector is created by concatenating character-level feature \(e_{i}^{e}\in\ \mathbb{R}^{d_{e}}\), word-level feature \(e_{i}^{w}\in\ \mathbb{R}^{d_{w}}\) and POS tag feature \(e_{i}^{p}\in\ \mathbb{R}^{d_{p}}\). Note that the word embedding feature \(e_{i}^{w}\) is obtained from the pre-trained BERT-domain model. The final embedding vector can be formulated as \(X_{i}=e_{i}^{c}\oplus\ e_{i}^{w}\oplus\ e_{i}^{p}\), where \(i\in\{1,\ 2,\ \ldots,\ k\}\); \(e_{i}^{c}\), \(e_{i}^{w}\) and \(e_{i}^{p}\) are the character, word and POS feature embeddings, respectively, and \(d_{c}\), \(d_{w}\) and \(d_{p}\) are the dimensions of character, word and POS feature vector space, respectively; \(\oplus\) denotes the concatenation operation. Figure 2: BERT post-training on domain corpus. Word Embedding (BERT Embedder)  
Word embedding is a powerful technique for capturing a word's semantic and syntactic meanings. It has been a common technique and is widely utilized for a variety of natural language processing applications, such as named-entity recognition. Furthermore, the embeddings from domain-adapted model (BERT-domain) can capture higher semantic and syntactic meanings of a word for the target domain. To obtain word embedding, we adapt the BERT-domain model. The embedding size is 12 \(\times\) 768 for each word token with a padded sequence length of 512. We utilize the feature-based BERT-domain model to obtain only the embeddings of the input text by freezing all the layers. For feature selection from the BERT-domain model, the BERT authors examined that concatenating the last four layers produces the best results for feature-based approach, which is also observed through our experiments. We adapt this setting by applying mean pooling on the last four hidden states. We denote a word represented by a word embedding as \(e_{i}^{w}\ \in\ \mathbb{R}^{dw}\); for each word \(w_{k}\) in a sequence S, where \(d_{w}\) is the dimension of embedding vector space. Character-Level Embedding (CharCNN: Character-level Convolutional Neural Network)  
Character-level features have shown to be effective for extracting morphological information [8] and have been adopted to improve the learning of misspelling and out-of-vocabulary word representations from the user-generated text (i.e., user reviews). Therefore, we employ a convolutional neural network (ConvNets) [30] to adopt character-level representation of words. Let C be the character vocabulary of a word token with a vector of dimension d, we compose the character embedding matrix \(X_{emb}\ \in\ \mathbb{R}^{d\times C}\). Then, we perform a convolution operation between X\({}_{\text{emb}}\) and a filter (kernel) \(H\ \in\ \mathbb{R}^{d\times w}\) to extract features map \(X_{f}\) from the word. Each filter has a shape \((w_{i},\ d)\) where \(w_{i}\) is the filter size of the \(\ i\)th filter:  
\[X_{f}=Conv1D(X_{emb})\ \in\ \mathbb{R}^{C-w_{i}+1} \tag{1}\]  
\[X_{p}=MaxPool\Big{(}ReLU\Big{(}X_{f}\Big{)}\Big{)} \tag{2}\]  
\[e_{i}^{c}=concat(Xp) \tag{3}\]  
Figure 3: Embedding vector from different level of representations. We use nonlinear activation function ReLU following the max-over-time-pooling to each feature map \(X_{f}\). Finally, we concatenate the pooled output to obtain the character-level representation of a word. Part-Of-Speech (POS) Tag Feature Embedding  
We use Part-Of-Speech (POS) tag features to extract syntactic information in our network model for the NER task.