Toward a Standardized and More Accurate Indonesian Part-of-Speech Tagging

Abstract: Previous work in Indonesian part-of-speech (POS) tagging are hard to compare as they are not evaluated on a common dataset. Furthermore, in spite of the success of neural network models for English POS tagging, they are rarely explored for Indonesian. In this paper, we explored various techniques for Indonesian POS tagging, including rule-based, CRF, and neural network-based models. We evaluated our models on the IDN Tagged Corpus. A new state-of-the-art of 97.47 F1 score is achieved with a recurrent neural network. To provide a standard for future work, we release the dataset split that we used publicly.

Part-of-speech (POS) tagging is a process to tag tokens in a string with their corresponding part-of-speech (e.g., noun, verb, etc). POS tagging is considered as one of the most basic tasks in NLP, as it is usually the first component in an NLP pipeline. This is because POS tags are shown to be useful features in various NLP tasks, such as named entity recognition BIBREF0 , BIBREF1 , machine translation BIBREF2 , BIBREF3 and constituency parsing BIBREF4 . Therefore, for any language, building a successful NLP system usually requires a well-performing POS tagger.
There are quite a number of research on Indonesian POS tagging BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 . However, almost all of them are not evaluated on a common dataset. Even when they are, their train-test split are not the same. This lack of a common benchmark dataset makes a fair comparison among these works difficult. Moreover, despite the success of neural network models for English POS tagging BIBREF9 , BIBREF10 , the use of neural networks is generally unexplored for Indonesian. As a result, published results may not reflect the actual state-of-the-art performance of Indonesian POS tagger.
In this work, we explored different neural network architectures for Indonesian POS tagging. We evaluated our experiments on the IDN Tagged Corpus BIBREF11 . Our best model achieves 97.47 INLINEFORM0 score, a new state-of-the-art result for Indonesian POS tagging on the dataset. We release the dataset split that we used to serve as a benchmark for future work.
Pisceldo et al. BIBREF5 built an Indonesian POS tagger by employing a conditional random field (CRF) BIBREF12 and a maximum entropy model. They used contextual unigram and bigram features and achieved accuracy scores of 80-90% on PANL10N dataset tagged manually using their proposed tagset. The dataset consists of 15K sentences. Another work used a hidden Markov model enhanced with an affix tree to better handle out-of-vocabulary (OOV) words BIBREF6 . They evaluated their models on the same PANL10N dataset and achieved more than 90% overall accuracy and roughly 70% accuracy for the OOV cases. We note that while the datasets are the same, the split could be different. Thus, making a fair comparison between them is difficult.
Dinakaramani et al. BIBREF11 proposed IDN Tagged Corpus, a new manually annotated POS tagging corpus for Indonesian. The corpus consists of 10K sentences and 250K tokens, and its tagset is different than that of the PANL10N dataset. The corpus is available online. A rule-based tagger is developed in BIBREF7 using the aformentioned dataset, and is able to achieve an accuracy of 80%.
One of the neural network-based POS taggers for Indonesian is proposed in BIBREF8 . They used a feedforward neural network with an architecture similar to that proposed in BIBREF13 . They evaluated their methods on the new POS tagging corpus BIBREF11 and separated the evaluation of multi- and single-word expressions. They experimented with several word embedding algorithms trained on Indonesian Wikipedia data and reported macro-averaged INLINEFORM0 score of 91 and 73 for the single- and multi-word expression cases respectively. We remark that the choice of macro-averaged INLINEFORM1 score is more suitable than accuracy for POS tagging because of the class imbalance in the dataset. There are too many words with NN as the true POS tag, so accuracy is not the best metric in such case.
We used the IDN Tagged Corpus proposed in BIBREF11 . The corpus contains 10K sentences and 250K tokens that are tagged manually. Due to the small size, we used 5-fold cross-validation to split the corpus into training, development, and test sets. We did not split multi-word expressions but treated them as if they are a single token. All 5 folds of the dataset are available publicly to serve as a benchmark for future work.
We used two simple baselines: majority tag (Major) and memorization (Memo). Major simply predicts the majority POS tag found in the training set for all words. Memo remembers the word-tag assignments from the training set and uses them to predict the tags on the test set. If there is an unknown word, it simply outputs the majority tag found in the training set.
We adopted a rule-based tagger designed by Rashel et al. BIBREF14 as one of our comparisons. Firstly, the tagger tags named entities and multi-word expressions based on a dictionary. Then, it uses MorphInd BIBREF15 to tag the rest of the words. Finally, they employ 15 hand-crafted rules to resolve ambiguous tags in the post-processing step. We want to note that we did not use their provided tokenizer since the IDN Tagged Corpus dataset is already tokenized. Their implementation is available online.
We used CRF BIBREF12 as another comparison since it is the most common non-neural model for sequence labeling tasks. We employed contextual words as well as affixes as features. For some context window size INLINEFORM0 , the complete list of features is:
the current word, as well as INLINEFORM0 preceding and succeeding words;
two and three leading characters of the current word and INLINEFORM0 preceding and succeeding words;
two and three trailing characters of the current word and INLINEFORM0 preceding and succeeding words.
The last two features are meant to capture prefixes and suffixes in Indonesian which usually consist of two or three characters. One advantage of this feature extraction approach is that it does not require language-specific tools such as stemmer or morphological segmenter. This advantage is particularly useful for Indonesian which does not have well-established tools for such purposes. We padded the input sentence with padding tokens to ensure that every token has enough preceding and succeeding words for context window size INLINEFORM0 . For the implementation, we used pycrfsuite.
Our neural network-based POS tagger can be divided into 3 steps: embedding, encoding, and prediction. First, the tagger embeds the words and optionally additional features of such words (e.g., affixes). From this embedding process, we get vector representations of the words and the features. Next, the tagger learns contextual information in the encoding step via either a feedforward network with context window or a bidirectional LSTM BIBREF16 . Finally, in prediction step, the tagger predicts the POS tags from the output of the encoding step using either a softmax or a CRF layer.
Embedding. In the embedding step, the tagger obtains vector representations of each word and additional features. We experimented with several additional features: prefixes, suffixes, and characters. Prefix features are the first 2 and 3 characters of the word. Likewise, suffix features are the last 2 and 3 characters of the word. For the character features, we followed BIBREF9 by embedding each character and composing the resulting vectors with a max-pooled CNN. The final embedding of a word is then the concatenation of all these vectors. Fig. FIGREF17 shows an illustration of the process.
Encoding. In the encoding step, the tagger learns contextual information by using either a feedforward network with context window or a bidirectional LSTM (biLSTM). The feedforward network accepts as input the concatenation of the embedding of the current word and INLINEFORM0 preceding and succeeding words for some context window size INLINEFORM1 . Formally, given a sequence of word embedding INLINEFORM2 , the input of the feedforward network at timestep INLINEFORM3 is DISPLAYFORM0 
where INLINEFORM0 denotes a concatenation. The feedforward network then computes DISPLAYFORM0 
 where INLINEFORM0 is the output vector, INLINEFORM1 is a dropout mask vector, and INLINEFORM2 are parameters. The output vector INLINEFORM3 has length equal to the number of possible tags. Its INLINEFORM4 -th component defines the (unnormalized) log probability of the INLINEFORM5 -th word having tag INLINEFORM6 .
On the other hand, the biLSTM accepts as input the sequence of word embeddings, and for each timestep, the output from the forward and backward LSTM are concatenated to form the final output. Formally, the output at each timestep INLINEFORM0 can be expressed as DISPLAYFORM0 
where DISPLAYFORM0 
 The vector INLINEFORM0 is then passed through INLINEFORM1 as before to obtain INLINEFORM2 .
Prediction. In the prediction step, the tagger predicts the POS tag of the INLINEFORM0 -th word based on the output vector INLINEFORM1 . We tested two approaches: a softmax layer with greedy decoding and a CRF layer with Viterbi decoding. With a softmax layer, the tagger simply normalizes INLINEFORM2 and predicts using greedy decoding, i.e. picking the tag with the highest probability. In contrast, with a CRF layer, the tagger treats INLINEFORM3 as emission probability scores, models the tag-to-tag transition probability scores, and uses Viterbi algorithm to select the most probable tag sequence as the prediction. We refer readers to BIBREF17 to read more about how the CRF layer and Viterbi decoding work. We want to note that when we only embed words, encode using feedforward network, and predict using greedy decoding, the tagger is effectively the same as that in BIBREF8 . Also, when only the word and character features are used, with a biLSTM and CRF layer, the tagger is effectively the same as that in BIBREF9 . Our implementation code is available online.
For all models, we preprocessed the dataset by lowercasing all words, except when the characters were embedded. For the CRF model, we used L2 regularization whose coefficient was tuned to the development set. As we mentioned previously, we tuned the context window size INLINEFORM0 to the development set as well.
For the neural tagger, we set the size of the word, affix, and character embedding to 100, 20, and 30 respectively. We applied dropout regularization to the embedding layers. The max-pooled CNN has 30 filters for each filter width. We set the feedforward network and the biLSTM to have 100 hidden units. We put a dropout layer before the biLSTM input layer. We tuned the learning rate, dropout rate, context window size, and CNN filter width to the development set. As we said earlier, we experimented with different configurations in the embedding, encoding, and prediction step. We evaluated each configuration on the development set as well.
At training time, we used a batch size of 8, decayed the learning rate by half if the INLINEFORM0 score on the development set did not improve after 2 epochs, and stopped the training early if the score still did not improve after decaying the learning rate 5 times. To address the exploding gradient problem, we normalized the gradient norm at 1, following the suggestion in BIBREF18 . To handle the out-of-vocabulary problem, we converted singleton words and affixes occurring fewer than 5 times in the training data into a special token for unknown words/affixes.
Since the dataset is highly imbalanced (majority of words are nouns), using accuracy score as the evaluation metric is not appropriate as it gives a high score to a model that always predicts nouns regardless of input. Therefore, we decided to use INLINEFORM0 score which considers both precision and recall of the predictions.
Since there are multiple tags, there are two flavors to compute an overall INLINEFORM0 score: micro and macro average. For POS tagging task where the tags do not span multiple words, micro-average INLINEFORM1 score is exactly the same as accuracy score. Thus, macro-average INLINEFORM2 score is our only option. However, there is still an issue. Macro-average INLINEFORM3 score computes the overall INLINEFORM4 score by averaging the INLINEFORM5 score of each tag. This approach means that when the model wrongly predicts a rarely occurring tag (e.g., foreign word), it is penalized as heavily as it does a frequent tag. To address this problem, we used weighted macro-average INLINEFORM6 score which takes into account the tag proportion imbalance. It computes the weighted average of the scores where each weight is equal to the corresponding tag's proportion in the dataset. This functionality is available in the scikit-learn library.
Firstly, we report on our tuning experiments for the neural tagger. Table TABREF27 shows the evaluation results of the many configurations of our neural tagger on the development set. We group the results by the encoding and prediction step configuration. For each group, we show the highest INLINEFORM0 score among many embedding configurations. As we can see, biLSTM with CRF layer achieves 97.60 INLINEFORM1 score, the best score on the development set. This result agrees with many previous work in neural sequence labeling that a bidirectional LSTM with CRF layer performs best BIBREF10 , BIBREF17 , BIBREF9 . Therefore, we will use this tagger to represent the neural model hereinafter.
To understand the performance of the neural model for each tag, we plot the confusion matrix from the development set of the first fold in Fig. FIGREF30 . The figure shows that the model can predict most tags almost perfectly, except for X and WH tag. The X tag is described as "a word or part of a sentence which its category is unknown or uncertain". The X tag is rather rare, as it only appears 397 times out of over 250K tokens. Some words annotated as X are typos and slang words. Some foreign terms and abbreviations are also annotated with X. The model might get confused as such words are usually tagged with a noun tag (NN or NNP). We also see that the model seems to confuse question words (WH) such as apa (what) or siapa (who) as SC since these words may be used in subordinate clauses as well. Looking at the data closely, we found that the tagging of such words are inconsistent. This inconsistency contributes to the inability of the model to distinguish the two tags well.
Next, we present the result of evaluating the baselines and other comparisons on the test set in Table TABREF28 . The INLINEFORM0 scores are averaged over the 5 cross-validation folds. We see that Major baseline performs very poorly compared to the Memo baseline, which surprisingly achieves over 90 INLINEFORM1 points. This result suggests that Memo is a more suitable baseline for this dataset in contrast with Major. The result also provides evidence to the usefulness of our evaluation metric which heavily penalizes a simple majority vote model. Furthermore, we notice that the rule-based tagger by Rashel et al. BIBREF7 performs worse than Memo, indicating that Memo is not just suitable but also quite a strong baseline. Moving on, we observe how CRF has 6 points advantage over Memo, signaling that incorporating contextual features and modeling tag-to-tag transitions are useful. Lastly, the biLSTM with CRF tagger performs the best with 97.47 INLINEFORM2 score.
To understand how each feature in the embedding step affects the neural tagger, we performed feature ablation on the development set and put the result in Table TABREF29 . We see that with only words as features (first row), the neural tagger only achieves 96.06 INLINEFORM0 score. Employing character features boosts the score up to 97.42, a gain of 1.36 points. Adding prefix and suffix features improves the performance further by 0.08 and 0.10 points respectively. From this result, we see that it is the character features that positively affect the neural tagger the most.
We experimented with several baselines and comparisons for Indonesian POS tagging task. Our comparisons include a rule-based tagger, a well-established probabilistic model for sequence labeling (CRF), and a neural model. We tested many configurations for our neural model: the features (words, affixes, characters), the architecture (feedforward, biLSTM), and the output layer (softmax, CRF). We evaluated all our models on the IDN Tagged Corpus BIBREF11 , a manually annotated and publicly available Indonesian POS tagging dataset. Our best model achieves 97.47 INLINEFORM0 score, a new state-of-the-art result on the dataset. We make our cross-validation split available publicly to serve as a benchmark for future work.

Figure and table captions:
Figure 1. Illustration of the embedding step. The word and its affixes are embedded to obtain their vector representations. Character embeddings of the word are composed with a max-pooled CNN. The final word embedding is the concatenation of all the result vectors.
Table I DEV F1 SCORE OF EACH NEURAL TAGGER ARCHITECTURE
Table II TEST F1 SCORE OF EACH METHOD, AVERAGED OVER 5 FOLDS
Figure 2. Confusion matrix of the best biLSTM with CRF tagger from the development set of the first fold. The tagger seems to have difficulties dealing with words annotated as X and confuse WH as SC.

Questions about this paper:
what neural network models were explored?

Answers:
Feedforward, biLSTM