Introduction
Following previous research on automatic detection and correction of dt-mistakes in Dutch BIBREF0, this paper investigates another stumbling block for both native and non-native speakers of Dutch: the correct use of die and dat. The multiplicity of syntactic functions and the dependency on the antecedent's gender and number make this a challenging task for both human and computer. The grammar concerning die and dat is threefold. Firstly, they can be used as dependent or independent demonstrative pronouns (aanwijzend voornaamwoord), with the first replacing the article before the noun it modifies and the latter being a noun phrase that refers to a preceding/following noun phrase or sentence. The choice between the two pronouns depends on the gender and number of the antecedent: dat refers to neuter, singular nouns and sentences, while die refers to masculine, singular nouns and plural nouns independent of their gender. Secondly, die and dat can be used as relative pronouns introducing relative clauses (betrekkelijk voornaamwoord), which provide additional information about the directly preceding antecedent it modifies. Similar rules as for demonstrative pronouns apply: masculine, singular nouns and plural nouns are followed by relative pronoun die, neuter singular nouns by dat. Lastly, dat can be used as a subordinating conjunction (onderschikkend voegwoord) introducing a subordinating clause. An brief overview of the grammar is given in Table TABREF1.
The aim is to develop (1) a binary classification model that automatically detects, predicts and corrects die and dat instances in texts. Furthermore, the correct die/dat instance and the syntactic function of the predicted die and dat are jointly predicted in (2) a multitask classification model. Whereas research on neural-based, machine learning approaches for Dutch demonstrative and relative pronoun resolution - especially for die and dat - is to our knowledge non-existing, this project can serve as a starting point for further research on machine learning applications concerning Dutch subordinating conjunctions, demonstrative pronouns and relative pronouns.
Related Work
The incentive for this research project is the detection and correction system for dt-mistakes in Dutch BIBREF0. For that task, a system with a context encoder - a bidirectional LSTM with attention mechanism - and verb encoder - of which the outputs are then fed to a feedforward neural network - has been developed to predict different verb suffixes. As mentioned above, this project explores the possibility of constructing a neural network system for correcting Dutch demonstrative and relative pronouns die and dat. The task is also called pronoun resolution or anaphora resolution. Anaphora resolution and pronoun prediction has been major research subjects in machine translation research. In BIBREF3, for example, the effect of multiple English coreference resolvers on the pronoun translation in English-Dutch machine translation system with deep transfer has been investigated. Niton, Morawiecki and Ogrodnizuk (2018) developed a fully connected network with three layers in combination with a sieve-based architecture for Polish coreference resolution BIBREF4. Not only in machine translation, but also in general much research has been conducted on machine learning approaches towards coreference resolution BIBREF5BIBREF6BIBREF7 and pronoun resolution BIBREF8, BIBREF9. However, little to no research has been conducted specifically on die/dat correction.
Dataset
The datasets used for training, validation and testing contain sentences extracted from the Europarl corpus BIBREF1 and SoNaR corpus BIBREF2. The Europarl corpus is an open-source parallel corpus containing proceedings of the European Parliament. The Dutch section consists of 2,333,816 sentences and 53,487,257 words. The SoNaR corpus comprises two corpora: SONAR500 and SONAR1. The SONAR500 corpus consists of more than 500 million words obtained from different domains. Examples of text types are newsletters, newspaper articles, legal texts, subtitles and blog posts. All texts except for texts from social media have been automatically tokenized, POS tagged and lemmatized. It contains significantly more data and more varied data than the Europarl corpus. Due to the high amount of data in the corpus, only three subparts are used: Wikipedia texts, reports and newspaper articles. These subparts are chosen because the number of wrongly used die and dat is expected to be low.
Preprocessing
The sentences in the Europarl corpus are tokenized and parsed using the Dutch version of TreeTagger BIBREF10. Only sentences which contain at least one die or dat are extracted from the corpora. Subsequently, each single occurrence of die and dat is detected and replaced by a unique token ('PREDICT'). When there are multiple occurrences in one sentence, only one occurrence is replaced at a time. Consequently, a sentence can appear multiple times in the training and test dataset with the unique token for die and dat at a different place in the sentence. Each sentence is paired with its automatically assigned ground truth label for die and dat. The Europarl dataset, on the one hand, contains 70,057 dat-labeled and 33,814 die-labeled sentences. The resulting train and test sets consist of 103,871 (Europarl) and 1,269,091 (SoNaR) sentences. The SoNaR dataset, on the other hand, has more than ten times the number of labeled sentences with 736,987 dat-labeled and 532,104 die-labeled. Considering the imbalance in both datasets, it may be argued that dat occurs more frequently than die due to its syntactic function as subordinating conjunction and not to its use as demonstrative pronoun whereas it can only refer to singular, neutral nouns. As for the multitask classification model, the POS tags for die and dat present in the SoNaR corpus are extracted and stored as ground truth labels: 407,848 subordinating conjunction, 387,292 relative pronoun and 473,951 demonstrative pronoun. From a brief qualitative assessment on the POS tags for die and dat in both corpora, the POS tags in the SoNaR corpus appear to be more reliable than the POS tags generated by TreeTagger in the Europarl corpus. Therefore, only the SoNaR dataset is used for the multitask classification. An overview of the datasets after preprocessing is given in Table TABREF2.
Binary Classification Model ::: Model Architecture
For the binary classification model that predicts the correct die or dat for each sentence, a Bidirectional Long-Short Term Memory (BiLSTM) neural network is computed. Whereas the antecedent can be rather distant from the relative or demonstrative pronoun due to adjectives and sentence boundaries, an LSTM architecture is chosen over a regular Recurrent Neural Network while the latter does not cope well with learning non-trivial long-distance dependencies BIBREF11. Furthermore, a bidirectional LSTM is chosen over a single left-to-right LSTM, whereas the antecedent can be either before or after the die or dat. The architecture of the binary classification model is provided in Fig. FIGREF7. The input sentence is first sent through an embedding layer where each token is transformed to a 100-dimensional word embedding which have been initially trained on the dataset of sentences containing at least one die or dat using the Word2Vec Skip-gram model BIBREF12. The weights of the embedding layer are trainable. The word embeddings are then sent through a BiLSTM layer. The bidirectional LSTM concatenates the outputs of two LSTMs: the left-to-right $LSTM_{forward}$ computes the states $\overrightarrow{h_1}..\overrightarrow{h_N}$ and the right-to-left $LSTM_{backward}$ computes the states $\overleftarrow{h_N}..\overleftarrow{h_1}$. This means that at time $t$ for input $x$, represented by its word embedding $E(x)$, the bidirectional LSTM outputs the following:
The concatenated output is then sent through a maxpooling layer, linear layer and, eventually, a softmax layer to get a probability distribution over the two classes. In order to prevent the model from overfitting and co-adapting too much, dropout regularization is implemented in the embedding layer and the linear layer. In both layers, dropout is set to $p = 0.5$ which randomly zeroes out nodes in the layer using samples from a Bernoulli distribution.
Binary Classification Model ::: Experimental Set-Up
Each dataset is randomly divided into a training (70%), validation (15%) and test set (15%). The data is fed to the model in batches of 128 samples and reshuffled at every epoch. The objective function that is minimized is Binary Cross-Entropy:
where $y_i$ is the ground truth label (0 for dat and 1 for die) and $p(\hat{y}_i)$ is the probability of the predicted label for all $N$ input sentences of the train set. The weights are optimized by Stochastic Gradient Descent with learning rate = 0.01 and momentum = 0.9. The data is fed to the model in 24 epochs.
Binary Classification Model ::: Results
An overview of the performance results is given in Table TABREF11. We compare model performance when trained and tested on the two corpora individually and experiment with different settings of the two corpora in order to investigate the effect of dataset changes on model performance. There are three settings: full in which the datasets contain full sentences, windowed in which sentences are windowed around the unique prediction token without exceeding sentence boundaries (five tokens before and after the token, including token), and windowed no_boundaries in which the windows can exceed sentence boundaries. When limiting the input sentences to windowed sentences in the Europarl corpus(2), model performance increases significantly on all metrics, especially for die prediction performance. The difference in model performance when trained and tested on the Europarl (2) and SoNaR (3) windowed datasets is particularly noticeable in the precision, recall and F1 scores. Model performance for dat prediction is better for the Europarl dataset than for the SoNaR dataset, while model performance for die prediction is notably better for the SoNaR dataset than for the Europarl dataset. Lastly, a change in windowing seems to have a positive impact on the overall model performance: the model trained and tested on the SoNaR dataset with windows exceeding sentence boundaries (3) outperforms that on the SoNaR dataset with windows within sentence boundaries (4) on every metric.
Multitask Classification Model ::: Model Architecture
The second model performs two prediction tasks. The first prediction task remains the binary classification of die and dat. The second prediction task concerns the prediction of three parts-of-speech (POS) or word classes, namely subordinating conjunction, relative pronoun and demonstrative pronoun. An overview of the model architectures is given in Fig. FIGREF13. For the BiLSTM model, the first layer is the embedding layer where the weights are initialized by means of the 200-dimensional pre-trained embedding matrix. The weights are updated after every epoch. The second layer consists of two bidirectional LSTMs where the output of the first bidirectional LSTM serves as input to the second bidirectional LSTM. The layer has dropout regularization equal to 0.2. The two-layer bidirectional LSTM concatenates the outputs at time $t$ into a 64-dimensional vector and sends it through a maxpooling layer. Until this point, the two task share the same parameters. The model than splits into two separate linear layers. The left linear layer transforms the 64-dimensional vector to a two-dimensional vector on which the softmax is computed. The softmax outputs the probability distribution over the dat and die labels. The right linear layer transforms the 64-dimensional vector to a three-dimensional vector on which the softmax is computed as well. The softmax outputs the probability distribution over the subordinating conjunction, relative pronoun and demonstrative pronoun labels. The second multitask classification model takes the immediate context around the 'PREDICT' token as additional input. Both the windowed sentence and context are first transformed into their word embedding representations. They are, then, separately sent through a sentence encoder and context encoder, respectively. The sentence encoder has the same architecture as the second and third layer of the BiLSTM model, namely a two-layer bidirectional LSTM and a maxpooling layer. For the context encoder, we experiment with two different architectures: a feedforward neural network and a one-layer bidirectional LSTM with dropout = 0.2 with a maxpooling layer on top. Both sentence and context encoder output a 64-dimensional vector which are, consequently, concatenated to a 128-dimensional vector. As in the BiLSTM model, the resulting vector is sent through two separate linear layers to output probability distributions for both the die/dat and POS prediction task.
Multitask Classification Model ::: Experimental Set-up
As discussed in Section SECREF4, the POS ground truth labels in SoNaR-based datasets are more reliable than the POS labels in the Europarl-based datasets that are generated by TreeTagger. Consequently, only the SoNaR dataset is used for training and testing. The dataset is randomly divided into a training (70%), validation (15%) and test (15%) set. The data is fed into the model in batches of 516 samples and the data is reshuffled at every epoch. For die/dat prediction, the Binary Cross-Entropy loss function is minimized. The weights are optimized by Stochastic Gradient Descent with learning rate = 0.01 and momentum = 0.9. For POS prediction, Cross-Entropy is minimized:
where $C$ is the number of classes, in this case three, $y_{i,c}$ is the binary indicator (0 or 1) if class label $c$ is the correct predicted classification for input sentence $i$ and $p$ is the probability of sentence $i$ having class label $c$. The weights are optimized using Adam optimization with learning rate = 0.0001. The data is fed to the model in 35 epochs.
Multitask Classification Model ::: Results
An overview of the performance results for die/dat prediction is given in Table TABREF19. The same dataset settings as for the binary classification model are used: full in which the datasets contain full sentences, windowed in which sentences are windowed around the unique prediction token without exceeding sentence boundaries (five tokens before and after the token, including token), and windowed no_boundaries in which the windows can exceed sentence boundaries. As mentioned in section SECREF4, we only use the SoNaR dataset. The multitask classification models generally perform better with the windowed no_boundaries dataset setting. Concerning the model architectures, it can be concluded that altering the model architecture has no large impact on model performance for die/dat prediction. However, altering the model architecture from an architecture with merely a sentence encoder to an architecture with both sentence and context encoder does have a more significant positive impact on model performance for POS prediction (Table TABREF20). For that prediction task, the multitask classification model with a bidirectional LSTM context encoder trained and tested on windowed SoNaR sentences reaches best performance results on almost all evaluation metrics.
Discussion
In Section SECREF5, a first classification model based on neural networks is computed to predict die and dat labels. The binary classification model consists of an embedding layer, a bidirectional LSTM, a maxpooling layer and a linear layer. The softmax is taken over the output of the last layer and provides a probability distribution over die and dat prediction labels. The sentences receive the prediction label with the highest probability. It is trained, validated and tested four times using four different database settings. From an analysis of the performance metric results, several conclusions can be drawn. Firstly, in all cases, the model appears to predict the dat label more precisely than the die label. This may be caused by the higher number of dat than die instances in training, validation and test datasets extracted from the Europarl and SoNaR corpus. Secondly, when the dataset is more balanced, as in the SoNaR corpus, the difference in performance between die and dat labels decreases as expected. Thirdly, die/dat prediction performance increases when the window over the sentences is not limited to sentence boundaries (SoNaR windowed, no_boundaries). A probable reason for that higher performance is that the model's ability to detect antecedents in the preceeding or following sentence, while it is not able to do so when it is trained and tested on boundary-constraint windowed sentences (SoNaR windowed). Lastly, it appears that performance of the model drops significantly when the binary classification model is trained and tested on full sentences (Europarl full). In conclusion, the binary classification model performs best when it is trained on the larger, more evenly balanced SoNaR corpus that consists of windowed sentences that are not limited to sentence boundaries. A clear performance overview of the best performing binary classification and multitask classification models for die/dat prediction can be found in Table TABREF21.
In Section SECREF6, several multitask classification models are constructed to jointly execute two prediction tasks: die/dat prediction and POS prediction. The BiLSTM multitask classification model consists of an embedding layer, two consecutive bidirectional LSTMs and a maxpooling layer. The output of the maxpooling layer is used as input to two separate linear layers followed by a softmax layer. The two softmax layers yield a probability distribution for die/dat and POS labels. The model trained and tested on windowed SoNaR sentences that exceed sentence boundaries performs better than the model on boundary-constraint windowed sentences and full sentences. The best performing BiLSTM multitask classification model (Model 2) outperforms the best binary classification model (Model 1) on every evaluation metric for die/dat prediction. This could arguably be due to the increased batch size, the doubled embedding dimension, the extra bidirectional LSTM layer, the influence of the second prediction task and/or the split in sentence and context encoder. Firstly, the data is divided into batch sizes of 512 instead of 128. Table TABREF22 shows, however, that there is little consistent difference in performance when batch size is 512 or 128. Therefore, it can be suggested that an increased batch size has no directly positive influence on model performance. Secondly, the input data is transformed to 200-dimensional word embeddings instead of 100-dimensional word embeddings. From the results displayed in Table TABREF22, it appears that a change in word embedding dimension could be causing an slight increase in model performance. Thirdly, the multitask model contains two bidirectional LSTM layers opposed to the binary model that has only one layer. Table TABREF23 shows the influence of the number of layers on the performance of the binary classification model. When the binary classification model has an additional bidirectional LSTM layer, all the evaluation metrics rise with approximately 2%. However, when the binary classification model has three bidirectional LSTM layers, model performance drops significantly. It appears that the doubled number of layers is indeed one of the reasons why the multitask classification models perform better than the binary classification model. However, not every rise in number of layers necessarily influences a model's performance in a positive manner. Concerning the influence of the POS prediction task on die/dat prediction performance and syntactic knowledge in general, a comparison between a two-layer bidirectional LSTM binary classification model and the two-layer bidirectional LSTM multitask classification model is made and displayed in Table TABREF24. It seems that the integration of POS knowledge positively influences die/dat prediction performance, while all evaluation metrics have increased. When examining the influence of a context encoder on die/dat prediction performance of Model 3 and Model 4, the evaluation metrics of Model 2, 3 and 4 are compared. The metric scores are fairly similar which leads to the conclusion that the addition of a context encoder has little to no further influence on die/dat prediction performance. Moreover, the encoder architecture does not cause a considerable difference in die/dat prediction performance between the model with a feedforward context encoder (Model 3) and the model with a bidirectional LSTM context encoder (Model 4). It can thus be suggested that a model does not necessarily profit from a different architecture and that an extra focus on immediate context is not additionally advantageous for the die/dat prediction task.
Contrary to the little to no impact on die/dat prediction performance, the context encoder - especially the bidirectional LSTM context encoder - does have a direct positive impact on POS prediction performance. The difference in POS prediction performance between the three multitask prediction models can be found in Table TABREF25. The model with the bidirectional LSTM context encoder outperforms the other two multitask classification models on every evaluation metric. Considering its highest POS prediction performance and high die/dat prediction performance, it is suggested that the multitask prediction model with bidirectional LSTM context encoder (Model 4) is the overall best model.
Conclusion
Deciding which pronoun to use in various contexts can be a complicated task. The correct use of die and dat as Dutch pronouns entails knowing the antecedent and - if the antecedent is a noun - its grammatical gender and number. We experimented with neural network models to examine whether die and dat instances in sentences can be computationally predicted and, if necessary, corrected. Our binary classification model reaches a promising 84.56 % accuracy. In addition, we extended that model to three multitask classification models that not only predict die and dat, but also predicts the POS (demonstrative pronoun, relative pronoun and subordinating conjunction). By increasing the word embedding dimension, doubling the number of bidirectional LSTM layers and integrating POS knowledge in the model, the multitask classification models raise die/dat prediction performance by approximately 4 %. Concerning POS prediction performance, the multitask classification model consisting of a sentence and context encoder performs best on all evaluation metrics and reaches a accuracy of 87.78 %.
There are ample opportunities to further analyze, enhance and/or extend the die/dat prediction model. A qualitative study of the learned model weights, for example, could provide more insight in the prediction mechanism of the models. We already obtain excellent results with a simple neural architecture comprising relatively few parameters. We believe that more complex architectures such as a transformer architecture BIBREF13 with multihead attention will improve results. It might also be interesting to look at the possibility of integrating a language model such as BERT BIBREF14 in the classification model (e.g., as pretrained embeddings). Moreover, the binary classification task could be extended to a multiclass classification task to predict not only die and dat labels, but also respectively equivalent deze and dit labels. The difference between die/dat and deze/dat, however, entails a difference in temporal and spatial information: while die/dat indicates a physically near or earlier mentioned antecedent, deze/dit implies that the antecedent is physically distant or later mentioned in the text. That difference may possibly cause a prediction model to base its predictions on other tokens in a text.

Question: What are the sizes of both datasets?
Answer: The Dutch section consists of 2,333,816 sentences and 53,487,257 words., The SONAR500 corpus consists of more than 500 million words obtained from different domains.