Explaining How Transformers Use Context to Build Predictions
Year: 2023

Abstract:
Language Generation Models produce words based on the previous context. Although existing methods offer input attributions as explanations for a model’s prediction, it is still unclear how prior words affect the model’s decision throughout the layers. In this work, we leverage recent advances in explainability of the Transformer and present a procedure to analyze models for language generation. Using contrastive examples, we compare the alignment of our explanations with evidence of the linguistic phenomena, and show that our method consistently aligns better than gradient-based and perturbation-based baselines. Then, we investigate the role of MLPs inside the Transformer and show that they learn features that help the model predict words that are grammatically acceptable. Lastly, we apply our method to Neural Machine Translation models, and demonstrate that they generate human-like source-target alignments for building predictions.

heading: 1 Introduction
text: Language Generation Models, like Transformerbased Language Models (Brown et al., 2020; Zhang et al., 2022a) have recently revolutionized the field of Natural Language Processing (NLP). Despite this, there is still a gap in our understanding of how they are able to produce language that closely resembles that of humans. This means that we are unable to determine the cause of a model’s failure in specific instances, which can result in the generation of hallucinated content or toxic output.
The majority of previous work in explainability of NLP model predictions has focused on analyzing them on downstream tasks, generally with a small output space, such as text classification or Natural Language Inference (Atanasova et al., 2020; Bastings et al., 2022; Zaman and Belinkov, 2022). This line of research includes a large body of work focusing on the analysis of the attention mechanism
(Jain and Wallace, 2019; Serrano and Smith, 2019; Pruthi et al., 2020), and on applying gradient-based methods (Li et al., 2016a; Sundararajan et al., 2017) to obtain input attribution scores.
Recently, several works have tackled the interpretability of Transformers (Vaswani et al., 2017) on the Language Modeling task. Elhage et al. (2021) studied the Transformer from the residual stream perspective, depicted in Figure 1, where different components (MLPs, attention heads...) read and write to subspaces of the residual stream. This
5486
approach has aided in explaining certain behaviours of language models, like induction heads (Olsson et al., 2022), where attention heads search over the context for previous repetitions of the same token and copy the next token, or even specialized heads solving the Indirect Object Identification (IOI) task (Wang et al., 2023). Similarly, MLPs inside the Transformer have also been studied as elements writing into the residual stream. Geva et al. (2022) observed that MLP blocks can act as key-value memories, where values add to the residual, thus promoting the prediction of words that convey similar semantic meaning.
Furthermore, the attention mechanism in the Transformer, composed of attention heads, an output weight matrix, and a layer normalization, can be decomposed into an interpretable operation (Kobayashi et al., 2020, 2021), providing layerwise explanations which have proven to be highly faithful (Ferrando et al., 2022b,a).
In this work, we propose explaining the predictions of Transformers language generators by combining the residual stream analysis perspective with the attention decomposition. Our approach measures the amount of logit (pre-activation of the softmax) added or subtracted by each token representation at each layer. We then track the logit contributions back to the model’s input by aggregating across layers (Logit explanation). Additionally, we consider the mixing of information in intermediate layers by using ALTI (Ferrando et al., 2022b) (ALTI-Logit explanation).
To evaluate the proposed interpretability methods, we follow the recently introduced contrastive explanations framework (Yin and Neubig, 2022), which aims to explain why the model predicted one token instead of a foil token, a priori explained by some linguistic phenomena evidence. Then, we analyze the role of MLPs and show that they aid the model in determining predictions that follow grammar rules. Finally, we demonstrate that NMT models generate human-like source-target alignments for building translations.1

heading: 2 Approach

heading: 2.1 Residual Stream
text: Given a language generation timestep t, the output of the last layer,2 xLt ∈ Rd, is projected to the
1The code accompanying the paper is available at https://github.com/mt-upc/logit-explanations.
2We refer to it as a row vector.
LN
LN
softmax
computer laptop

heading: 2.2 Multi-head Attention as a Sum of Vectors
text: Inspired by the decomposition of the Post-LN selfattention block done by Kobayashi et al. (2021), we apply a similar approach to the Pre-LN setting, common in current LMs (see full derivation in Appendix A). The output of the self-attention block at each generation step t can be expressed as
x̃lt = t∑
j
T lt,j(x l−1 j ) + b l O (5)
where T lt,j : Rd 7→ Rd is an affine transformation applied to each layer’s input token representation (or residual stream) xl−1j ∈ Rd:
T lt,j(x l−1 j ) =
H∑
h
( xl−1j L lW l,hV A l,h t,jW l,h O +A l,h t,jθ l,h )
(6) with W l,hV ∈ Rd×dh the matrix generating the values, W l,hO ∈ Rdh×d the attention output matrix (per head) and blO ∈ Rd its associated bias. Al,h ∈ Rt×t is the attention weight matrix of each head, θl,h ∈ Rd remaining terms originated from biases, and Ll ∈ Rd×d combines centering, normalizing, and scaling operations of the layer normalization (see Appendix A).

heading: 2.3 Layer-wise Contributions to the Logits
text: Combining Equation (4) and Equation (5) we get4:
logitw = L∑
l
oltUw
︸ ︷︷ ︸ ∆logitl
w←MLPl
+
L∑
l
t∑
j
T lt,j(x l−1 j )Uw
︸ ︷︷ ︸ ∆logitl
w←Self-attnl
+x0tUw
(7) The logit’s update of each self-attention, ∆logitl w←Self-attnl , can be expanded into individual
4Biases are removed to save space.
2 and
1
4 to x
l−1
3 , which results

heading: 2.4 Tracking Logit Updates to the Input Tokens
text: that token representations are formed by linearly combining the representations from the preceding layer, i.e. xli = ∑ j c l i,jx l−1 j , with ∑ j c l i,j = 1. Each cli,j refers to the contribution of x l−1 j to x l i. By multiplying the layer-wise coefficient matrices, M l = C l ·C2 · · ·C1, one can describe each intermediate layer representation as a linear combination of the model input tokens, xli = ∑ sm l i,sx 0 s.
Column s of M l−1 contains the proportion of the s-th input token’s contribution encoded in each token representation entering layer l. We can obtain the update performed by each model input token (Figure 3, right) to the logit of a next prediction token w as
∆logitlw←s = ∆logit l w←xl−1 M l−1 s (10)
We refer to Appendix B for a more detailed explanation. The final contribution of the s-th input token to the prediction of token w can be obtained as the sum of its logit updates at each layer:
∆logitw←s = L∑
l
∆logitlw←s (11)
We denote this method the ALTI-Logit explanation. Note that if we don’t consider mixing of contextual information, M l−1 becomes the identity matrix, and we get the Logit explanation (Equation (9)).

heading: 2.5 Contrastive Explanations
text: Contrastive explanations (Yin and Neubig, 2022) aim to explain why the model predicted one target token w instead of another foil token f . We can explain this decision by determining how much each token contributed to the final logit difference between w and f : logit(w−f). Following Equation (9) and Equation (11), we can define the Contrastive Logit and Contrastive ALTI-Logit 5 saliency scores of input tokens as their update to the logit difference:
∆logit(w−f)←s = ∆logitw←s −∆logitf←s (12)

heading: 3 Experimental Setup
text: We evaluate the quality of our proposed method through contrastive explanations. Following Yin and Neubig (2022) we use a subset of BLiMP dataset (Warstadt et al., 2020), which contains sentence pairs with small variations in grammatical
5Throughout the paper we use Logit and ALTI-Logit to refer also to their contrastive variant.
Phenomena ID Example (Acceptable/Unacceptable)
Anaphor Agreement aga Karla could listen to herself/himself. ana Eva approached herself/themselves.
Argument Structure asp Gerald is hated by the teachers/pie.
Determiner-Noun Agreement dna Eva has scared these children/child. dnai Tammy was observing that man/men. dnaa The driver sees that unlucky person/people. dnaai Phillip liked that smooth horse/horses.
NPI Licensing npi Even Danielle also/ever leaves.
Subject-Verb Agreement darn The grandfathers of Diana drink/drinks. ipsv Many people have/has hidden away. rpsv Most associations buy/buys those libraries.
subject, which makes this dataset well-suited for evaluating saliency methods. Indirect object identification (IOI) is a feature present in sentences that have an initial dependent clause, like "After Lee and Evelyn went to the lake", followed by a main clause, like "Lee gave a grape to Evelyn". The indirect object "Evelyn" and the subject "Lee" are found in the initial clause. In all examples of IOI dataset, the main clause refers to the subject again, which gives an object to the IO. The goal of the IOI task is to predict the final word in the sentence to be the IO. In IOI examples, the rule for predicting the IO is the IO itself being in the first clause.
We use GPT-2 XL (1.5B) model (Radford et al., 2019), as in (Yin and Neubig, 2022), as well as other autoregressive Transformer language models, such as GPT-2 Small (124M), and GPT-2 Large models (774M), OPT 125M (Zhang et al., 2022b), and BLOOM’s 560M and 1.1B variants (BigScience Workshop, 2022), through HuggingFace library (Wolf et al., 2020).
Alignment Metrics. Following Yin and Neubig (2022), we define the evidence as a binary vector b ∈ Rt (with as many components as the number of previous tokens), with all zeros except in the position of the tokens inside the evidence, i.e. the tokens which the prediction depends on, extracted by the rule. Explanations are vectors, also ∈ Rt. To measure the alignment between an explanation and the evidence we use MRR (Mean Reciprocal Analysis). Sorting the tokens in descending order, MRR evaluates the average of the inverse of the rank of the first token that is part of b. Although Yin and Neubig (2022) use also dot-product and Probes Needed metrics for measuring alignments, dot-product favors Grad Norm explanations since it gives positive scores only, and Probes Needed is closely related to MRR, giving redundant results.

heading: 4 Contrastive Methods
text: Yin and Neubig (2022) proposed extending different common input attribution methods to the contrastive setting. In §5 we compare their explanations with the ones obtained with our proposed contrastive methods (Equation (12)).

heading: 4.1 Input Erasure
text: Erasure-based methods remove parts of the input and measure the change in the model’s prediction (Li et al., 2016b), where the higher the prediction

heading: 4.3 Gradient × Input

heading: 4.2 Gradient Norm

heading: 5 Results
text: In the following sections we provide results on the alignment between the explanations of different methods and linguistic evidence, as well as an analysis of observed model behaviours through the lens of ALTI-Logit.

heading: 5.1 Alignment Results
text: In Figure 4 we present the MRR results of GPT-2 Small averaged across dataset categories, while the extended results for every subset can be found at Appendix C, Table 7. In Appendix C, Figure 11 we expand Figure 4 across different models. We can observe that Logit and ALTI-Logit explanations consistently align better with the evidence of linguistic phenomena than common gradientbased and erasure-based baselines. Note that for BLiMP the average we show in Figure 4 is across 9 different subsets. In Table 3 we show an example comparing different contrastive explanations, where Grad Norm, G×I and Erasure explanations don’t align with the evidence to solve the subjectverb agreement (report), and disagree between each other.
We find similar alignment results for Logit and ALTI-Logit methods. However, we observe that ALTI-Logit aligns better at tasks where the tokens of the linguistic evidence are far from the prediction. This is especially noticeable in Subjectverb agreement datasets (including SVA and darn), where ALTI-Logit shows higher alignments than any other method across all models. This might indicate that incorporating information about contextual mixing is advantageous for dealing with large contexts.
Despite the generally accurate performance of the models examined in this study (Figure 12 and Figure 13, Appendix D), there are cases where
the unacceptable token gets predicted with a higher probability. In order to gain a deeper understanding of the variations in model behavior between correct and incorrect predictions, we analyze the logit update generated by the input tokens associated with the linguistic evidence. This analysis, conducted using ALTI-Logit (Figure 5), reveals differences in the distributions. These findings suggest that the tokens representing the linguistic evidence play a crucial role in achieving accurate predictions, and if their contribution is only marginal, the likelihood of failure increases considerably.

heading: 5.2 Layer-wise Analysis with ALTI-Logit
text: In the line plots in Figures 6 and 7 we provide the MRR alignment results across layers of GPT2Small and GPT2-XL for two different linguistic phenomena. Models behave similarly across subsets inside the same phenomena, like in SubjectVerb Agreement (SVA and darn), and Anaphor Agreement (aga and ana) in Appendix E. The model’s alignment trend also stays similar, even though the distance between the prediction and the evidence is different across subsets (SVA’s distance is 4 times darn’s).
In the boxplots in Figure 6, we show the distribution of self-attention updates to the logit difference between the acceptable and the unacceptable predictions, ∆logitl
(w−f)←Self-attnl . As a general pattern, we observe that models tend to update more heavily on the layers where the alignment with linguistic phenomena is higher. This conclusion holds for larger models too, see the darn example in Appendix H.2, where large logit updates are found in layers 28, 35, and 40, matching the layers where alignment peaks (Figure 7 Top). In IOI and SVA tasks both models align with the evidence and increase their logit update towards the last layers. This indicates that models solve these phenomena once they have acquired sufficient contextual information.
Our findings in the IOI task support those by Wang et al. (2023). In GPT-2 Small we observe high logit difference updates coming from the Indirect Object (IO) in layers 10 and 11. We further study the heads in those layers (Table 4), where Wang et al. (2023) found ‘Name Mover Heads’ and ‘Negative Mover Heads’. These heads rely on the IO to increase (Name Mover Heads) and decrease
Name Mover Head L10 H7 Then, Yvette and Angie were working at the mountain. Yvette decided to give a banana to Angie Name Mover Head L10 H10 Then, Yvette and Angie were working at the mountain. Yvette decided to give a banana to Angie Name Mover Head L11 H1 Then, Yvette and Angie were working at the mountain. Yvette decided to give a banana to Angie Negative Name Mover Head L11 H8 Then, Yvette and Angie were working at the mountain. Yvette decided to give a banana to Angie Negative Name Mover Head L12 H11 Then, Yvette and Angie were working at the mountain. Yvette decided to give a banana to Angie

heading: 6 Analysis of MLPs
text: each kliv l i (weighted row in W l 2). The update in the logit’s difference between the target and foil tokens by each value i is therefore:
∆logitl (w−f)←klivli = ∆logitl w←klivli −∆logitl f←klivli
(19) In Figure 8, we show some examples of the contribution of each weighted value kliv l i to the logit difference between the acceptable target token and the unacceptable one, at different layers and datasets. We can observe that there is a small subset of values that consistently increase the difference in logits helping to solve the linguistic task. Some of them include the value i=383 in layer 10 (Figure 8 (a)), which increases the logit of singular nouns and reduces the plural ones when the determiner is this or that. For instance, in the sentence “William described this ___”, value i=383 increases the logit difference between movie and movies. In dimension 3038 we find a value upweighting the logits of the plural nouns over the singular ones when the determiner is these or those (Figure 8 (b)). These values help solve the linguistic task at hand across different subsets, for instance, the value in dimension i = 2187 is in charge of promoting the singular form of the verb when the head of the subject is singular too. This occurs in both darn and SVA subsets.
LN
LN
source
target prefix
target prefixsource

heading: 7 Neural Machine Translation
text: Alignment. Source-target alignments derived from attention weights in NMT systems can be unreliable (Zenkel et al., 2019; Li et al., 2019; Garg et al., 2019), with upper layers producing better alignments. A limitation of using this method to interpret model predictions is that the ground truth target word may not match the model’s actual prediction. However, by measuring how the encoder token representations update the logits of the reference words, ∆logitlw←ei , we can more precisely explain which source word causes the final logit of the reference word, even if it is not one of the top predictions.
Following Kobayashi et al. (2020) and Ding et al. (2019) setting, we train a 6-layer Transformer model for the German-English (De-En) translation task using Europarl v7 corpus9 Koehn (2005). We also evaluate on M2M, a 12 layer multilingual model (Fan et al., 2021). We use Vilar et al. (2006) dataset, consisting of 508 De-En human annotated sentence pairs with alignments, and compare them with our extracted alignments using Alignment Error Rate (AER). We also show results of other attention-based alignments extraction methods. Vector Norms take the norm of the transformed vectors in Equation (21), Distance Vectors-Output measures the distance between the transformed vectors and the attention block output ỹc,lt . SD-SmoothGrad relies on gradients to extract alignments. In Table 5 we show that our proposed method achieves lower AER values, which indicates that NMT models generate human-like alignments for building model predictions.

heading: 8 Related Work
text: The projection of LMs representations and model parameters to the vocabulary space has been a subject of previous research (Belrose et al., 2023; Din et al., 2023). Geva et al. (2021, 2022) view feed-
9http://www.statmt.org/europarl/v7

heading: 9 Conclusions

heading: 10 Limitations
text: The experimental methodology employed in this study for both contrastive explanations and NMT is not directly extensible to languages other than English, due to the scarcity of resources such as models and annotations.
The datasets employed in this study to evaluate contrastive explanations across various linguistic paradigms are restricted to sentences that possess a well-defined structure. As a result, it is possible that the conclusions drawn may not be generalizable to the broader distribution of sentences.
Lastly, it should be noted that the method proposed in this study should not be used as a definitive explanation of model predictions in any other context. It is recommended to use the method as a debugging tool and should be employed in conjunction with other methods to gain a comprehensive understanding of model predictions.

heading: 11 Ethics statement
text: It is acknowledged that the experiments reported in this study are limited to high-resource languages. However, the methodology employed is languageindependent and may be applied to other languages in the future, provided that adequate annotated data becomes available.

heading: 12 Acknowledgements
text: We would like to thank the anonymous reviewers for their useful comments. Javier Ferrando, Gerard I. Gállego and Ioannis Tsiamas are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00 / AEI / 10.13039/501100011033.

heading: A Pre-LN Self-attention Decomposition
text: At position t, each head of a Pre-LN self-attention mechanism computes:
zl,ht = t∑
j
( LNl(xl−1j )W l,h V + b
l,h V︸ ︷︷ ︸
j-th value
) Al,ht,j (23)
By representing attention heads as parallel independent components, we can express the output of the self-attention as
x̃lt = H∑
h
zl,ht W l,h O + b l O (24)
leading to:
x̃lt =
t∑
j
H∑
h
( LNl(xl−1j )W l,h V + b l,h V ) Al,ht,jW l,h O + b l O (25)
The layer normalization computes:
LNl(xl−1j ) = xl−1j − µ(xl−1j )
σ(xl−1j ) ⊙ γl + βl (26)
with µ and σ computing the mean and standard deviation, and γl ∈ Rd and βl ∈ Rd refer to learned element-wise transformation and bias respectively. Considering σ(xl−1j ) as a constant, LN can be treated as a constant affine transformation:
LN(xl−1j ) = x l−1 j L l + βl (27)
where Ll ∈ Rd×d represents a matrix that combines centering, normalizing, and scaling operations together.
Using Equation (27) in Equation (25):
x̃lt =
t∑
j
H∑
h
(( (xl−1j L l + βl)W l,hV + b l,h V ) Al,ht,jW l,h O ) +blO
= t∑
j
H∑
h
(( xl−1j L lW l,hV + β lW l,hV + b l,h V ) Al,ht,jW l,h O ) +blO
= t∑
j
H∑
h
( xl−1j L lW l,hV A l,h t,jW l,h O + β lW l,hV A l,h t,jW l,h O + b l,h V A l,h t,jW l,h O ) +blO
= t∑
j
H∑
h
( xl−1j L lW l,hV A l,h t,jW l,h O +A l,h t,j ( βlW l,hV W l,h O + b l,h V W l,h O )) +blO (28)
Considering θl,h = ( βlW l,hV + b l,h V ) W l,hO
x̃lt = t∑
j
H∑
h
( xl−1j L lW l,hV A l,h t,jW l,h O +A l,h t,jθ l,h ) +blO (29)
For each j-th input term, H affine transformations are applied to xj . Furthermore, all heads’ operations can be further grouped into a single affine transformation:
x̃lt =
t∑
j
( xl−1j L l H∑
h
W l,hV A l,h t,jW l,h O +
H∑
h
Al,ht,jθ l,h ) +blO (30)
So, we can write x̃lt as a sum of t affine transformations, and the output bias:
x̃lt =
t∑
j
T lt,j(x l−1 j ) + b l O (31)

heading: B Tracking Logits to the Input with Rollout
text: The rollout method (Abnar and Zuidema, 2020) assumes any intermediate representation is a linear combination of the model inputs, xl−1j = ∑ sm l−1 j,s x 0 s , where m l−1 j,s is a score indicating the contribution of input token s to the l − 1 representation (or residual path) of token j. By dividing the logit update performed by xl−1j among the model inputs (∆logit l w,j←x0s ) based on their contributions to x l−1 j , we obtain:
∆logitl w←xl−1j
= ∑
s
∆logitlw,j←x0s
= ∑
s
ml−1j,s ∆logit l w←xl−1j
(32)
Based on the total logit update produced in layer l, we have that:
∆logitl w←Self-attnl =
∑
j
∆logitl w←xl−1j
= ∑
j
∑
s
∆logitlw,j←x0s
= ∑
j
∑
s
ml−1j,s ∆logit l w←xl−1j
= ∑
s
∑
j
ml−1j,s ∆logit l w←xl−1j
= ∑
s
∆logitlw←s (33)
So, we have obtained Equation (10):
∆logitlw←s = ∆logit l w←xl−1 M l−1 s (34)

heading: C Results
text: C.1 GPT-2 Small Results
C.2 GPT-2 XL Results

heading: E MRR Alignment across layers

heading: D Model Predictions

heading: F MLPs Logit Difference Update

heading: G Self-attention Logit Difference Update

heading: H Qualitative Contrastive Exaplantions
text: H.1 Explanations of Different Contrastive Methods
H.2 GPT-2 XL ALTI-Logit across layers
H.3 ALTI-Logit (IOI) across Models
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 9
3 A2. Did you discuss any potential risks of your work? Section 9
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 2
3 B1. Did you cite the creators of artifacts you used? Section 2
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 9
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3
C 3 Did you run computational experiments? Section 3 and 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Not applicable. Left blank.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: However, the methodology employed is languageindependent and may be applied to other languages in the future, provided that adequate annotated data becomes available