TüReuth Legal at SemEval-2023 Task 6: Modelling Local and Global Structure of Judgements for Rhetorical Role Prediction
Year: 2023

Abstract:
This paper describes our system for SemEval2023 Task 6: LegalEval: Understanding Legal Texts. We only participate in Sub-Task (A), Predicting Rhetorical Roles. Our final submission achieves 73.35 test set F1 score, ranking 17th of 27 participants. The proposed method combines global and local models of label distributions and transitions between labels. Through our analyses, we show that especially modelling the temporal distribution of labels contributes positively to performance.

heading: 1 Introduction
text: This paper describes our system for SemEval-2023 Task 6: LegalEval: Understanding Legal Texts. While the shared task hosts 3 sub-tasks, we only participate in sub-task (A), predicting Rhetorical Roles (RRs) from pre-segmented judgments. Refer to the overview paper (Modi et al., 2023) for a detailed description of the task and data.
Briefly, the task is as follows: Given a judgment that is pre-segmented into paragraphs (often containing only 1 sentence) and an inventory of Rhetorical Roles, predict the appropriate RR for every paragraph. RRs structure the judgment into semantically coherent units and provide information on what is discussed in each unit. For example, the RR PREAMBLE denotes formal introductory statements of judgments. In total, there are 13 RRs in this task.
The shared task provides 245 segmented and labelled Indian judgements (in English) for training, 30 segmented and labelled judgments for development, and 50 segmented but unlabelled test set judgments for evaluation. Test set labels are not released by the organisers.
The main challenge of this task is how to integrate document level information and paragraph level information. Since solving the task requires precise understanding of paragraph content
and the context, fine-tuning pre-trained large language models (LMs) is the common starting point. However, LMs can not process long documents such as judgments on standard hardware at once. Therefore, we develop and evaluate a method to use document-level information in order to refine paragraph-level predictions from a fine-tuned LMs. In particular, we model the positions in a document where RRs usually occur, structural constraints that indicate whether some RRs may or may not be predicted when other RRs have already been predicted earlier, and lexical constraints motivated by domain knowledge.
Throughout this paper, we assume the following notation: R is the set of Rhetorical Roles, r ∈ R denotes some RR, a judgement J consists of a sequence of paragraphs s1, . . . , sT where T is the number of paragraphs in J . Paragraphs are indexed by t (we think of them as a time-series).

heading: 2 Related Work
text: Closest to our method is previous work by the shared task organisers (Malik et al., 2021; Kalamkar et al., 2022). Both works propose to first encode each sentence of a judgement by LM. Then, another sequence processor, such as LSTM or CRF, is applied to predict RRs from the sequence of sentence encodings. Kalamkar et al. (2022) focus on how to arrive at sentence encodings from token encodings computed by a BERT model. On the other hand, Malik et al. (2021) focus on label shift. They train additional models to predict whether 2 consecutive sentences have the same RR. Then, they use this information as feature in their sequence labelling models.
In this work, we continue the approach of Malik et al. (2021) and include more structural modelling. However, in our experiments we found models to predict whether consecutive sentences share the same RR to yield insufficient performance on the negative class (when RRs are different). Most
262
likely, the reason is that, as noted by Malik et al. (2021), most consecutive sentences share the same RR, resulting in very imbalanced data. Therefore, we choose a binary model of whether certain transitions are possible or not instead, which is less expressive but trivial to learn.
Furthermore, treating whole judgments as datapoints for supervised sequence labelling yields only 245 train datapoints for this shared task, which may be problematic because of overfitting. Also, endto-end training of sentence encoders and sequence labelling models is very expensive. Therefore, we do not use any sequence level neural models, but combine scores from individual models during inference.

heading: 3 Method
text: Our method combines sentence-level information with document-level information in order to improve performance of models only using limited contexts. Concretely, we
• Fine-tune a pre-trained language model to predict RRs from individual sentences
• Learn a conditional distribution of RRs given all RRs predicted so far
• Learn a binary score for all ordered pairs of RRs that indicates whether the respective transition from the previous RR to the following RR is allowed
• Learn a continuous distribution over positions in the document that indicates how likely it is for the respective RR to appear at this position in the document
• Hard code lexical rules based on domain knowledge
In the following, we describe each of the individual models in detail. Then, we describe how to combine all these models to assign labels to all paragraphs of a given judgement.

heading: 3.1 Models
text: This section contains descriptions of the different models of local or global document information. Sentence-level LMs, Local Transition Scores, and Domain Knowledge model information about individual or adjacent paragraphs. Position Scores and Global Transition Scores model global distributions or constraints regarding where RRs may appear in a given judgement.
Language Model Scores We use NLTK (Bird et al., 2009) to sentence tokenize each judgment. Each sentence is assigned the rhetorical role of its corresponding paragraph in the training data. Then, we fine tune the roberta-base model (Liu et al., 2019) for 2 epochs with learning rate 2 · 10−5 and no weight decay. Given a sentence s, the LM computes a probability distribution over RRs.
For inference, in case a paragraph contains multiple sentences, we multiply probabilities for each sentence to get probabilities for the whole paragraph. We refer to the prediction probabilities of RR r ∈ R for paragraph st as LM(st, r). LM may also represent an ensemble of LMs by averaging prediction probabilities.
Global Transition Scores We learn conditional Bernoulli distributions Brnext,A(t) that indicate the probability of a RR given the complete history of RRs predicted so far. The history of previously predicted RRs is represented by a vector A(t) = [ Ak(t) | k ∈ R ] ∈ {0, 1}|R| of binary variables Ak(t) ∈ {0, 1} that indicate whether RR k ∈ R has been predicted as label for any paragraph s1, . . . , st.
Intuitively, this distribution models interactions of predicted RRs that block or reinforce prediction of some RRs later in the judgement. For example, after having predicted PREAMBLE and any other RR, it is not possible to predict PREAMBLE again.
Because we cannot empirically estimate parameters for all distributions Brnext,A(t) due to data sparseness, we estimate parameters of the distributions in the following way: We construct a dataset Xglobal, Yglobal where Xglobal contains all vectors A(t) actually encountered in the dataset and Yglobal contains binary vectors whose components indicate whether the respective RR appears as label of any paragraph st+1, . . . , sT . Then, we learn a MLP to map binary vectors A(t) to probabilities that indicate how likely it is for any rhetorical role to still appear in the judgement. The MLP is trained by minimising the binary cross entropy loss when predicting labels ∈ Yglobal from Xglobal. This allows us to generalise to vectors A(t) not encountered in the training set.
Local Transition Scores For each ordered pair of RRs (rprev, rnext) ∈ R×R we store whether the respective transition is encountered in the dataset. To align this information with the other probability
based scores, we represent transitions that appear in the dataset as a Bernoulli distribution Brprev,rnext with parameter 1, and we represent transitions that do not appear in the dataset as a Bernoulli distribution Brprev,rnext with parameter 0.
Furthermore, we learn a Multinomial prior distribution Prior over all RRs that indicates how likely the respective RR is to be the label of the first paragraph s0 in any judgement.
Position Scores For each RR r ∈ R, we learn a continuous distribution Posr over the interval [0; 1]. Posr(t) for t ∈ [0; 1] indicates the likelihood of RR r appearing at relative position t in the judgment. In reality, there is a finite number of paragraphs s1, . . . , st, . . . , sT in each judgment, meaning relative positions tT for all t = 1, . . . , T are discrete and not continuous. To generalise to judgments of different lengths, however, we relax this property and assume continuous relative positions.
Each distribution Posr is modelled as a mixture of two Beta distributions, which are defined on the interval [0; 1]. Therefore, each distribution Posr has 5 parameters, α1 and β1 (the parameters of the first Beta distribution), α2 and β2 (the parameters of the second Beta distribution), and w ∈ [0; 1] (the mixture weight). Then, the density of Posr is expressed as
Posr(t) =
[ w · Beta(t, α1, β1)
+(1−w) · Beta(t, α2, β2)
We learn parameters of distributions Posr for each RR by collecting all relative positions where r appears in any judgement of the dataset. If r is the RR assigned to paragraph st in a judgment with T paragraphs in total, the collected value is t T . Then, we map relative positions to one of 100 discrete bins. From these binned relative positions, we compute the empirical CDF (i.e. cumulative histogram) and fit the mixture of Beta distributions by black-box optimisation using the minimize function of the scikit-learn library (Pedregosa et al., 2011) with L-BFGS-B optimization method. The minimised loss is the euclidean distance of the CDF of Posr (evaluated at bin boundaries) and the empirical CDF of the RR as described above.
In Figure 1, we show the resulting distributions for some RRs. In particular, we want to highlight that different RRs appear in different places in judgments and we exploit this property by explicitly modelling the temporal distribution of RRs in a judgment.
Domain Knowledge Often, statistical methods may be complemented by domain knowledge in order to refine predictions. This is especially the case in venerable and long-standing fields such as law, where experts have a very good understanding of the subject. In the following, we introduce relevant concepts observed in the given data:
While legal documents may be unstructured (Kalamkar et al., 2022), court decisions always follow the same structure. After a preamble, the court must answer the legal questions raised in the case and reach a final decision based on the facts, the relevant statute law, the case law, and the arguments of the lawyers. Therefore, we implement rules that prevent predicting PREAMBLE when any RR other than NONE was already predicted after PREAMBLE. This applies not only to PREAMBLE, but also prediction of RATIO blocks the prediction of PREAMBLE, FAC, RLC, ISSUES, ARGUMENT BY PETITIONER, ARGUMENT BY RESPONDENT, ANALYSIS, STA, PRECEDENT RELIED and PRECEDENT NOT RELIED, whereas the prediction of RPC also prevents the prediction of RATIO.
Additionally, we prevent predicting certain RRs when certain conditions are met. First, when one of the strings “JUDGMENT”, “J U D G M E N T”, “ORDER”, “ORDERS”, and “O R D E R” appear in a paragraph labelled as PREAMBLE, the next paragraph cannot be labelled as PREAMBLE.
Furthermore, we specify phrases that, if present, enforce or prevent prediction of certain RRs. For example, the phrases “appeal allowed” and “appeal dismissed” enforce prediction of RR RPC, whereas the phrase “learned counsel” prevents the prediction of any RR of PREAMBLE, FACTS, RLC, ISSUES, STA, PRECEDENT RELIED and PRECEDENT NOT RELIED. The full list of phrases is in Appendix B.

heading: 3.2 Inference
text: When assigning RRs to paragraphs of an unseen judgement at test time, we use all information described in Section 3.1. The judgment consist of pre-segmented paragraphs s1, . . . , sT . We model global information by states S(t, r, A) that consist of the current paragraph index t ∈ {1, . . . , T}, the assigned RR r ∈ R, and the binary vector A ∈ {0, 1}|R| that indicates which RRs have already been predicted for paragraphs s1, . . . , st. The maximum probability path through states is computed by the Viterbi algorithm. In the following, we describe how to compute non-zero state transition scores. Let Sprev = S(t, rprev, A) and Snext = S(t+ 1, rnext, A
′), where A′ is identical to A, but the binary indicator for rnext is also set to 1 (if not already the case). The transition probability Pr[Sprev → Snext] is calculated as
Pr[Sprev → Snext] =   LM(st, rnext)
·Brnext,A ·Brprev,rnext ·Posrnext(t+ 1)
In the case of the first paragraph, Brprev,rnext is replaced with the prior probability Prior(rnext), rprev is undefined and A contains only 0. To avoid numerical problems, all scores (which are probabilities and therefore ≥ 0) are transformed to log space. Finally, lexical domain knowledge is used to enforce or block certain transitions, i.e. by setting Pr[Sprev → Snext] to 0 irrespective of other scores.
Remember that all transitions not described above have 0 probability. In particular, it is only possible to transition to a state of the next paragraph index. This enforces that each paragraph is assigned exactly one label by each path.
Note that since both T and R are finite, the set of states is also finite. However, it is very large (containing > 10K states for each paragraph index). Therefore, we only consider the 100 states with maximum score for each paragraph index t to make inference efficient. Doing so, we do not notice any performance degradation.

heading: 4 Results
text: In this section, we present the final performance of our method and further insights regarding the individual models described in Section 3.1.
Test Set Results On the test set, our approach achieves 73.35 micro-averaged F1 score.1 In the shared task ranking, it achieves rank 17 of 27. This score is achieved when using all models described in Section 3.1 and only using train data to fit models. When combining train and development data, we do not get a better result in this case. We assume the reason is the the amount of train data is already sufficient to reach maximum performance of all models except fine-tuning the LMs. Also note that LM scores are averaged prediction probabilities of an ensemble of 5 roberta-base models fine-tuned independently.
Ablation Study In the following, we evaluate the contribution of individual models. Because the number of test set evaluation runs was limited by the organisers and test labels are not released, we report ablations using the development set for evaluation.
First, observe the progression in Table 1. Clearly, the most important part by far is the LM, which extracts the semantic information from sentences. This is a conditio sine qua non. RRs cannot be correctly inferred without knowledge of the text’s meaning. Then, Position Scores boost performance by almost 4 points. This shows that modelling global document structure is helpful, especially because Positional Scores can be integrated without much effort into most sequence labelling models. This is not the case for Global Transition Scores, which require keeping track of previously predicted labels and therefore make inference more complicated. Still, Global Transition Scores lead to a performance improvement of almost 1 point. In contrast, Local Transition Scores do not yield further improvement of performance. Most likely, our chosen model, only allowing transitions observed in the training set, is not expressive enough to contribute useful information beyond sentence-level predictions by the LM. Therefore, continuing work by Malik et al. (2021) and including more expressive models of local transitions could be interesting future work. Finally, lexical rules informed by domain knowledge yield another small improvement of a little over 1 point. This is in line with many observations that when given sufficient data, statistical approaches yield better performance, especially because manually coding rules can only reach sufficient recall with a lot of effort. However, the fact that there is improvement shows that hu-
1We report scores in range 0 to 100 for better readability
mans are still able to identify high-precision rules that are not obeyed by LMs.
In the next analysis step, we want to quantify the contribution of individual models. To this end, we propose the following method: For each individual model M (i.e. LM, Pos, . . . ) and configuration of used models, we substract the f1 score achieved when not using M from the f1 score achieved when using M . Then, for each individual model M , we calculate the mean over all differences. This score gives an estimate of the performance improvement when adding the respective model to any subset of other models. However, we only evaluate configurations that use LM scores, because not using them does not make sense. The results are:
M Pos GTS LTS DK ∆ 2.98 2.06 0.25 0.64
where ∆ is the average difference between configurations using the model and configurations not using the model.
The results show that all models contribute positively, i.e. no model reduces performance on average. The strongest effect is achieved by the global models. Local models do not yield such improvements, most likely because of the weaknesses discussed above. Also note that the cumulative difference scores very closely match the performance difference between only using LM and using all models in Table 1. Scores for all configurations are in Appendix A.
Choice of LM We experimented with other LMs than roberta-base, which we eventually use for analyses in this paper and our submission to the shared task. Other LMs we tried include but are not limited to T5 (Raffel et al., 2020), SciBERT (Beltagy et al., 2019), and Legal-BERT (Zheng
et al., 2021). Even if using other LMs eventually yields better performance on the test set, the negative results on the development set show that the gains are not consistent. Therefore, to make more use of the capabilities of pre-trained LMs, a different setup (or possibly fine-tuning hyperparameters) than explored here would be necessary.
However, we still want to demonstrate that the results of our analysis are not an artifact of the language model we use, which arguably is the component that influences performance the most. Therefore, we replicate the ablation study from Table 1 for Legal-BERT and also report the results in Table 1.
Here, we can see that the performance of the LM alone is weaker, but in contrast the performance gain resulting from including Position Scores is much higher. Perhaps not surprisingly, this shows that global modelling is more beneficial when using a very strong local model, such as LM, is not possible or feasible. On the other hand, effects of all other models are reduced:
M Pos GTS LTS DK ∆ 7.15 1.41 0.81 0.17
This shows that the general observations regarding different models agree also for different LMs.

heading: 5 Conclusion
text: In this work, we propose and evaluate different models of global and local judgment structure. Besides using pre-trained language models, the greatest performance gains result from modelling the temporal distribution of labels, i.e. where in the document a label occurs more often. Furthermore, this information can be easily integrated with most sequence labelling methods. Otherwise, we also show that modelling how the possibility of predicting RRs depend on previously predicted RRs and integrating domain knowledge improve performance to a lesser degree. We do not find relevant improvements from restricting transitions between RRs to such transitions that appear in the training data.
As possible next steps, we think it would be interesting to improve performance of LMs, e.g. by showing them more context, and to develop better models of local transitions between RRs. Finally, we make our code available on GitHub.2
2https://github.com/LGirrbach/ Tuereuth-Legal-at-SemEval-Task-6

heading: Acknowledgements
text: We thank the organisers of the shared task for their efforts and the reviewers for helpful comments and suggestions.

heading: A Full Ablation Results
text: In Table 2, we list the full ablation results, using an ensemble of 5 independently fine-tuned roberta-base models as LM. Note that we exclude trivial configuration such as not using any model or only using domain knowledge.

heading: B Lexical Rules
text: In Table 3, we list all phrases that prevent prediction of a certain RR. This means, if a phrase from the list given for a RR appears in a paragraph, the respective RR can not be predicted as RR of the paragraph. All phrases are case sensitive. Furthermore we include the following lexical rules:
• Paragraphs that are all in caps are PREAMBLE or NONE
• Parapgraphs with relative position > 0.9 that inlcude the word “costs” are RPC
• Paragraphs that contain one of “petition” or “appeal” and also contain one of “is dismissed” or “is allowed” are RPC

heading: Future Work
text: (2021) and including more expressive models of local transitions could be interesting future work. Finally, lexical rules informed by domain knowledge yield another small improvement of a little over 1 point. This is in line with many observations that when given sufficient data, statistical approaches yield better performance, especially because manually coding rules can only reach sufficient recall with a lot of effort. However, the fact that there is improvement shows that hu-
1We report scores in range 0 to 100 for better readability
mans are still able to identify high-precision rules that are not obeyed by LMs. In the next analysis step, we want to quantify the contribution of individual models. To this end, we propose the following method: For each individual model M (i. e. LM, Pos,. ) and configuration of used models, we substract the f1 score achieved when not using M from the f1 score achieved when using M. Then, for each individual model M , we calculate the mean over all differences. This score gives an estimate of the performance improvement when adding the respective model to any subset of other models. However, we only evaluate configurations that use LM scores, because not using them does not make sense. The results are:
M Pos GTS LTS DK ∆ 2. 98 2. 06 0. 25 0. 64
where ∆ is the average difference between configurations using the model and configurations not using the model. The results show that all models contribute positively, i. e. no model reduces performance on average. The strongest effect is achieved by the global models. Local models do not yield such improvements, most likely because of the weaknesses discussed above. Also note that the cumulative difference scores very closely match the performance difference between only using LM and using all models in Table 1. Scores for all configurations are in Appendix A. Choice of LM We experimented with other LMs than roberta-base, which we eventually use for analyses in this paper and our submission to the shared task. Other LMs we tried include but are not limited to T5 (Raffel et al. , 2020), SciBERT (Beltagy et al. , 2019), and Legal-BERT (Zheng
et al. , 2021). Even if using other LMs eventually yields better performance on the test set, the negative results on the development set show that the gains are not consistent. Therefore, to make more use of the capabilities of pre-trained LMs, a different setup (or possibly fine-tuning hyperparameters) than explored here would be necessary. However, we still want to demonstrate that the results of our analysis are not an artifact of the language model we use, which arguably is the component that influences performance the most. Therefore, we replicate the ablation study from Table 1 for Legal-BERT and also report the results in Table 1. Here, we can see that the performance of the LM alone is weaker, but in contrast the performance gain resulting from including Position Scores is much higher. Perhaps not surprisingly, this shows that global modelling is more beneficial when using a very strong local model, such as LM, is not possible or feasible. On the other hand, effects of all other models are reduced:
M Pos GTS LTS DK ∆ 7. 15 1. 41 0. 81 0. 17
This shows that the general observations regarding different models agree also for different LMs