## 4 Results
### Determining Institution Outcome from Text of Written Decision
#### 4.1.2 Interpreting XGBoost and CNN Predictions for Written Decisions
To analyze words in context, the attention vector of the CNN-attention model (see Figure 2) can be used to highlight features across the sequence of tokens [60]. As an example of a granted petition, Figure 6 illustrates case number IPR2020-00806, which was initiated by a petition filed by Comcast Cable Communications LLC on 17 April 2020 to invalidate Rovi Guides Inc.'s U.S. patent number 8,001,564, which covers an aspect of electronic television programming guides. Figure 6 shows an excerpt of the decision text, highlighting tokens according to the relative magnitude of attention at corresponding positions in the attention vector. Figure 6 shows that attention can highlight important arguments in the text, for example, in this case, whether the petition was redundant of other cases and examination. The attention highlights references to \"previously presented\" asserted grounds, and the ability of the judges to use their discretion to deny the petitions on that basis. To compare attention highlighting to other methods of identifying important features, we compare the ranking of the top 20 mean attention paid to particular words (across the positions at which they are found for a kernel width of 8, as shown in Figure 6) to the words with the top 20 words by SHAP value found by XGBoost classification, for the exemplary decision in IPR2020-00806. We further compare the results of using the LIME method, which identifies the features that affect classification the most by perturbing features around a particular example [93], on the IPR2020-00806 written decision. Table 1 shows the ranked lists side by side, along with the top 20 words by TF-IDF for the example. SHAP and attention values agree well with each other, which validates the attention values as a measure of the significance of word features. As the table shows, TF-IDF terms, by contrast, are unique to petitions on patents on distinct technological inventions. By contrast, both XGBoost and CNN classifiers tend to focus on terms more connected to legal and procedural aspects of the decision, such as \"notice\", \"stage\", and \"ordered\". The LIME classification, interestingly, includes both legal terms found by attention, and technical terms specific to the patent at issue in the case, such as \"television\", which was also a term with a very high TF-IDF in this document. This suggests that LIME's methodology of  
Figure 5: Summary plot of the highest ranking words obtained using SHAP (**left**) and the highest ranking words identified by the gain score metric in XGBoost (**right**) in the test set for written decision classification. The SHAP plot shows clouds of points representing individual documents; a positive SHAP value indicates a contribution to a positive classification, i.e., denial, and a negative value a contribution to a classification as a grant. finding local perturbations will tend to identify specific words to the document. Unlike LIME, attention values more consistently reflect terms that are more generally applicable across the category of documents.