## 4 Results
### Determining Institution Outcome from Text of Written Decision
#### 4.1.2 Interpreting XGBoost and CNN Predictions for Written Decisions  
After classifying written decisions using both XGBoost/TF-IDF and CNN-Attention models, the trained models should be interpreted to obtain insight on what features are relevant to understanding the text of the decision. In particular, we can ask what features the trained model identified as important, or in the case of the deep model, attended to, in making its predictions on the test set. As an initial matter, we evaluate pretrained word embeddings in addition to the TF-IDF and end-to-end deep learning methods shown in this paper. Two different sets of pretrained embeddings were used to transform written decisions in the PTAB dataset: GloVe [46] word embeddings pretrained on a corpus of 2014 Wikipedia entries and Gigaword 5 compendium of English-language news articles (obtained from [[https://nlp.stanford.edu/projects/glove/](https://nlp.stanford.edu/projects/glove/)]([https://nlp.stanford.edu/projects/glove/](https://nlp.stanford.edu/projects/glove/)), accessed on 3 April 2022), as well as Law2 Vec, which is pretrained like Word2 Vec but on a corpus of international legislation, U.S. statutes, and U.S. Supreme Court decisions [101]. In both cases, pretrained embeddings miss words that are particularly relevant for our more specialized domain which includes both legal terms specific to U.S. patent law as well as the technical content of patents. For example, among frequently occurring words that have legal significance in patent law, Law2 Vec missed: embodiments, inherency, unobvious, indefiniteness, and lexicographer. GloVe missed the latter as well as: unpatentable, unpatentability, nonobviousness, preambles, indefiniteness, nonobvious, reexaminations, and patentably. Among the frequently occurring technical terms, by way of example, Law2 Vec missed: microfluidic, rotatable, subcarriers, playlist, oligonucleotide, applet, cached, and encrypted. GloVe missed relatively few common technical terms; however, it did miss certain technical terms that are often used in the patent context, such as \"operably\" and \"removably\", while still missing certain technical terms such as \"photosensor\". To be able to properly interpret documents, therefore, we utilize the document vocabulary (excluding proper nouns, stop words, and other short words) to generate TF-IDF vectors for XGBoost classification and learn embeddings through end-to-end training of the CNN-Attention model. Figure 5 shows the highly ranking SHAP values across the testing data set [76], as well as the highly ranked words based on the \"gain\", i.e., gain in score towards the objective function obtained by splitting trees based on that feature [74; 75]. The highest rank words are similar in the two lists. Unsurprisingly, among the most significant words is \"denied\". That said, because common stop words and short words are excluded (as described in Section 3.2, words of negation such as \"not\" or \"no\" do not contribute to the classification. The classifier must instead make its decision based on longer words. Another significant word, for example, is \"stage\". This is consistent with our intuition that when the PTAB judges are deciding to make a decision to proceed to trial, they will likely note that their findings of a reasonable likelihood of success in the patent are only relevant to this initial stage of the proceeding. A similar intuition holds for the word \"purposes\", which is associated with judges making a preliminary factual or legal determination for the purposes of an initial determination, which they may revise after trial when they make their final decision in the case. Notably, for shorter lengths of text, or smaller vocabularies, it would be possible to look at combinations of words that appear together, such as bigrams, trigrams, \(n\)-grams. However, combinatorial explosion given the vocabulary size and document length makes \(n\)-grams untenable for analyzing full-length written decisions and briefs.