## 4 Results
### Predicting Institution Outcome Based on Patent Owner's Preliminary Response Brief
This time-dependent drop in prediction accuracy has been observed in other legal prediction work. As Figure 8 shows, when stop words and pre-processing text are removed, as was done for the written decisions, neither the XGBoost nor CNN method could robustly predict both classes (grant and denial) better than chance. That is, the models perform better on the majority granted class than the minority denial class. The training data are biased towards grants of institution: 56.6% of training cases are grants, and the other 43.4% are denials. The CNN-Attention model produced better performance in a few training runs. Accordingly, subsequent studies described here focus on the CNN-Attention model. Because CNN-Attention takes into account word context, unlike the TF-IDF model, stop words (common words, e.g., articles such as \"the\" and \"an\") are included in the vocabulary. The vocabulary is capped at a maximum size of 20,000 words. Any other words are replaced with an out-of-vocabulary token (\"<OOV>\"). The maximum tokenized document  
Figure 8: Metrics for XGBoost/TF-IDF and CNN-Attention models for predicting the institution outcome (grant/denial) using the full text of the patent owner’s preliminary response brief. The plots show averages with standard deviations as error bars for five (CNN) and three (XGBoost) training runs. The training set consists of 2631 cases filed between 1 July 2018 and 30 November 2020. Results are shown for no maximum vocabulary size but rejecting stop words (labeled “no stop words” as described in Section 3.2, and limiting the vocabulary to 20,000 words but including stop words (“full text”). The \(F1\)-score (harmonic mean between precision and recall) is shown for the grant and denial classes for: (**top**) cases filed between 1 December 2020 and 31 March 2021, and (**bottom**) cases filed between 1 January 2021 and 31 March 2021 (see accompanying text for further details). length is increased to 8000 to avoid cutoffs since inclusion of stop works will lengthen the integer-encoded document vectors used for the CNN model. Figure 8 shows that the CNN-Attention model can achieve adequate performance with the CNN-Attention model. However, the XGBoost/TF-IDF model still fails to provide adequate prediction performance for both classes. Further hyperparameter optimization for XGBoost was un-availing, therefore, it appears that Figure 8 essentially represents the best case scenario for XGBoost performance. By contrast, the CNN model is able to perform better than chance on both the granted and denial cases, with an average of 57.8% accuracy on denial and 57.5% accuracy on grant outcomes. The CNN predictions based on the response brief may be examined by visualizing the attention layers as discussed above. Figure 9 shows selected examples of cases in the test data set which were correctly predicted. Figure 9a shows attention most highlighted (colored most intensely green) where the patent owner argues that the petition \"substantially overlaps\" with another preliminary response (which suggests that they are not raising new arguments) and a technical argument about a \"computer\". The model is correctly classifying the example as a granted institution, i.e., defeat for the patent owner. The highlighting of this argument suggests that rather than making an argument about an _identical_ redundant case, if an argument is about merely \"substantially\" identical claims, the argument will tend to lose. Figure 9b shows the highest attention level (most intense green color) for arguments in which the patent owner touts the worldwide attention that its technology received. Since the patent owner's brief in example (b) was correctly predicted as being granted, it suggests that arguments based on the adoption of the technology by \"world leaders\" may not be persuasive. Figure 9c shows excerpts from a brief that the model correctly predicted as a win for the patent owner (institution denial).