## 4 Results
### Determining Institution Outcome from Text of Written Decision
#### 4.1.1 Comparing Performance of XGBoost and CNN Methods
The training data are made up of written institution decisions of grant or denial (i.e., excluding mixed decisions) on petitions filed through 31 December 2017 (which generally implicates decisions through 30 June 2018, since the decision is due 6 months after the petition is filed). The test data are made up of institution decisions on petitions filed 1 January 2019 and published up to the data collection cutoff of 31 December 2021. Moreover, as Figure 3 shows, up to 2018, a substantial fraction of institution decisions had mixed outcomes (some claims and grounds instituted, others now). As explained in the Background section, in April 2018 the U.S. Supreme Court ruled that mixed institution outcomes are unlawful. As the graph suggests, in the aftermath of the decision, the grant rate increased for 2018 petitions, as PTAB judges were obligated to grant a petition in full even if only a single ground or claim was deemed likely to succeed. Over time, this normalizes somewhat, corresponding to PTAB's increased rejection of petitions on the basis of being duplicative. Figure 4 compares the performance of the two classification methods used for this paper, as detailed in Sections 3.3 and 3.4: (1) classifying documents represented as vectors of TF-IDF for all words in the vocabulary (after removing short words and stop words as described in Section 3.2), and (2) classifying documents represented as vectors of integer  
Figure 3: Institution decisions by petition filing year in the data set (combining IPR, PGR, and CBM as we do for the analysis in this paper) from the AIA’s effective date in September 2012 through the data set cutoff of 31 December 2021, showing grant (blue, bottom), denial (orange, top), and mixed (brown, middle) outcomes. Notably, in 2018, the U.S. Supreme Court’s _SAS_ decision eliminating the possibility of a mixed outcome. tokens using the CNN model with attention shown in Figure 2. As a baseline, logistic regression was performed using sklearn in Python 3.7 [98] with a limited vocabulary. As Figure 4 shows, the most accurate and best method at avoiding type I and type II errors (i.e., with good specificity and sensitivity) is XGBoost with TF-IDF document vectorization. Logistic regression underperforms both XGBoost and the CNN-Attention model. Interestingly, the best performance was found for logistic regression when limiting the vocabulary size to the 2000 most frequent words. As the vocabulary size increases, logistic regression performance deteriorates, suggesting potential overfitting. The CNN-Attention method's underperfomance appears to be largely due to problems with identifying denials. This is not necessarily because denials were underrepresented in the training data. The training data set was 53% grants, whereas the test data set was 59% grants. As discussed above, this difference is because mixed outcomes were excluded from the training set, and the test set is entirely from after when mixed outcomes were disallowed. However, the time frames for the training and test sets are also not connected to the CNN-Attention model's underperformance on denials. The model run for training set petitions filed between 1 June 2018 and 31 December 2019, i.e., all after mixed outcomes were eliminated, and the training set petitions filed after 1 April 2020, provided similar results: the average recall for the granted class was 0.92 and the average recall for denial class was 0.62. Figure 4: Results of predicting grant or denial in written decisions on petitions filed after 1 January 2019, for the model trained on written decisions for petitions filed up to 31 December 2017. The metrics are averaged over 5 runs for CNN-Attention and 3 runs for XGBoost (error bars indicate standard deviation). The graph at the upper left shows accuracy, calculated by dividing the number of correct predictions by the test set size.