## 4 Results  
### Determining Institution Outcome from Text of Written Decision  
#### 4.1.1 Comparing Performance of XGBoost and CNN Methods  
To determine the viability of text classification for PTAB documents, and validate our approaches to interpretable machine learning, this paper starts with the classification of written institution decisions (institution decisions) issued by PTAB judges. As explained in the Background section, at the institution stage, PTAB judges determine whether a petition for post-grant review should proceed to trial. Their decision is announced in a written opinion that states the decision and explains the factual and legal bases for their decision as well as their reasoning. The decisions do not follow a fixed pattern or structure, and the judges may use different kinds of language to announce the decision; therefore, a form-based search cannot reliably derive the outcome from the text of the written decision. We pursue the task of using interpretable machine learning to classify written document text, and use that classifier to predict whether a decision is a grant or denial of institution. One important consideration is that a patent challenger may file multiple petitions against the same patent, and, because patent owners may have filed a lawsuit against multiple defendants, different patent challengers may also challenge the same patent. Because the petitions may expose similar weaknesses in the patent, the result of the petitions may be correlated. Recently, PTAB judges have begun to discourage multiple petitions, for example by asking challengers to rank their petitions or rejecting duplicative petitions [100]. Notably, this is another example of how the problem changes over time in litigation. In any event, to avoid the complexities of redundant petitions, we eschew cross-validation and instead evaluate classification methods on a time-separated data set. The balance between grant and denial classes, however, also changes over time. Figure 2: Schematic of CNN-Attention architecture used in this paper. The input is a sequence of \(L\) integer tokens, which is processed to a matrix of token embedding vectors with trainable embedding weights. The embeddings are fed into CNNs which vary by kernel width as shown in the diagram. The output is fed into a feed forward attention layer, which is shown in the inset at the bottom, based on the structure described in [81]. The attention at that layer will have a dimensionality equal to the sequence length minus the kernel width, allowing values across the sequence to be visualized. The outputs of the attention layer, which will have a dimensionality equal to the number of convolutional filters, are then concatenated to form an, in this case, 11 times 256 or 2816-dimensional output, which is in turn fed to a densely connected layer with a 256-dimensional output, and then a final densely connected layer with sigmoid activation function to make a binary classification. Figure 3 shows how the total number of institution decisions, and the balance between grant and denial, have changed over time. Notably, the low total number for 2012 is due to petitions not filed until the AIA post-grant review system went into effect in September. The low number for 2021 reflects the typical six month lag between petitions and institution decisions, since only petitions through June 30 will have been guaranteed to have an institution decision by data collection cutoff. Over this time, the grant/denial balance has fluctuated. For example, 60.5% of petitions in the data set that were filed in 2018 were granted, a fraction which fell to 58.7% in 2019, and rose back to 61.1% in 2020. This is unsurprising, as variations in the grant/denial ratio may be due to changes in how PTAB regards duplicative petitions, as well as changes in the quality of patents being asserted in litigation and parties' refinement of litigation strategies. Consequently, we use time-separated multi-year data sets for both training and testing.