## 5 Discussion  
This paper's goal is to demonstrate a proof-of-concept of machine learning techniques applied to a realistic and important problem in litigation analysis, post-grant patent review cases. The results demonstrate that applying XGBoost classification to a TF-IDF document representation can successfully predict whether a written decision represents a grant (success for the patent challenger) or denial (success for the patent owner) in the first phase of the patent review challenge. Then, XGBoost scores or SHAP scores calculated using the XGBoost model can rank terms based on their significance to the classification. The results further show that an interpretable deep neural network learning method based on multiple CNN and attention (using an architecture shown in Figure 2) can be successful as well. On the written decision classification problem, CNN performs less well than XGBoost/TF-IDF (e.g., 82% versus 90% accuracy, respectively), largely  
Figure 9: Excerpts of response brief predictions highlighting attention for a kernel size of 8. Green-labeled words are locations where the attention vectors have values greater the median for the sample (excluding the attention at zero tokens where the text was padded), with more intense green corresponding to higher levels of attention. Pink-labeled words are locations where attention is below the sample of the median, and more intense pink corresponds to lower attention at that location in the text. The excerpts are labeled with the case (proceeding) number and whether the result was presented in the fails to disclose or suggest key limitations of the challenged claims for thesedue to underperformance on denial decisions. The predictions of the CNN model are also interpretable using the LIME method, as demonstrated by validating the key terms identified by LIME. The results also validate the use of the attention layer of the CNN model which can also be visualized for a document by highlighting key terms within their context in the document. Predicting case outcome from the patent owner's preliminary response brief proves to be much more challenging. This is not unexpected. The written decision classification is not trivial--the documents have no formal structure and there is no consistent way in which judges state the decision--but the document itself discloses the outcome. The response brief will be much more variable because the patent owner must react to the issues the patent challenger brings up, and can make different strategic decisions about what arguments to pursue. XGBoost/TF-IDF was unable to predict outcomes based on the response brief. However, prediction accuracy of the CNN model proved better than chance (approximately 57%) for both the majority and minority classes in our problem domain (granted and denial, respectively). This paper thus establishes at least a preliminary proof-of-concept that complex deep network models can predict success based on the text of a litigation brief. Moreover, the results demonstrate that the attention layer visualization of the deep learning model is able to provide a highlighting of key terms and arguments that are important for determining whether the brief led to a grant or denial. The inherently interpretable deep learning architecture introduced in this paper has an important efficiency advantage over post-hoc analysis methods such as LIME or other sensitivity analysis methods. For LIME or sensitivity analysis, the model must be evaluated multiple times to generate perturbations [93], whereas attention is generated by evaluation the model just once. There are important caveats to our analysis. Three major caveats are summarized in turn below. First, as shown in Figure 3, there are changes in grant/denial propensity over time. However, while those fluctuations are not necessarily large, there are significant differences in how PTAB adjudicates cases over time.