## 2 Background and Related Work
### Machine Learning and Natural Language Processing in Legal Applications
While they looked at patent text rather than litigation documents, word2 vec embeddings were classified using with random forests (RF) ensemble decision tree methods to use textual features of patents along with other features (such as numbers of references and forward citations) to predict the litigation risk of patents [49]. Similar work has also been done with other features and traits of patents following [PERSON]'s groundbreaking work [17; 50]. TF-IDF with an AdaBoost decision-tree based classifier has been used to predict motion outcome based on judicial complaints. In that work, the authors also vectorized document text using word2 vec and doc2 vec pretrained embeddings along with pre-selected features like attorney name, and coded case types were used to train classifiers that achieved an accuracy of over 60% on a 15 year data set of motions to strike complaints in Connecticut state civil tort and vehicular cases [51]. Convolutional neural networks (CNN) have been used with Bag of Words (BoW) vector representations for verdict prediction for cases in courts in India [52]. In this paper, we apply eXtreme Gradient Boosting (XGBoost) [53], which is a boosted decision tree-based ensemble machine learning method, to classify documents using TF-IDF vectors. Along those lines, researchers in China found that XGBoost outperformed methods including KNN, SVM, and Naive Bayes Classifiers for the prediction of theft crimes based on textual information [54]. Another recent study used XGBoost for prediction tasks on a database of consumer lawsuits in Brazil [55]. Shifting focus to PTAB proceedings in particular, researchers have used PTAB decisions as an empirical testbed for legal NLP applications; for example, developing document retrieval methods by extracting patents relevant to a PTAB determination [56]. Another work used pre-selected features from post-grant reviews to develop a model for patent quality [57]. The first work seeking to predict institution decision outcomes for PTAB proceedings employed Random Forests and SVM on TF-IDF vectorizations of patent claim text [58]. Although they reported greater than 50% accuracy, their study did not distinguish between cases where parties joined ongoing proceedings or settled proceedings by terminating them from petition grant and denial outcomes [58]. Subsequently, in 2018 another group also used SVM but on three different kinds of variables [59]. The researchers considered a \"text-based approach\", \"entity network\", and \"domain knowledge\". For the first, they computed paragraph-level attention using a CNN and hierarchical attention framework derived by [60] (further discussed below). The second, entity network, was an embedded graph using Node2 Vec that included information on the patent owner, petitioner, and judges. The final domain knowledge category included characteristics of the patent, such as the number of references and forward citations, and the number of claims, and IPR proceedings, such as the number of words in the petition, and the institution rate of judges. They found an accuracy of 74% and 0.83 AUC for balanced data (they had a 67% institution rate in their dataset), noting that the entity network provided the greatest contribution to accurate prediction, while the weakest was domain knowledge. They also found that when testing temporally out-of-domain cases filed later in time than the training cases, the best results were for earlier cases in the test set. While the researchers carefully considered their selection of features, some of the results suggest potential overfitting. For example, one significant word the model identified was \"DJC\", which lacks general meaning. Furthermore, even temporally separated test cases and training cases may be related, which makes it a challenging area to establish generalization. Moreover, some features are duplicative. For example the identity of judges reflects the subject matter of patents, since PTAB judges are specialized to particular subject matter categories [20].