## 5 Discussion
Changes in governing law can be significant; for example, as shown in Figure 3, in 2018 the law changed such that a \"mixed\" outcome was no longer possible. As discussed in Section 4.2, for example, in 2019, PTAB began denying institution on new grounds of redundancy with trial court proceedings on the same patents. PTAB also began denying institutions on more parallel proceedings brought on the same patent than they did before [26]. This basis for denial proved critical in the success of response briefs after this change, as shown in Figure 9, in which key terms relating to arguments based on redundant proceedings are highlighted. Accordingly, predictive methods in litigation must be dynamic to account for changes in jurisprudence as well as governing law. Second, training data sets are still small, which means that there is significant risk of overfitting and inability to generalize. The limited size of training data sets is inherent to the problem domain: only so many PTAB cases have been filed since the proceedings began in 2018. Future directions for addressing the limitation of training data set sizes may include looking to trial court cases in patent litigation where patent validity issues like those brought before PTAB are at issue. The shortcoming of this approach is that some issues, such as the aforementioned \"redundancy argument\", do not apply in district court. Third, repeat litigants may skew results. The pre-processing method described in Section 3.2 endeavors to exclude entity-identifying information, such as proper nouns and the email addresses of attorneys. However, some entities are responsible for a large amount of patent litigation and correspondingly large numbers of PTAB proceedings. Among these are patent assertion entities (PAEs), perjoratively called \"patent trolls\" who sue many defendants, often over a long period of time and over different patents covering sometimes related or sometimes divergent subject matter [18]. Additionally, repeat litigants may be advancing different quality levels of patents. Some PAEs may be suing defendants on weak patents, seeking settlements--but others may be selective about asserting only strong patents to ensure durable returns [102]. In this study, we confirmed that the training and test sets did not overlap in terms of parties; however, the presence of repeat litigants can bias training or skew test results. In particular, because training data sets are not large (the total data set only has over 11,000 examples at this point), repeat litigants risk systematically skewing results if they, for example, have similar patterns in their briefs. Future work is necessary to detect consistent patterns shown by repeat litigants, which may be helpful in removing bias from training sets or even learning patterns consistently associated with success or failure by such litigants. For example, deep learning may be able to identify a \"signature\" associated with PAE's litigation strategy or brief writing. Another challenge that all real-world NLP must address is significant noise in the documents. The data set that we use is of PDF documents, which are not necessarily readily converted to clean text files. As a result, certain words can be distorted: in one example, the word \"the\" was replaced by \"ttthhheee\". Noisy words cause a problem for a classifier, because meaningful words are lost, and the training process can be skewed by overfitting to them. However, such noise is inherent to realistic litigation classifiers, at least in the United States context. In the United States, litigation documents are stored in PDF format and there is no requirement that they be machine readable noise-free. As of early 2022, court decisions and efforts in the U.S. Congress are at work are seeking to make all court documents freely accessible to the public (currently they are available only for a charge)--but these will all be in PDF format.