## 2 Background and Related Work
### Deep Learning and Interpretability: Application to Litigation Analysis and Prediction  
Recent advances in hardware and neural network architecture have enabled a movement away from TF-IDF and other document vectorization methods to deep learning: end-to-end machine learning to obtain document models and classifiers, in which the text itself is the input to the model [47]. Deep learning can be both powerful and flexible, as models can learn contextual semantic relationships between words and phrases on their own. However, because deep learning approaches generally rely on neural network black box models, due to their multilayer nonlinear structure, they can predict an outcome but are difficult to interpret and understand [61]. By way of example of deep learning applications to litigation, in 2018, a group studying criminal cases in China developed a model that used long-short term memory (LSTM) recurrent neural network (RNN) model with an attention layer to do simultaneous criminal charge and attribute prediction based on analyzing text description of facts [12]. Further work by researchers working in the Chinese legal context found that extracting semantic features was superior to identifying features using SVM, and that increased model complexity was necessary to account for the nonstandard and out-of-vocabulary nature of legal terminology [62; 63]. Deep learning methods also include transfer learning approaches in which word embeddings are pretrained on a broader corpus and then applied or fine-tuned in the classification. A popular neural network architecture for self-supervised training of word embeddings is BERT (bidirectional encoder representations from transformers), which Google developed to handle natural language search [64]. Recent work by [PERSON] et al. involved pre-training on a legal domain-specific LEGAL-BERT, which was found to outperform BERT in legal text classification tasks [65]. Another work used domain-specific pre-training with self-supervised learning to learn from the context of a case citation the case holding [66]. Another group recently combined Legal-BERT with TF-IDF to do legal statute retrieval [67]. One of the critical problems for applying AI to law is explainability or interpretability. Explainable AI/interpretable machine learning is a growing field of research, as it has become clear that for prediction models to be applied in practice, practitioners and consumers need to understand the basis for predictions. Interpretable models are also critical to gain theoretical insight from the classification of empirical data [19; 68]. Interpretable machine learning is particularly important for legal practitioners since law is based on reasoning, and so it is essential to understand the basis for why decisions are made [69]. Interpretable models can also reveal ways in which the law is biased. More generally, for example, analysis of word embeddings in a large document corpus showed that embeddings learned a semantic correlation between vocabulary and gender that showed employment bias--for example, the term \"computer program\" was associated with men in [70]. Interpretability is not as much an issue for the classical machine learning and decision tree-based models described above. For example, TF-IDF can be viewed as a metric for relevance in addition to being a feature [71]. One group has proposed visualizing TF-IDF values of terms via word clouds to determine the basis for patent invalidity decisions at the Federal Circuit [72]. TF-IDF combined with SVM classification has also extracted the legal and factual basis of construction law-related verdicts [73]. XGBoost can be analyzed to identify feature importance through various scores that are based on characteristics of boosted tree models, such as the number of times a feature is used to split trees, or the gain in score towards the objective function obtained by splitting trees based on a feature [74; 75].