# Evaluation of Diversification Techniques for Legal Information Retrieval +
## 2 Legal Document Ranking Using Diversification
### Relevance Judgments  
Evaluation of diversification requires a data corpus, a set of query topics and a set of relevance judgments, preferably assessed by domain experts for each query. One of the difficulties in evaluating methods designed to introduce diversity in the legal document ranking process is the lack of standard testing data. While TREC added a diversity task to the web track in 2009, this dataset was designed assuming a general web search, and so, it is not possible to adapt it to our setting. Having only the document corpus, we need to define: (a) the query topics; (b) a method to derive the subtopics for each topic; and (c) a method to annotate the corpus for each topic. In the absence of a standard dataset specifically tailored for this purpose, we looked for an objective way to evaluate and assess the performances of various diversification methods on our corpus. We do acknowledge the fact that the process of automatic query generation is at best an imperfect approximation of what a real person would do. To this end, we have employed the following way to annotate our corpus with relevance judgments for each query:  
User profiles/queries: The West American Digest System is a taxonomy of identifying points of law from reported cases and organizing them by topic and key number. It is used to organize the entire body of American law. We used the West Law Digest Topics as candidate user queries. In other words, each topic was issued as a candidate query to our retrieval system. Outlier queries, whether too specific/rare or too general, were removed using the interquartile range, below or above values \(Q1\) and \(Q3\), sequentially in terms of the number of hits in the result set and the score distribution for the hits, \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Range** \\ \hline algorithms tested & MMR, Max-min, Max-sum, Mono-objective, \\  & LexRank, Biased LexRank, DivRank, Grasshopper \\ tradeoff \(l\) values & 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 \\ candidate set size n = \(|N|\) & 100 \\ result set size k = \(|S|\) & 5, 10, 20, 30 \\ \# of sample queries & 298 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Parameters tested in the experiments. demanding in parallel a minimum coverage of \(min|N|\) results. In total, we kept 289 queries. Table 2 provides a sample of the topics we further consider as user queries. Query assessments and ground-truth: For each topic/query, we kept the \(top-n\) results. An LDA [18] topic model, using an open source implementation ([[http://mallet.cs.umass.edu/](http://mallet.cs.umass.edu/)]([http://mallet.cs.umass.edu/](http://mallet.cs.umass.edu/))), was trained on the \(top-n\) results for each query. Topic modeling gives us a way to infer the latent structure behind a collection of documents since each trained topic is a set of keywords with corresponding weights. Table 3 provides a sample of the top keywords for each topic based on the result set for User Query 1: Abandoned and Lost Property. Based on the resulting topic distribution, with an acceptance threshold of 20%, we access a document as relevant for an topic/aspect. The chosen threshold value, 20%, in the outcome of the LDA method to annotate the corpus for each topic affects the density of our topic/aspect distribution and not the generalizability of the proposed approach or the evaluation outcome. We performed experiments with other settings, i.e., 10% and 30% achieving comparable results. We do acknowledge the fact that if we had chosen a threshold based on interquartile range, as we did in the \"user profiles/queries\" paragraph, the aforementioned threshold would sound less ad hoc. Nevertheless, this would come at the expense of increasing the complexity both of the paper and the system code. Table 4 provides a sample of the topic distribution for User Query 1: Abandoned and Lost Property.