# Evaluation of Diversification Techniques for Legal Information Retrieval +
## 2 Legal Document Ranking Using Diversification
### Results  
As a baseline to compare diversification methods, we consider the simple ranking produced by cosine similarity and log-based \(tf-idf\) indexing schema. For each query, our initial set \(N\) contains the \(top-n\) query results. The interpolation parameter \(\lambda\in[0..1]\) is tuned in 0.1 steps separately for each method. We present the evaluation results for the methods employed, using the aforementioned evaluation metrics, at cut-off values of 5, 10, 20 and 30, as typical in TREC evaluations. Results are presented with fixed parameter n = \(|N|\). Note that each of the diversification variations is applied in combination with each of the diversification algorithms and for each user query. Figure 2 shows the \(a\)-normalized discounted cumulative gain (a-nDCG) of each method for different values of \(\lambda\). Interestingly, web search result diversification methods (MMR, Max-sum, Max-min and Mono-objective) outperformed the baseline ranking, while text summarization methods (LexRank, Biased LexRank and Grasshopper, as it was utilized without a network citation graph) failed to improve the baseline ranking performing lower than the baseline ranking at all levels across all metrics. Graph-based methods' (DivRank) results vary across the different values of \(\lambda\). We attribute this finding to the extreme sparse network of citations since our dataset covers a short time period (three years). The trending behavior of MMR, Max-min and Max-sum is very similar, especially at levels @10, and @20, while at level @5, Max-min and Max-sum presented nearly identical a-nDCG values in many  
Figure 2: \(a\)-Normalized discounted cumulative gain (a-nDCG) at various levels @5, @10, @20, @30 for baseline, MMR, Max-sum, Max-min, Mono-objective, LexRank, Biased LexRank, DivRank and Grasshopper methods. (**a**) alpha-nDCG@5; (**b**) alpha-nDCG@10; (**c**) alpha-nDCG@20; (**d**) alpha-nDCG@30. (Best viewed in color.) \(\lambda\) values (e.g., 0.1, 0.2, 0.4, 0.6, 0.7). Finally, MMR constantly achieves better results with respect to the rest of the methods, followed by Max-min and Max-sum. Mono-objective, despite the fact that it performs better than the baseline in all \(\lambda\) values, still always presents a lower performance when compared to MMR, Max-min and Max-sum. It is clear that web search result diversification approaches (MMR, Max-sum, Max-min and Mono-objective) tend to perform better than the selected baseline ranking method. Moreover, as \(\lambda\) increases, preference for diversity, as well as a-nDCG accuracy increase for all tested methods. Figure 3 depicts the normalized expected reciprocal rank-intent aware (nERR-IA) plots for each method with respect to different values of \(\lambda\). It is clear that web search result diversification approaches (MMR, Max-sum, Max-min and Mono-objective) tend to perform better than the selected baseline ranking method. Moreover, as \(\lambda\) increases, preference for diversity, as well as nERR-IA accuracy increase for all tested methods. Text summarization methods (LexRank, Biased LexRank) and Grasshopper once again failed to improve the baseline ranking at all levels across all metrics, while as in a-nDCG plots, DivRank results vary across the different values of \(\lambda\). MMR constantly achieves better results with respect to the rest of the methods. We also observed that Max-min tends to perform better than Max-sum. There were a few cases where both methods presented nearly similar performance, especially in lower recall levels (e.g., for nERR-IA@5 when \(\lambda\) equals 0.1, 0.4, 0.6, 0.7). Once again, Mono-objective presents a lower performance when compared to MMR, Max-min and Max-sum for the nERR-IA metric for all \(\lambda\) values applied. Figure 4 shows the subtopic-recall (S-recall) at various levels @5, @10, @20, @30 of each method for different values of \(\lambda\). It is clear that the web search result diversification methods (MMR, Max-sum, Max-min and Mono-objective) tend to perform better than the baseline ranking.