# Evaluation of Diversification Techniques for Legal Information Retrieval +
## 2 Legal Document Ranking Using Diversification
### Results
As \(\lambda\) increases, preference to diversity increases for all methods, except MMR. Subtopic-recall accuracy of all methods,  
Figure 3: Normalized expected reciprocal rank-intent aware (nERR-IA) at various levels @5, @10, @20, @30 for baseline and the MMR, Max-sum, Max-min, Mono-objective, LexRank, Biased LexRank, DivRank and Grasshopper methods. (**a**) nERR-IA@5; (**b**) nERR-IA@10; (**c**) nERR-IA@20; (**d**) nERR-IA@30. (Best viewed in color.) except MMR, increases when increasing \(\lambda\). For lower levels (e.g., @5, @10), MMR clearly outperforms other methods, while for upper levels (e.g., @20, @30), MMR and Max-min scores are comparable. We also observe that Max-min tends to perform better than Max-sum, which in turn constantly achieves better results than Mono-objective. Finally, LexRank, Biased LexRank and Grasshopper approaches fail to improve the baseline ranking at all levels across all metrics. Overall, we noticed a similar trending behavior with the ones discussed for Figures 2 and 3. In summary, among all of the results, we note that the trends in the graphs look very similar. Clearly enough, the utilized web search diversification methods (MMR, Max-sum, Max-min, Mono-objective) and DivRank statistically significantly, using the paired two-sided \(t\)-test, outperform the baseline method, offering legislation stakeholders broader insight with respect to their information needs. Furthermore, trends across the evaluation metric graphs highlight balance boundaries for legal IR systems between reinforcing relevant documents or sampling the information space around the legal query. Table 5 summarizes the average results of the diversification methods. Statistically-significant values, using the paired two-sided \(t\)-test with \(p_{value}<0.05\) are denoted with \({}^{\circ}\) and with \(p_{value}<0.01\) with \({}^{*}\). The effectiveness of diversification methods is also depicted in Table 6, which illustrates the result sets for three example queries, using our case law dataset (\(|S|=30\) and \(|N|=100\)) with \(\lambda=0\) (no diversification), \(\lambda=0.1\) (light diversification), \(\lambda=0.5\) (moderate diversification) and \(\lambda=0.9\) (high diversification). Only MMR results are shown since, in almost all variations, it outperforms other approaches. Due to space limitations, we show the case title for each entry. The hyperlink for the full text for each entry can be formed using the format ([[http://www.austli.edu.au/au/cases/cth/FCA/](http://www.austli.edu.au/au/cases/cth/FCA/)]([http://www.austli.edu.au/au/cases/cth/FCA/](http://www.austli.edu.au/au/cases/cth/FCA/)) [year]/[docNum].html) and substituting the given values in the Table 6. For example, for the first entry of the Table 6 the corresponding hyperlink, substituting values 2008 and 1503 for \(year\) and \(docNum\) is ([[http://www.austli.edu.au/cases/cth/FCA/2008/1503.html](http://www.austli.edu.au/cases/cth/FCA/2008/1503.html)]([http://www.austli.edu.au/cases/cth/FCA/2008/1503.html](http://www.austli.edu.au/cases/cth/FCA/2008/1503.html))). Figure 4: Subtopic recall (S-recall) at various levels @5, @10, @20, @30 for the baseline and the MMR, Max-sum, Max-min, Mono-objective, LexRank, Biased LexRank, DivRank and Grasshopper methods. (a) S-recall@5; (b) S-recall@10; (c) S-recall@20; (d) S-recall@30. (Best viewed in color.)