On the Effective Communication of the Results of Empirical Studies, Part II

60 Vand. L. Rev. 801 (2007) While law professors are increasingly making use of data in their scholarship and while the data work housed in their studies is (generally) of a high quality, they have been less effective at communicating the products of their labor. A strong devotion to tabular, rather than graphical, displays, and claims about "statistical significance" rather than substantive importance, are just two areas requiring improvement. Here, as in Part I, we attempt to adapt a burgeoning literature in the social and statistical sciences to the unique interests of legal scholars. Our proposals are many in number, but none is particularly difficult to implement. More to the point, we believe that law professors should want to implement them. If other fields are any indication, moving toward more appropriate and accessible presentations of data will heighten the impact of empirical legal scholarship regardless of the audience-no doubt a desirable goal in a discipline that rightfully prides itself on its contributions to forming legal and public policy. I. INTRODUCTION To assess the controversial claim that affirmative action in U.S. law schools causes blacks to fail the bar exam, Daniel E. Ho deploys an innovative approach.1 Professor Ho "matches" students on all relevant observable variables,2 except the key causal variable-the tier of their law school-and then compares bar passage rates. Figure 1, reprinted from Ho's article, displays the results. Ho's work has received no shortage of kudos,4 but surely another is in order: the author knows how to communicate research results. From Figure 1, readers can easily grasp the study's key takeaway; namely, claims about the repercussions of affirmation action on bar passage rates are overblown. Similarly qualified black students, regardless of the tier of their law school, perform at the same (i.e., statistically indistinguishable) level. Why Ho's graphic display is so powerful and, indeed, why it may help explain the impact of his article, is no mystery. First, while assessing the effect of school tier on bar performance required complex calculations, Ho deemphasizes them;5 he instead focuses on communicating substance, not statistics. No one can look at Ho's figure and fail to see that all but one of the black circles and lines fall near zero (indicating no causal effect). second, not only does the author well illustrate the substantive effect of school tier on bar passage rates, he also effectively conveys his uncertainty about that (non)effect. Because the dark horizontal lines (indicating 95% confidence intervals) intersect zero for all black students, we can safely conclude that the impact of tier is statistically indistinct from zero. A presentation depicting only the results and not Ho's uncertainty about them may well have led causal readers astray, especially about black students in the lowest tiers. Finally, we applaud Ho's use of a figure to convey his findings. Had he employed a tabular display, as do many scholars publishing in the law reviews, he would have missed an opportunity to present his results in the most accessible and powerful way possible. In short, in conveying the findings of his important study, Ho followed the three key principles of effective communication: 1. Communicate Substance, not Statistics 2. When Performing Inference, Convey Uncertainty 3. Graph Data and Results Our earlier article, On the Effective Communication of the Results of Empirical Studies, Part I (hereinafter Communication I),6 explores these principles in some detail. It also offers some general rules for creating visually effective displays of data.7 In other disciplines, adherence to these principles has generated benefits for the producers and consumers of empirical research, and we have no doubt that Law will see similarly salutary effects. Most crucially, as we explained in Communication I, moving towards more appropriate and accessible data presentations will enhance the impact of empirical legal scholarship-regardless of whether the intended audience consists of other scholars, students, policy makers, judges, or practicing attorneys. …