Source: https://e-discoveryteam.com/tag/costs/
Timestamp: 2019-04-19 22:49:57+00:00

Document:
This is a continuation of a blog that I started last week. Suggest you read Part One before this.
Here is the workflow for the simplified six-step plan. The first three steps repeat until you have a viable plan where the costs estimate is proportional under Rule 26(b)(1).
The document review begins with Multimodal Search of the ESI. Multimodal means that all modes of search are used to try to find relevant documents. Multimodal search uses a variety of techniques in an evolving, iterated process. It is never limited to a single search technique, such as keyword. All methods are used as deemed appropriate based upon the data to be reviewed and the software tools available. The basic types of search are shown in the search pyramid.
In Step One we use a multimodal approach, but we typically begin with keyword and concept searches. Also, in most projects we will run similarity searches of all kinds to make the review more complete and broaden the reach of the keyword and concept searches. Sometimes we may even use a linear search, expert manual review at the base of the search pyramid. For instance, it might be helpful to see all communications that a key witness had on a certain day. The two-word stand-alone call me email when seen in context can sometimes be invaluable to proving your case.
I do not want to go into too much detail of the types of searches we do in this first step because each vendor’s document review software has different types of searches built it. Still, the basic types of search shown in the pyramid can be found in most software, although AI, active machine learning on top, is still only found in the best.
Multimodal search, wherein a variety of techniques are used in an evolving, iterated process, is new to the legal profession, but not to Information Science. That is the field of scientific study which is, among many other things, concerned with computer search of large volumes of data. Although the e-Discovery Team’s promotion of multimodal search techniques to find evidence only goes back about ten years, Multimodal is a well-established search technique in Information Science. The pioneer professor who first popularized this search method was Marcia J. Bates, and her article, The Design of Browsing and Berrypicking Techniques for the Online Search Interface, 13 Online Info. Rev. 407, 409–11, 414, 418, 421–22 (1989). Professor Bates of UCLA did not use the term multimodal, that is my own small innovation, instead she coined the word “berrypicking” to describe the use of all types of search to find relevant texts. I prefer the term “multimodal” to “berrypicking,” but they are basically the same techniques.
An important thing we learned early on is that successful searching requires what I called “berrypicking.” . . .
Berrypicking involves 1) searching many different places/sources, 2) using different search techniques in different places, and 3) changing your search goal as you go along and learn things along the way. . . .
This may seem fairly obvious when stated this way, but, in fact, many searchers erroneously think they will find everything they want in just one place, and second, many information systems have been designed to permit only one kind of searching, and inhibit the searcher from using the more effective berrypicking technique.
In real-life searches in manual sources, end users may begin with just one feature of a broader topic, or just one relevant reference, and move through a variety of sources. Each new piece of information they encounter gives them new ideas and directions to follow and, consequently, a new conception of the query. At each stage they are not just modifying the search terms used in order to get a better match for a single query. Rather the query itself (as well as the search terms used) is continually shifting, in part or whole. This type of search is here called an evolving search.
Furthermore, at each stage, with each different conception of the query, the user may identify useful information and references. In other words, the query is satisfied not by a single final retrieved set, but by a series of selections of individual references and bits of information at each stage of the ever-modifying search. A bit-at-a-time retrieval of this sort is here called berrypicking. This term is used by analogy to picking huckleberries or blueberries in the forest. The berries are scattered on the bushes; they do not come in bunches. One must pick them one at a time. One could do berrypicking of information without the search need itself changing (evolving), but in this article the attention is given to searches that combine both of these features.
I independently noticed evolving search as a routine phenomena in legal search and only recently found Professor Bates’ prior descriptions. I have written about this often in the field of legal search (although never previously crediting Professor Bates) under the names “concept drift” or “evolving relevance.” See Eg. Concept Drift and Consistency: Two Keys To Document Review Quality – Part Two (e-Discovery Team, 1/24/16). Also see Voorhees, Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness, 36 Info. Processing & Mgmt 697 (2000) at page 714.
SIDE NOTE: The somewhat related term query drift in information science refers to a different phenomena in machine learning. In query drift the concept of document relevance unintentionally changes from the use of indiscriminate pseudorelevance feedback. Cormack, Buttcher & Clarke, Information Retrieval Implementation and Evaluation of Search Engines (MIT Press 2010) at pg. 277. This can lead to severe negative relevance feedback loops where the AI is trained incorrectly. Not good. If that happens a lot of other bad things can and usually do happen. It must be avoided.
Yes. That means that skilled humans must still play a key role in all aspects of the delivery and production of goods and services, lawyers too.
UCLA Berkeley Professor Bates first wrote about concept shift when using early computer assisted search in the late 1980s. She found that users might execute a query, skim some of the resulting documents, and then learn things which slightly changes their information need. They then refine their query, not only in order to better express their information need, but also because the information need itself has now changed. This was a new concept at the time because under the Classical Model Of Information Retrieval an information need is single and unchanging. Professor Bates illustrated the old Classical Model with the following diagram.
The Classical Model was misguided. All search projects, including the legal search for evidence, are an evolving process where the understanding of the information need progresses, improves, as the information is reviewed. See diagram below for the multimodal berrypicking type approach. Note the importance of human thinking to this approach.
Bates argues that searches are evolving and occur bit by bit. That is to say, a person constantly changes his or her search terms in response to the results returned from the information retrieval system. Thus, a simple linear model does not capture the nature of information retrieval because the very act of searching causes feedback which causes the user to modify his or her cognitive model of the information being searched for.
Multimodal search assumes that the information need evolves over the course of a document review. It is never just run one search and then review all of the documents found in the search. That linear approach was used in version 1.0 of predictive coding, and is still used by most lawyers today. The dominant model in law today is linear, wherein a negotiated list of keyword is used to run one search. I called this failed method “Go Fish” and a few judges, like Judge Peck, picked up on that name. Losey, R., Adventures in Electronic Discovery (West 2011); Child’s Game of ‘Go Fish’ is a Poor Model for e-Discovery Search; Moore v. Publicis Groupe & MSL Group, 287 F.R.D. 182, 190-91, 2012 WL 607412, at *10 (S.D.N.Y. Feb. 24, 2012) (J. Peck).
The popular, but ineffective Go Fish approach is like the Classical Information Retrieval Model in that only a single list of keywords is used as the query. The keywords are not refined over time as the documents are reviewed. This is a mono-modal process. It is contradicted by our evolving multimodal process, Step One in our Six-Step plan. In the first step we run many, many searches and review some of the results of each search, some of the documents, and then change the searches accordingly.
Each search run is sampled by quick reviews and its effectiveness evaluated, tested. For instance, did a search of what you expected would be an unusual word turn up far more hits than anticipated? Did the keyword show up in all kinds of documents that had nothing to do with the case? For example, a couple of minutes of review might show that what you thought would be a carefully and rarely used word, Privileged, was in fact part of the standard signature line of one custodian. All his emails had the keyword Privileged on them. The keyword in these circumstances may be a surprise failure, at least as to that one custodian. These kind of unexpected language usages and surprise failures are commonplace, especially with neophyte lawyers.
Sampling here does not mean random sampling, but rather judgmental sampling, just picking a few representative hit documents and reviewing them. Were a fair number of berries found in that new search bush, or not? In our example, assume that your sample review of the documents with “Privileged” showed that the word was only part of one person’s standard signature on every one of their emails. When a new search is run wherein this custodian is excluded, the search results may now test favorably. You may devise other searches that exclude or limit the keyword “Privileged” whenever it is found in a signature.
There are many computer search tools used in a multimodal search method, but the most important tool of all is not algorithmic, but human. The most important search tool is the human ability to think the whole time you are looking for tasty berries. (The all important “T” in Professor Bates’ diagram above.) This means the ability to improvise, to spontaneously respond and react to unexpected circumstances. This mean ad hoc searches that change with time and experience. It is not a linear, set it and forget it, keyword cull-in and read all documents approach. This was true in the early days of automated search with Professor Bates berrypicking work in the late 1980s, and is still true today. Indeed, since the complexity of ESI has expanded a million times since then, our thinking, improvisation and teamwork are now more important than ever.
The goal in Step Two is to identify effective searches. Typically, that means where most of the results are relevant, greater than 50%. Ideally we would like to see roughly 80% relevancy. Alternatively, search hits that are very few in number, and thus inexpensive to review them all, may be accepted. For instance, you may try a search that only has ten documents, which you could review in just a minute. You may just find one relevant, but it could be important. The acceptable range of number of documents to review in Bottom Line Driven Review will always take cost into consideration. That is where Step-Three comes in, Estimation. What will it costs to review the documents found?
It is not enough to come up with effective searches, which is the goal of Steps One and Two, the costs involved to review all of the documents returned with these searches must also be considered. It may still cost way too much to review the documents when considering the proportionality factors under 26(b)(1) as discussed in Part One of this article. The plan of review must always take the cost of review into consideration.
In Part One we described an estimation method that I like to use to calculate the cost of an ESI review. When the projected cost, the estimate, is proportional in your judgment (and, where appropriate, in the judge’s judgment), then you conclude your iterative process of refining searches. You can then move onto the next Step-Four of preparing your discovery plan and making disclosures of that plan.
Still, although early agreement on scope of review is often impossible, as the requesting party always wants you to spend more, you can usually move past this initial disagreement by agreeing to phased discovery. The requesting party can reserve its objections to your plan, but still agree it is adequate for phase one. Usually we find that after that phase one production is completed the requesting party’s demands for more are either eliminated or considerably tempered. It may well now to possible to reach a reasonable final agreement.
Here is where you start to carry out your discovery plan. In this stage you finish looking at the documents and coding them for Responsiveness (relevant), Irrelevant (not responsive), Privileged (relevant but privileged, and so logged and withheld) and Confidential (all levels, from just notations and legends, to redactions, to withhold and log. A fifth temporary document code is used for communication purposes throughout a project: Undetermined. Issue tagging is usually a waste of time and should be avoided. Instead, you should rely on search to find documents to support various points. There are typically only a dozen or so documents of importance at trial anyway, no matter what the original corpus size.
I highly recommend use of professional document review attorneys to assist you in this step. The so-called “contract lawyers” specialize in electronic document review and do so at a very low cost, typically in the neighborhood of $50 per hour. The best of them, who may often command slightly higher rates, are speed readers with high comprehension. They also know what to look for in different kinds of cases. Some have impressive backgrounds. Of course, good management of these resources is required. They should have their own management and team leaders. Outside attorneys signing Rule 26(g) will also need to supervise them carefully, especially as to relevance intricacies. The day will come when a court will find it unreasonable not to employ these attorneys in a document review. The savings is dramatic and this in turn increases the persuasiveness of your cost burden argument.
The last step is transfer of the appropriate information to the requesting party and designated members of your team. Production is typically followed by later delivery of a Log of all documents withheld, even though responsive or relevant. The withheld logged documents are typically: Attorney-Client Communications protected from disclosure under the client’s privilege; or, Attorney Work-Product documents protected from disclosure under the attorney’s privilege. Two different privileges. The attorney’s work-product privilege is frequently waived in some part, although often very small. The client’s communications with its attorneys is, however, an inviolate privilege that is never waived.
Typically you should produce in stages and not wait until project completion. The only exception might be where the requesting party would rather wait and receive one big production instead of a series of small productions. That is very rare. So plan on multiple productions. We suggest the first production be small and serve as a test of the receiving party’s abilities and otherwise get the bugs out of the system.
In this essay I have shown the method I use in document reviews to control costs by use of estimation and multimodal search. I call this a Bottom Line Driven approach. The six step process is designed to help uncover the costs of review as part of the review itself. This kind of experienced based estimate is an ideal way to meet the evidentiary burdens of a proportionality objection under revised Rules 26(b)(1) and 32(b)(2). It provides the hard facts needed to be specific as to what you will review and what you will not and the likely costs involved.
The six-step approach described here uses the costs incurred at the front end of the project to predict the total expense. The costs are controlled by use of best practices, such as contract review lawyers, but primarily by limiting the number of documents reviewed. Although it is somewhat easier to follow this approach using predictive coding and document ranking, it can still be done without that search feature. You can try this approach using any review software. It works well in small or medium sized projects with fairly simple issues. For large complex projects we still recommend using the eight-step predictive coding approach as taught in the TarCourse.com.
U.S. Magistrate Judge G. Michael Harvey is the court successor to the position held by e-discovery famous Magistrate Judge John Facciola (shown here). Judge Harvey’s latest Order on e-Discovery is his best so far. Oxbow Carbon & Minerals LLC v. Union Pacific Railroad Company, No. 11-cv-1049 (PLF/GMH), 2017 WL 4011136, (D.D.C. Sept. 11, 2017).
Oxbow contains an especially good analysis and discussion of the six proportionality factors in revised Rule 26(b)(1), FRCP. It also contains useful metrics on cost that establish a new comparative benchmark for simple email discovery. It is a must-read in the growing list of cases on proportionality and document review.
Oxbow is a very large antitrust case brought by five related companies that mine and sell coal and petroleum coke. The suit is against the Union Pacific and BNSF Railway Company. The complaint alleges a conspiracy by the railroads to fix the prices of shipping coal. Over $1.5 Billion in damages are sought, including treble damages under the Sherman Antitrust Act.
Oxbow is a big case by anyone’s standards. For that reason it should not be surprising that the plaintiffs opposition to the discovery request on proportionality grounds was denied. The final projected costs of the disputed discovery was $85,000. Still, the ruling in Oxbow is interesting for at least two reasons.
First, the scholarly opinion by Judge Harvey contains an excellent collection and discussion of case law on proportionality and discovery. You will want to keep this case handy as a research starter on a number of issues, especially if your case is in D.C.C.
Judge Harvey, shown here in a court room sketch as a young DOJ trial attorney, continues the scholarly writing tradition of his predecessor, Judge John Facciola. This has long been one of the top courts in the country. The DOJ is also usually known for its high quality lawyers. We are in good hands with judges like G. Michael Harvey.
Immediately prior to Michael Harvey becoming a magistrate judge, he served for over seven years in the U.S. Attorney’s National Security Section “where he prosecuted high-profile cases involving espionage, terrorism, export violations, alien smuggling, unauthorized disclosure of classified information, threats against high-ranking officials, and cybercrime.” Impressive. His current term as Magistrate Judge will expire on February 12, 2023.
The second reason the e-Discovery Team likes the Oxbow opinion is because it shares some of the usually hidden costs involved in a big document review project. The producing party here used what is currently the most popular search method among non-e-discovery lawyers, one that is antiquated for a project of this size, namely negotiated keyword filtering.
This method hearkens back to the earliest days of electronic document review in the late eighties and early nineties. In most projects today by e-discovery experts, if keyword filtering is used at all to exclude data from review, it is based on tested and refined keywords, not mere negotiated keywords. Typically negotiated keywords are tested and refined on the unfiltered data of one or two custodians before used to filter out the rest.
Careful study of the facts in the opinion allow us to calculate certain metrics on costs and effectiveness, from both the economic and information scientific perspective. We can not only calculate costs, but also the relevance prevalence rates and precision. We can also make some educated guesses on recall. These metrics can assist in quality control. As we will see, the producing plaintiffs’ team actually did quite well on Precision – 65%, but we can only guess on the Recall rate. We can watch that unfold as the case progresses. It is still in its early discovery stages.
The information distilled from Oxbow provides a current benchmark for your own doc review efforts. I will do the analysis for you to facilitate comparisons. That can get tricky. (Even some of the numbers stated in this opinion do not add up.) Comparisons can help you to prove that your own efforts were reasonable, or to challenge the reasonability of the responding party’s efforts.
In Oxbow the parties successfully completed phase one email discovery on the email of nineteen custodians of plaintiffs employees. This Order here concerns a requested twentieth custodian, what I would call the beginning of phase two discovery. That new custodian was the CEO of the plaintiff coal companies, William I. Koch, Oxbow’s founder, CEO, and principle owner.
Yes, Bill Koch is one of the infamous billionaire conservative activists Koch brothers, although he keeps a much lower political profile than his brothers Charles and David Koch. According to news reports the 2017 Oxbow employee Christmas party was held at Mar-a-Lago, where Koch is a longtime member, and President Trump made an appearance at the party. Need I say more?
Plaintiffs argued that the email request from William Koch was one step too far, that Koch had nothing of value or interest to say on the subjects in the antitrust litigation. This position was unsupported by the facts.
Judge Michael Harvey does a good job in Oxbow explaining the governing law and how it has shifted with the 2015 revisions to the Rules.
[U]nder the amended Rule 26, discovery must be relevant and “proportional to the needs of the case.” Fed. R. Civ. P. 26(b)(1). To determine whether a discovery request is proportional, courts weigh the following six factors: “(1) the importance of the issues at stake in this action; (2) the amount in controversy; (3) the parties’ relative access to relevant information; (4) the parties’ resources; (5) the importance of the discovery in resolving the issues; and (6) whether the burden or expense of the proposed discovery outweighs its likely benefit.” Williams v. BASF Catalysts, LLC, Civ. Action No. 11-1754, 2017 WL 3317295, at *4 (D.N.J. Aug. 3, 2017) (citing Fed. R. Civ. P. 26(b)(1)); Arrow Enter. Computing Solutions, Inc. v. BlueAlly, LLC, No. 5:15-CV-37-FL, 2017 WL 876266, at *4 (E.D.N.C. Mar. 3, 2017); FTC v. Staples, Inc., Civ. Action No. 15-2115 (EGS), 2016 WL 4194045, at *2 (D.D.C. Feb. 26, 2016).
The Court is unpersuaded by Oxbow’s arguments. In its briefing, Oxbow declines to address any of the other proportionality factors highlighted in Rule 26—namely, the importance of the issues at stake in this action, the amount in controversy, the parties’ relative access to relevant information, the parties’ resources, or the importance of the discovery in resolving the issues in this case, see Fed. R. Civ. P. 26(b)(1)—stressing only that the burden and cost of complying with Defendants’ request would outweigh its likely benefit. Id. Weighing the six Rule 26 proportionality factors, however, demonstrates that adding Koch as a custodian of documents to be searched for material responsive to Defendants’ discovery requests in this matter will be neither unduly burdensome nor unreasonably expensive in light of the facts of this case. Likewise, the Court finds that the instant circumstances do not warrant shifting the costs of doing so to Defendants. Accordingly, Defendants’ motion to compel will be granted and Plaintiffs shall be ordered to produce all remaining responsive documents from Koch’s file, the cost of which Plaintiffs shall bear.
The Court appreciates that Koch’s files do not appear to contain as a high a proportion of responsive documents as the files of custodians who dealt exclusively with Oxbow’s coal and petcoke business, but it strains reason to suggest that the principal owner and CEO of a company, who has publicly commented on the importance and magnitude of litigation to which his company is a party and in which the financial health of his company is at issue, see Def. Ex. 22 [Dkt. 105-23] at 2-3, would have no unique information relevant to that litigation in his possession. While it may be too early in the production process to determine exactly how significant Koch’s records are, the categories of relevant documents identified by Defendants after reviewing the approximately 1,300 documents produced from Koch’s files indicates to the Court that Defendants’ discovery request has merit and is not intended to be the first strike in a war of attrition or a coercion tactic. Accordingly, the Court concludes that this factor favors granting Defendants’ proposed discovery.
The Court disagrees. The cost of reviewing and producing Koch’s documents does not strike the undersigned as unduly burdensome or disproportionate, especially given the discovery conducted to date and the damages that Oxbow seeks in this action. Plaintiffs’ counsel explained at the second hearing in this matter that Oxbow has spent $1.391 million to date on reviewing and producing approximately 584,000 documents from its nineteen other custodians and Oxbow’s email archive. See 8/24/17 TR. at 44:22-45:10. And again, Oxbow seeks tens of millions of dollars from Defendants. Through that lens, the estimated cost of reviewing and producing Koch’s responsive documents—even considering the total approximate cost of $142,000 for that effort, which includes the expense of the sampling effort—while certainly high, is not so unreasonably high as to warrant rejecting Defendants’ request out of hand. See Zubulake v. UBS Warburg, LLC, 217 F.R.D. 309, 321 (S.D.N.Y. 2003) (explaining, in the context of a cost-shifting request, that “[a] response to a discovery request costing $100,000 sounds (and is) costly, but in a case potentially worth millions of dollars, the cost of responding may not be unduly burdensome”); Xpedior Creditor Trust v. Credit Suisse First Boston (USA), Inc., 309 F. Supp. 2d 459, 466 (S.D.N.Y. 2003) (finding no “undue burden or expense” to justify cost-shifting where the requested discovery cost approximately $400,000 but the litigation involved at least $68.7 million in damages). Moreover, based on the parties’ representations at the second hearing in this matter, the projected number of responsive and unique documents in Koch’s files—approximately 10,000—is largely consistent with the number of responsive and unique documents produced by the other Oxbow custodians, and the responsiveness rate of Koch’s documents—11.67 percent—while low, is not the lowest among Oxbow’s custodians.
In light of the above analysis—including the undersigned’s assessment of each of the Rule 26 proportionality factors, all of which weigh in favor of granting Defendants’ motion—the Court is unwilling to find that the burden of reviewing the remaining 65,000 responsive documents for a fraction of the cost of discovery to date should preclude Defendants’ proposed request. See BlueAlly, 2017 WL 876266, at *5 (“This [last Rule 26] factor may combine all the previous factors into a final analysis of burdens versus benefits.” (citing Fed. R. Civ. P. 26 advisory committee’s notes)). For all of the reasons stated above, and absent any evidence establishing that Defendants are using the discovery of Koch’s records to wage a war of attrition or as a device to coerce Oxbow, the Court finds that Defendants’ motion must be granted.
Oxbow not only provides good legal analysis, it contains interesting disclosures of the costs involved in a large electronic document review project. The project in question was managed by plaintiffs’ counsel for Oxbow. They used the now popular, negotiated keyword search and linear hit review method. The keyword filtering used mere negotiated keywords that were not tested and refined on actual data. The linear review was limited to documents with a hit on one or more of the negotiated keywords. There were 584,000 such hit-documents. The rest of the documents of the nineteen custodians, the ones that did not have a keyword in them, were not reviewed. They were all presumed irrelevant.
Simple. But not necessarily the most effective or efficient method of document review. Certainly most e-discovery professionals would handle a large project like this in a different manner, especially considering the size of the case itself. Keyword search would be used, but only as part of a multimodal process that used additional search methods, such as concept searches, similarity searches and the all-powerful (if done right) predictive coding searches (AI). I have described this popular, but wrong methods before and called it the Distorted Search Pyramid. See eg. TAR Course: 8th Class (Keyword and Linear Review).
The parties completed phase one discovery on nineteen custodians with more than a few difficult negotiations and hearings concerning keyword negotiation. But they eventually got through it and the production was complete. After that, the defendants decided they wanted Koch’s relevant documents too. The defendants proposed using the same keyword filter on his data as was used on the first nineteen custodians.
[I]t strains reason to suggest that the principal owner and CEO of a company, who has publicly commented on the importance and magnitude of litigation to which his company is a party and in which the financial health of his company is at issue, see Def. Ex. 22 [Dkt. 105-23] at 2–3, would have no unique information relevant to that litigation in his possession.
Plaintiffs’ counsel explained at the second hearing in this matter that Oxbow has spent $1.391 million to date on reviewing and producing approximately 584,000 documents from its nineteen other custodians and Oxbow’s email archive.
The total cost of the phase one review was $1,391,000. Some may say yes, $2.38 per document is too expensive, and others may say no, $2.38 is very reasonable. Most experts would think that varying prices are in this range are to be expected and that the costs vary according to a number of factors, including market conditions and data complexity. In my view the only way to lower this price further is by use a full multimodal Predictive Coding Hybrid method, preferably one that uses IST instead of other kinds of continuous training.
In Oxbow the data appears to have been a fairly simple Outlook Exchange collection. The 584,000 documents referenced in the Opinion is the total keyword filter hit count for the first nineteen custodians. The files all appear to be ordinary vanilla emails and attachments. I wish we had the total custodian data count, in other words, the total number of deduplicated files for these nineteen custodians before the keyword culling, but we don’t. I would estimate the count to be between One and Three Million files, probably in the 1.5M area.
The defendants asked for plaintiffs to review the Koch files the same way they reviewed the others. Screen out all files that did not have a keyword on their negotiated list and review for relevance all of the others, namely all of the documents with hits and their families. Unlike the first nineteen custodians, the Opinion tells us the total data universe count for Koch. It was 467,614. Apparently, almost all of the 467,614 Koch files were emails and attachments.
Before the Koch files were processed, including deduping and keyword filtering, the parties could only guess at the total number of unique files would contain a keyword term. The plaintiffs estimated that there would be about 214,000 Koch files with a keyword hit. That would be the number of files that they would have to review for relevance. That 214,000 files estimate was highly speculative. There does not appear to have been any sampling done. It turns the estimate was way off, way too high, and with it, the cost estimate to review the Koch files.
Once the plaintiff actually processed the 467,614 Koch files and ran the keyword filter, there were only 45,639 files with a hit. That is 9.76%. When you add families of these documents to the review, which was their protocol, the total number of files to review increased to 82,000. After learning of the actual hit count plaintiffs were forced to reduce their cost estimate from $250,000 to $142,000. The new estimated cost of review is $1.73 per file. That is higher than the $1.17 per file cost when they estimated when there were 214,000 files to review, but still lower than their actual cost for the first nineteen custodians, $2.38.
The next thing that happened is that plaintiff’s reviewed a random sample of ten percent (10%) sample of the 82,000 total. The ten percent sample of hit documents and families ended up being 12,074 documents total. Plaintiff’s counsel reviewed these 12,074 documents and found that approximately 1,300 documents—11.67 percent of them—were actually relevant. These 1,300 documents were than produced or logged. Plaintiffs then reported to Judge Harvey that it cost them $57,197.95 to review these 12,074 documents. That is a cost of $4.74 per file, way higher than they had projected. But the higher cost is not surprising when you consider the heightened privilege expenses that you can expect when reviewing CEO emails.
Based on this experience plaintiff’s attorneys told Judge Harvey that it would likley cost another $85,000 to review the remaining 69,926 files (82,000-12,074). That would bring the total cost of the Phase Two review of Koch email, including the sample documents, to approximately $142,000. Judge Harvey points out several times in the opinion that that is “significantly less than Oxbow’s original estimate of $250,000.” The new $142,000 projected cost to review the last 69,926 files creates a rate of $1.22 per file. How plaintiffs’ counsel keeps coming up with different costs per file is not explained. That final number, $1.22 per file, shows that their last projection of $85,000 (and $142,000) is likely too low. Unless they change their review method, I do not see how or why the cost per file would be come down like that, especially since the sample review of the Koch emails cost $4.74 per file.
Plaintiffs’ counsel argued that there was not that much benefit from review of Koch files because only 11.67% were relevant. A technical way to say is by use of the term prevalence, meaning the percent of relevant documents. The prevalence of the Koch docs was 11.67%. They said that is too low a prevalence rate to justify the $142,000 expense under proportionality standards. It shows that there is not much benefit. The fallacies of this argument should be obvious. Koch is, after all, the CEO and owner of all plaintiff businesses. His unique relevant documents are obviously of more importance than the nineteen others that work for him. It is his company and he is the plaintiff in a major antitrust case. Of course he will be deposed and his documents reviewed.
Further, as Judge Harvey points out in Footnote Five of the Opinion, although Koch’s 11.67 % rate was low, one of the nineteen earlier custodians has a much lower rate than that, 3.03% and another had a 13.36% prevalence, which is only slightly higher than Koch’s 11.67%. For that reason Judge Harvey was not impressed by the fact that the average Prevalence rate of the first nineteen custodians was 65%.
That 65% prevalence rate is, in my experience, very high for this kind of keyword guessing approach. We often see prevalence rates less than 20% or 10% or less using this method. That in turn suggests that the keywords negotiated in the instant case were not overly burdensome on the responding party. This high a Precision, 65%, which is another way to look at it, means that there must be some trade-off in Recall. Typically with keyword search a high precision rate, one greater than 50%, is at the expense of some loss in Recall, meaning missing relevant documents. The 65% suggests that many relevant documents were likely left on the table, in other words, filtered out of the review because they did not contain one of the keywords. That is one reason why we try to never rely on keyword search alone, but to take a multimodal approach. The mono-modal approach of just using negotiated keywords is too unreliable, too risky. It will sometime miss key evidence that used unexpected text. That is why we teach Multimodal Culling.
Successful arguments on motions to compel require hard evidence. To meet your burden of proof you must present credible estimates of the costs of document review. This requires experience, market knowledge and complete data about the files (metadata).
The arguments on both sides of a motion to compel ESI production should begin with reliable metrics and statistics concerning the ESI that the requesting party wants the responding party to review. This requires disclosures by the responding party, but these data disclosures are limited and probably do not matter to anyone else in any event. Who care how many files are in Mr. X’s Outlook Mailbox?
In the motions counsel should explain the costs and billings involved in the proposed document review project and, ideally, also provide a comparative reference to other projects. This helps a judge to determine reasonability. Arguments can be based on your general experience with document review costs and on the specific costs already incurred in the instant case. Emphasis should be put on current project costs and prevalence rates.
Remember to do the cost per file calculations in every motion. They are great metrics for comparisons. Apparently that was never done in Oxbow because the plaintiffs counsel’s cost estimates varied so much. A per file cost analysis shows that they missed a key metric, one that could have helped their motion. The cost ranged from an actual cost of $2.38 per file for the first 584,000, to an 1.17 per file estimate to review 214,000 Koch files, to an estimate of $1.73 per file to review 82,000 Koch files, to an actual cost of $4.74 per file to review 12,074 Koch files, to another estimate of $1.22 per file to review the remaining 69,926 Koch files. The actual costs are way higher than their estimated costs.
Finally, in these motions counsel should mention all six factors under Rule 26(b)(1), not just cost burden. Still, in many cases like Oxbow Carbon & Minerals LLC v. Union Pacific Railroad Company, it may be appropriate to focus your memorandum on the cost estimates. Realistic concessions are always appreciated by the judiciary and help to narrow the issues.
This is the final part four of the series by Ralph Losey explaining his document review strategy. Please read part one, two, and three first.
Let us change the scenario somewhat for a final quick hypothetical. Assume 10,000,000 documents after bulk culling for your SMEs to review for relevance. Assume sampling by an SME showed a prevalence of 10% (again somewhat high), and a first-pass review rate of 10,000 files per hour (somewhat slow for Big Data reviews). This means you can assume that only around 1,000,000 documents will need final second pass protection review.
More sampling shows the contract reviewers will, using computer-assisted techniques (smart routing, etc.), be able to review them at 100 files per hour. The improvement in protections review from the small case example, where a speed of only 20 files per hour was assumed, is possible when you have larger document collections sizes like this (one million documents), high quality review software, and experienced review teams.
With this information from sampling you can now estimate a total first pass review cost of $1,000,000.00 ($1,000.00 x 1,000 hours), which compares pretty well to the $2,829,349.10 charge in Gabriel. Next you can estimate a total final pass Protection review cost of $500,000.00 ($50 x 10,000 hours).
Lets us also assume you know from experience that your other supervision fees and software costs are likely to total another $150,000.00. Your estimate for the project would be $1,650,000.00. That is 16.5 cents a document ($0.165).
Too high you say? Not proportionate to the value of the case? Not proportionate to the expected probative value in this case from these 10,000,000 documents, which is something your sampling can tell you and provide evidence to support? Then use the document ranking feature of AI enhanced document review to further reduce the second pass protection costs.
If the SME’s identification of 1,000,000 likely relevant documents was based on a 50% plus probability ranking using predictive coding, then try a higher-ranking cut off. Again, this is not too difficult to do with sampling and good software. Maybe a 75% plus ranking cut off brings the count down from 1,000,000 to 250,000. Or maybe you just arbitrarily decide to use the top ranked 250,000 documents because that is all you can afford, or think is proportionate for this data and this case. Either way, you are now passing the strongest document sets along for second pass review. You will probably be producing the most relevant documents, the ones likely to have the highest probative value to the case.
Using the higher cut off, assume that the cost for second pass Protection review would then be 25% of what it was, reduced from $500,000.00 for review of 1,000,000 documents, to $125,000.00 to review 250,000 documents. Your other fees and costs also drop in your experience by 50%, from $150,000.00 to $75,000.00. Your total estimate is now $1,200,000.00, instead of $1,650,000.00. You have gone down to twelve cents a document. Assume this $1,200,000.00 number is now within your legal judgment to be proportional to this document request in this very large lawsuit. It is now within your budget.
If this was still not proportional in your judgment, or the client’s opinion, there are many other things you could try, but I would focus on the bulk culling before the SME first pass relevancy review. For instance, change the custodian count, or date range, but do not keyword filter like they did in Gabriel. Bring the initial 10,000,000 documents down to 5,000,000 documents, then do the math (500,000+250,000+37,500). You are now talking about around $800,000.00, back to sixteen cents per document. Is that within the budget? Is that an amount that a court is likely to force you to spend anyway?
When a producing party uses advanced computer review software with predictive coding ranking, they can limit their review and production to the documents with the highest relevancy ranking. This gives everyone the most truth for the dollar. For that reason this is the best approach, not only for the producing party, who can thereby budget and avoid disproportionate burdens, but also for the requesting party. They can also limit their expense to review the many thousands of documents (sometimes millions of documents) produced to them. They can also focus on the documents with the highest value. They can be reasonably assured that AI enhanced processes have probably found the documents that really matter to the resolution of the case. The plaintiffs understood in Kleen Products, a large class action pending in District Court in Chicago, which is one reason they tried to force the defendants to use predictive coding in their productions.
In spite of the Kleen Products precedent, not all plaintiff’s counsel are as sophisticated as counsel in that large class action. A producing party, especially in a David and Goliath type case, will often have to sell the benefits of these new methods to the smaller requesting party. The requesting party will be more likely to cooperate if they understand the value to them of these methods. This will often require the producing party to provide some reasonable degree of transparency into the proposed review processes. For instance, if you have well known experts retained to direct the review, tell them. Share his or her qualifications and experience. Engage in frank and full relevancy dialogues with the requesting party. Make sure you are training the machine to find the documents that they really want. Clarify the target. If need be, share some examples early on of the relevant documents you will use in the training. Invite them to provide you with documents they consider relevant to use in the training. Try to make it a collaborative approach. If they refuse, document your efforts and take any disputes to the judge as early as possible.
A collaborative approach is an effective way to convince a requesting party that your estimate is reasonable and that you are not still stuck in the old paradigm of hide-the-ball discovery games. I cannot overstate how important it is to develop trust between opposing counsel on discovery. Often the only way to do that is through some level of transparency. You do not have to disclose all of your search secrets, but you may have to keep the requesting party informed and involved in the process. That is what cooperation looks like.
In order to be able to make a valid estimate for bottom line driven proportional review you must closely study the case and review the project goals. It is not enough to have handy per file cost estimates. This move to actual examination of the electronically stored information at issue, and study of the specific review tasks, is equivalent to the move in construction estimation from rough estimates based on average per square foot prices, to a site visit with inspection and measurements of all relevant conditions. No builder would bid on a project without first doing the detailed real world estimation work. We lawyers must do the same for this method to succeed.
Even in the same organization, and just dealing with email, the variances between document custodians can be tremendous. Some, for instance, may have large amounts of privileged communications. This kind of email takes the most time to review, and, if relevant, to log. High percentages of confidential documents, especially partially confidential, can also significantly drive up the costs of the second pass review. All of the many unique characteristics of electronic document collections can affect the speed of review and total costs of review. That is why you have to look at your data and test sample the emails in your collection to make accurate predictions. Estimation in the blind is never adequate. It would be like bidding on a construction project without first reading the plans and specifications.
Even when you have dealt with a particular client’s email collection before, a repeat customer so to speak, the estimates can still vary widely depending on the type of lawsuit, the issues, and the amount of money in controversy or general importance of the case.
Although this may seem counter-intuitive, the truth is, complex, big-ticket cases are the easiest to do e-discovery, especially if your goal is to do so in a proportional manner. If there is a billion dollars at issue, a reasonable budget for document review is large. On the other hand, proportional e-discovery in small cases is a real challenge, no matter how simple they supposedly are. Many cases that are small in monetary value are still very complex. And complex or not, all cases today have a lot of electronically stored information.
The medium size to small cases are where my bottom line driven proportional review has the highest application for cost control and the greatest promise to bring e-discovery to the masses. Give it a try, with or without AI enhanced predictive coding. For large cases, you should always employ AI enhanced methods. The ranking feature makes it easier to implement a proportional approach. Bottom line driven proportional review in big cases can save the greatest amount of money and provide much needed certainty and risk control.
The future of litigation involves new methods of technology-assisted discovery where Man and Machine work together to find the core truth. This day will come; in fact it is already here. As the science fiction writer William Gibson said: The future is already here – it’s just not evenly distributed yet. The key facts you need to try a case and do justice can be found in any size case, big and small, at an affordable price. But you have to embrace change and adopt new legal and technical methodologies. The Bottom Line Driven Proportional Review method is part of that answer, so too is advanced review software at affordable prices. It is already working for some today, it can work for you too.
For questions or comments, please feel free to contact the author at: Ralph.Losey@JacksonLewis.com or Ralph.Losey@Gmail.com.
 A random sample is step-three in my recommended eight-step methodology for AI enhanced review, aka predictive coding. See eg my blog at https://e-discoveryteam.com/car/ and the EDBP at http://www.edbp.com/search-review/predictive-coding/. For more on random sampling see Losey, R., Comparative Efficacy of Two Predictive Coding Reviews of 699,082 Enron Documents (Part Two) found at https://e-discoveryteam.com/2013/06/17/comparative-efficacy-of-two-predictive-coding-reviews-of-699082-enron-documents/; and Losey, R. Random Sample Calculations And My Prediction That 300,000 Lawyers Will Be Using Random Sampling By 2022 found at https://e-discoveryteam.com/2012/05/06/random-sample-calculations-and-my-prediction-that-300000-lawyers-will-be-using-random-sampling-by-2022/.
 Losey, R. Relevancy Ranking is the Key Feature of Predictive Coding Software found at https://e-discoveryteam.com/2013/08/25/relevancy-ranking-is-the-key-feature-of-predictive-coding-software/. Relevancy ranking only works well now with the best software on the market and requires proper, accurate, machine training. Even then it is not an exact measure of relevancy weight, and testing and quality controls should always be applied to know when and to what degree the ranking strata are reliable.
 The costs of review have come way, way down in the past few years for those who are using advanced methods. For some context on the fourteen cents a document number used in this example, back in 2007 the Department of Justice spent $9.09 per document for review in the Fannie Mae case, even though it used contract lawyers for the review work. In re Fannie Mae Securities Litig., 552 F.3d 814, 817 (D.C. Cir. 2009) ($6,000,000/660,000 emails). There were no comments by the court that this price was excessive when the government later came back and sought cost shifting. At about the same time Verizon paid $6.09 per document for a massive second review project that enjoyed large economies of scale and, again, utilized contract review lawyers. Roitblat, Kershaw, and Oot, Document categorization in legal electronic discovery: computer classification vs. manual review. Journal of the American Society for Information Science and Technology, 61(1):70–80, 2010 ($14,000,000 to review 2.3 million documents in four months). In 2011 I was still seeing an average cost of $5.00 per file for tested keyword multimodal reviews before advanced predictive coding engines started to become available.
 See: Losey, R., Why a Receiving Party Would Want to Use Predictive Coding? found at https://e-discoveryteam.com/2013/08/12/why-a-receiving-party-would-want-to-use-predictive-coding/.
 Relevancy dialogues, which I also call ESI Discovery Communications, is step-one in my standard eight step methodology for predictive coding. See the EDBP description found at: http://www.edbp.com/search-review/predictive-coding/.
 Kleen Products, LLC, et al. v. Packaging Corp. of Amer., et al., 2012 WL 4498465 (ND, Ill., Sept. 28, 2012). Also see: Losey, R., Attorneys Admonished by Judge Nolan Not to “Confuse Advocacy with Adversarial Conduct” and Instructed on the Proportionality Doctrine.

References: v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v.