\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 1 28-MAY-02 12:35

PAST AND POTENTIAL USES OF EMPIRICAL

RESEARCH IN CIVIL RULEMAKING

Thomas E. Willging*

INTRODUCTION

In the words of an experienced observer of and participant in the

civil rulemaking process, a “group of elite lawyers and law professors

who acted with little empirical evidence”1 drafted the original Federal

Rules of Civil Procedure (Rules).2 During the first fifty years after

adoption of the rules, rulemakers rarely commissioned or used empirical research to support their revisions.3 Until the last decade or so,

* Senior researcher at the Federal Judicial Center. Since 1988, the author has

represented the Center at meetings of the Advisory Committee on Civil Rules and has

participated in a majority of the research discussed in this Article. Any views

expressed in this Article are, of course, his own and not necessarily those of the

Federal Judicial Center, the Advisory Committee, or any other entity in the judicial

branch.

The Honorable David Levi, chair of the Advisory Committee on Civil Rules,

discussed his views on the subject of this Article in an interview with the author.

Professors Edward Cooper and Richard Marcus, reporter and special reporter,

respectively, to the Advisory Committee on Civil Rules; the Honorable Fern Smith,

Director of the Federal Judicial Center; Russell Wheeler; James Eaglin; Joe Cecil; and

David Rauma each gave many helpful suggestions in commenting on a previous draft.

Any remaining deficiencies are the author’s sole responsibility.

1 Richard L. Marcus, Of Babies and Bathwater: The Prospects for Procedural Progress,

59 BROOK. L. REV. 761, 782 (1993); see also Paul D. Carrington, The New Order in Judicial Rulemaking, 75 JUDICATURE 161, 165 (1991) (stating that the rulemakers worked

“in secret over 18 months” and that they allowed “only a brief period of public comment,” thus creating a basis for inferring that they did not seek empirical research).

2 I use the term “Rule” in this Article to refer to one of the Federal Rules of Civil

Procedure. If referring to any other rules, I will specifically designate the type of rule.

3 See Maurice Rosenberg, The Impact of Procedure-Impact Studies in the Administration of Justice, LAW & CONTEMP. PROBS., Summer 1988, at 13, 13 [hereinafter Rosenberg, Procedure-Impact Studies] (“Empirical research on the functioning of procedural

rules has had a slowly growing impact on the litigation process. . . . Regrettably, the

examples of such impacts have not been numerous.”). Most of the examples Professor Rosenberg cited were studies of the impact of existing rules and were not necessarily designed to inform rulemakers. Notable exceptions were the Federal Judicial

Center’s case management study, which the Advisory Committee cited in support of

1121
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 2 28-MAY-02 12:35

1122 notre dame law review [vol. 77:4

those rulemakers, with rare but notable exceptions,4 followed the lead

of the original drafters and promulgated rules without the benefit of

systematic empirical observation of the impact of current rules or empirical testing of proposed rule revisions.5

The tone set by the original rulemakers and their successors

came under attack in the late 1980s and early 1990s when commentators decried the lack of empirical support for major rule revisions relating to Rule 11 sanctions in 1983 and Rule 26(a) initial disclosures

in 1993.6 One commentator called boldly for a moratorium on

rulemaking and all other procedural change until a thorough, empirically focused study of past, present, and proposed procedural changes

the 1983 amendments to Rule 16, and Professor Rosenberg’s Columbia Project, which

the Advisory Committee cited in support of the 1970 amendments to the discovery

rules. Id. at 25–27. For the Federal Judicial Center study, see STEVEN FLANDERS, CASE

MANAGEMENT AND COURT MANAGEMENT IN UNITED STATES DISTRICT COURTS (1977).

For the Columbia Project, see WILLIAM A. GLASER, PRETRIAL DISCOVERY AND THE ADVERSARY SYSTEM (1968), and Maurice Rosenberg, Changes Ahead in Federal Pretrial Discovery, reprinted in 45 F.R.D. 479 (1969).

4 In addition to the examples cited supra note 3, a perusal of the committee

notes to the rules reveals the following two references to empirical work. Both references cite the same empirical study, but in different directions, demonstrating that

empirical research remains subject to differing interpretations by policymakers in the

context of specific issues and proposals.

In 1980, the Advisory Committee stated its belief that “abuse of discovery, while

very serious in certain cases, is not so general as to require such basic changes in the

rules that govern discovery in all cases. A very recent study of discovery in selected

metropolitan districts tends to support its belief.” FED. R. CIV. P. 26(f) advisory committee’s notes (1980 amend.) (citing PAUL R. CONNOLLY, EDITH A. HOLLEMAN &

MICHAEL J. KUHLMAN, JUDICIAL CONTROLS AND THE CIVIL LITIGATIVE PROCESS: DISCOVERY (1978)).

In 1983, the Committee adopted substantial changes to the discovery rules and

opened its note with the statement that “[e]xcessive discovery and evasion or resistance to reasonable discovery requests pose significant problems. Recent studies have

made some attempt to determine the sources and extent of the difficulties.” FED. R.

CIV. P. 26(f) advisory committee’s notes (1983 amend.) (citing PAUL R. CONNOLLY,

EDITH A. HOLLEMAN & MICHAEL J. KUHLMAN, JUDICIAL CONTROLS AND THE CIVIL LITIGATIVE PROCESS: DISCOVERY (1978), and three additional studies).

5 See Rosenberg, Procedure-Impact Studies, supra note 3, at 13.

6 See, e.g., Stephen B. Burbank, Ignorance and Procedural Law Reform: A Call for a

Moratorium, 59 BROOK. L. REV. 841, 844, 845 (1993) (stating that “amended Rule 11

was promulgated in a virtual empirical vacuum,” and that “there was little relevant

empirical evidence” regarding disclosure); Linda S. Mullenix, Hope over Experience:

Mandatory Informal Discovery and the Politics of Rulemaking, 69 N.C. L. REV. 795, 810

(1991) (revealing that when Rule 26(a) was proposed for adoption there had been

“virtually no empirical study of the current practice of such informal discovery, the

efficacy of such experiences, or the results of informal discovery”).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 3 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1123

could be completed.7 Another commentator, equally bold, called,

equally boldly, for limiting rule changes to those supported by controlled experiments, that is, research in which cases would be assigned

randomly to a litigation context in which a proposed rule would be

applied or not.8 Yet another commentator criticized the Advisory

Committee on Civil Rules’s (Advisory Committee or Committee) failure, prior to the 1993 amendment to Rule 26(a), to investigate empirically the experience of district courts with local rules regarding

disclosure9 and prophesized the demise of judicial rulemaking as we

then knew it, maintaining that politicization of the process could not

be avoided in the absence of empirical grounding for proposed rules

changes.10 Other commentators have called for explicit congressional

authorization to create a national body that would review proposed

“experimental” local rules and facilitate empirical evaluation of such

time-limited local “experiments.”11 In 1995, a self-study committee of

the Judicial Conference’s Standing Committee on Rules of Practice

and Procedure (Standing Committee) urged the Advisory Committees to seek and use empirical data in making decisions about proposed rules changes.12

7 See Burbank, supra note 6, at 855.

8 See Laurens Walker, Perfecting Federal Civil Rules: A Proposal for Restricted Field

Experiments, LAW & CONTEMP. PROBS., Summer 1988, at 67, 73; see also Richard A. Posner, The Summary Jury Trial and Other Methods of Alternative Dispute Resolution: Some Cautionary Observations, 53 U. CHI. L. REV. 366, 367 (1986) (“If we are to experiment with

alternatives to trials, let us really experiment; let us propose testable hypotheses, and

test them.”).

9 See generally Mullenix, supra note 6 (discussing informal discovery rules and the

rulemaking procedure).

10 See id. at 801 (“[T]he inevitable politicization of the Civil Rules Advisory Committee foreshadows the decline of that body’s role in procedural rule-drafting.”).

Professor Mullenix suggested that the Advisory Committee might have to forfeit the

pleasures of “genteel, deliberative rulemaking” and declared that “the question remains open whether the Advisory Committee is destined to go the way of the French

aristocracy.” Id. at 802.

11 See A. Leo Levin, Local Rules as Experiments: A Study in the Division of Power, 139

U. PA. L. REV. 1567, 1585–94 (1991); see also Carl Tobias, A Modest Reform for Federal

Procedural Rulemaking, LAW & CONTEMP. PROBS., Spring/Summer 2001, at 283,

286–87. These authors do not appear to use the term “experiment” in the scientific

or technical sense of establishing both experimental and control groups to test a proposal. See infra text accompanying notes 20–39.

12 See SUBCOMM. ON LONG RANGE PLANNING TO THE COMM. ON RULES OF PRACTICE,

PROCEDURE AND EVIDENCE OF THE JUDICIAL CONF. OF THE U.S., A SELF-STUDY OF FEDERAL JUDICIAL RULEMAKING, reprinted in 168 F.R.D. 679, 699 (1995) [hereinafter SELFSTUDY].
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 4 28-MAY-02 12:35

1124 notre dame law review [vol. 77:4

After these critiques—and perhaps spurred by them13—the Advisory Committee14 often solicited and otherwise encouraged empirical

studies regarding proposed rules changes. Since 1988, the Committee

frequently has consulted the Federal Judicial Center (FJC or Center)15

and occasionally contacted private research organizations16 for assistance in examining the current operation of the rules.17 Empirically

13 This Article does not attempt to trace the root causes of increased use of empirical data in rulemaking. In general, calls for empirical research appear to have

become an integral part of the policymaking culture. See, e.g., Stephen Daniels &

Joanne Martin, Punitive Damages, Change, and the Politics of Ideas: Defining Public Policy

Problems, 1998 WIS. L. REV. 71, 91 (“[T]he best way to define something as a problem

‘in our profoundly numerical contemporary culture’ is to measure it.” (quoting

DEBORAH A. STONE, POLICY PARADOX AND POLITICAL REASON 136 (1988))).

14 The Advisory Committees on Criminal Rules, Bankruptcy Rules, Appellate

Rules, and Evidence Rules also have used empirical evidence on occasion, and this

Article will discuss some of those uses. The Standing Committee rarely initiates the

drafting of proposed rules. Recently, however, the Standing Committee has taken the

lead in researching the need for rules relating to attorney conduct and has commissioned empirical work to support that effort. See Daniel R. Coquillette & Marie Leary,

Working Papers of the Committee on Rules of Practice and Procedure: Special Studies of Federal Rules Governing Attorney Conduct (Sept. 1997) (unpublished manuscript, on file with the author).

15 The Federal Judicial Center is an agency of the judicial branch of the federal

government created by Congress in 1967 to, among other purposes, “conduct research and study of the operation of the courts of the United States, and to stimulate

and coordinate such research and study on the part of other public and private persons and agencies.” 28 U.S.C. § 620(b)(1) (1994). Congress also authorized the

Center, if “consistent with the performance of the other functions set forth in this

section, to provide staff, research, and planning assistance to the Judicial Conference

of the United States and its committees.” For a historical-political account of the creation of the Center, including the centrality, in the minds of both Chief Justice Warren

and Administrative Office of the Courts Director Warren Olney III, of the proposed

Center’s research function, see Russell R. Wheeler, Empirical Research and the Politics of

Judicial Administration: Creating the Federal Judicial Center, LAW & CONTEMP. PROBS.,

Summer 1988, at 31, 31–53.

16 See, e.g., DEBORAH R. HENSLER, NICHOLAS M. PACE, BONITA DOMBAY-MOORE,

BETH GIDDENS, JENNIFER GROSS & ERIK K. MOLLER, CLASS ACTION DILEMMAS: PURSUING

PUBLIC GOALS FOR PRIVATE GAIN (2000); James S. Kakalik, Deborah R. Hensler, Daniel

McCaffrey, Marian Oshiro, Nicholas M. Pace & Mary E. Vaiana, Discovery Management:

Further Analysis of the Civil Justice Reform Act Evaluation Data, 39 B.C. L. REV. 613, 613

(1998). Both of the above studies were encouraged and used by the Advisory Committee in its deliberations on its 2001 proposals to amend Rule 23 and its 2000

amendments to the discovery rules, respectively.

17 In addition to soliciting empirical research, the Committee routinely contacts

national bar groups, such as the American Bar Association’s Section of Litigation and

the American College of Trial Lawyers to obtain their views about proposed rules.

Interest groups routinely attend Committee meetings, monitor Committee actions,

and present testimony or informal commentary on proposed rules.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 5 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1125

trained staff of the Center, usually the author, attend Advisory Committee meetings and public hearings to provide information to the

Committee in regard to empirical questions that may arise in the

course of deliberations. Based on the author’s perspective developed

in the course of representing the Center at Advisory Committee meetings and presenting Center research to the Committee, this Article

will discuss some of the advantages, disadvantages, potential benefits,

and distinct limitations of conducting empirical research to inform

the rulemaking process. For the most part, the Article draws on specific examples of research conducted during rulemaking. Typically,

the Center undertook the research discussed below at the request of

the Advisory Committee for the express purpose of examining empirical questions posed by rulemakers as they contemplated changes in

the rules.

Part I examines more closely the commentators’ calls for

rulemakers to seek and use empirical data to inform their deliberations. It discusses the variety of empirical research designs that researchers might use to generate meaningful information. Most

importantly, Part I distinguishes experimental research designs, which

can produce powerful inferences about causal relationships, from

more limited designs, such as observational field investigations and

case studies.

Part II describes fourteen recent examples of the Advisory Committee’s use of empirical research conducted at its behest. In the

course of describing the research, this Part explores specific instances

of linkages between the research and the proposed rules.

Part III reviews the examples described in Part II and discusses

patterns that emerge from recent uses of empirical research in drafting and promulgating amendments to the Federal Rules of Civil Procedure. Part III focuses on the timing of requests and reports, designs

used in conducting the research, the underlying research questions,

and the main purposes served by the requests. In addition, Part III

analyzes and summarizes outcomes of the rulemaking proposals and

notes any apparent relationship between the proposed rules and legislative proposals. It also explores elements of the current rulemaking

process that facilitate and limit the type of research that can be conducted, such as term limits for Committee members and chairs and

time limits on conducting the research.

Part IV analyzes the potential benefits of and limits imposed by

employing experimental research to support rulemaking. It asks what

conditions would permit experimental studies of proposed rules as

opposed to studies of the impact of rules already enacted. Specifically, this Part calls attention to the need for the Standing Committee
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 6 28-MAY-02 12:35

1126 notre dame law review [vol. 77:4

and the Advisory Committee to consider seeking general congressional authorization to permit the Standing Committee to adopt experimental rules to test the prospective impact of a proposed rule

change before a final rule is considered. Part IV concludes with a call

for the Standing Committee and the various advisory committees to

consider revising the paradigm they use for adopting major rules

changes and to consider creating a mechanism for conducting experimental research in appropriately limited circumstances.

I. WHAT IS EMPIRICAL RESEARCH AND WHAT TYPES OF EMPIRICAL

RESEARCH HAVE COMMENTATORS SOUGHT FOR THE

RULEMAKING PROCESS?

“Empirical research” is used in this Article to refer to information

collected through systematic observation and experience (in contrast,

for example, to information derived through theory or logic).18 Ordinarily, one speaks of such experience and observation as research (as

opposed to anecdote) only if a researcher systematically selects representative or otherwise appropriate cases for study, documents the observations, and applies the same criteria to describing each separate

observation, generally using a predetermined set of questions, sometimes referred to as a research protocol, survey instrument, or

questionnaire.

Research typically involves testing hypotheses generated by previous research, by academic theories, or, in the Center’s experience, by

policy questions of importance to the judiciary. Generally, an empirical statement is one that can be proven wrong.19 Empirical research

designs encompass experimental research, quasi-experimental research, observational studies, and case studies. Research methods include surveys and focus groups. Empirical research can take place in

the field, in a laboratory, or even in a library setting. Samples of subjects to be studied can be selected on a random basis or for the convenience of the researcher. Empirical research can be reported on a

quantitative or qualitative basis. Those terms will be defined and discussed below, with examples.

18 See W. PAUL VOGT, DICTIONARY OF STATISTICS AND METHODOLOGY: A NONTECHNICAL GUIDE FOR THE SOCIAL SCIENCES 95 (2d ed. 1999).

19 See Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 593 (1993) (“Scientific

methodology today is based on generating hypotheses and testing them to see if they

can be falsified; indeed, this methodology is what distinguishes science from other

fields of human inquiry.” (quoting Michael D. Green, Expert Witness and Sufficiency of

Evidence in Toxic Substances Litigation: The Legacy of Agent Orange and Bendectin Litigation,

86 NW
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 7 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1127

A. Research Design

1. Experimental Research

For purposes of this Article, the primary distinction to be drawn is

between experimental (including quasi-experimental) research designs and non-experimental research designs. The key element of a

controlled experiment is that it generally involves the random assignment of cases to experimental treatment and control groups in which

participants, including judges, will or will not be subject to the proposed rule, procedure, or program.20 The goal is to assess whether a

particular treatment (embodied, for example, in a draft rule) has the

predicted effect. Random assignment to treatment and control

groups has the effect of making the groups equal before the treatment is applied.

Statistical methods allow the researcher to test the hypothesis

(generally framed in the negative, that the treatment does not work)

and state the probability that any observed differences in outcomes

can be attributed to chance and not to the treatment. If the number

of cases is sufficiently large, the researcher can speak of results as being statistically significant at a ninety-five percent level, the benchmark chosen by most scientists for making such a statement.21 This

means that there is less than a one in twenty probability that a putative

causal relationship occurred by chance.22 This quality of experimental research—“the extent to which the results of a study . . . can be

attributed to the treatments”—is generally referred to as internal validity.23 Whether the results can be generalized or applied to other

contexts, which researchers denote as external validity, depends on

20 See FED. JUDICIAL CTR., EXPERIMENTATION IN THE LAW: REPORT OF THE FEDERAL

JUDICIAL CENTER ADVISORY COMMITTEE ON EXPERIMENTATION IN THE LAW 17–18

(1981).

21 See ANTHONY PARTRIDGE & ALAN LIND, A REEVALUATION OF THE CIVIL APPEALS

MANAGEMENT PLAN 22 (1983).

22 A finding of statistical significance, however, does not mean that a relationship

is necessarily important or meaningful. It simply means that enough cases were examined to rule out the effects of chance. See David H. Kaye & David A. Freedman,

Reference Guide on Statistics, in REFERENCE MANUAL ON SCIENTIFIC EVIDENCE 83, 124 (2d

ed. 2000) (stating that “significant differences are evidence that something besides

random error is at work”). The terms “practical” or “substantive” significance refer to

the strength of a relationship. See VOGT, supra note 18, at 219, 283. In designing an

experiment and framing hypotheses, a researcher will generally structure the experiment so that the relationships to be tested are meaningful.

23 VOGT, supra note 18, at 143.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 8 28-MAY-02 12:35

1128 notre dame law review [vol. 77:4

the extent to which the other situations are similar to the context in

which the research was conducted.24

Experimental research can be conducted in the field or in a laboratory setting.25 A good example of a procedural field experiment is

the U.S. Court of Appeals for the Second Circuit’s 1974 implementation of a program for attorneys, before briefing and arguing an appeal, to participate in a mandatory conference with an attorney

employed by the court. The court worked with the FJC to construct

an experiment to test the program, which is called the Civil Appeals

Management Program (CAMP).

The court first identified all cases that might benefit from a

mandatory conference. To provide a comparative foundation for the

research, the court then mandated that a conference be held in a

portion of the cases (the experimental group) and not in the remaining cases (the control group).26 Using docket numbers, the court assigned the first case to a court attorney, the next case to another court

attorney, and the third case to the control group until a sufficient

number of cases were in the experiment.27 Researchers then measured the effects of the program on the rate of settlement, the time

required for the appeals, the quality of briefs and arguments, and on

the attorneys’ satisfaction with the program.28 In comparing case out24 Id. at 105.

25 Laboratory research is beyond the scope of this Article. Social psychologists

and other social scientists have conducted extensive laboratory research that is highly

relevant to the legal system. For an overview of such research, see IRWIN A.

HOROWITZ, THOMAS E. WILLGING & KENNETH S. BORDENS, THE PSYCHOLOGY OF LAW

(2d ed. 1998). Journals, such as Law and Human Behavior; Psychology, Public Policy &

Law; and Law & Psychology Review, often publish the results of laboratory experiments

and literature reviews summarizing such results. For a recent example that illustrates

the scope of such research, see Dennis J. Devine, Laura D. Clayton, Benjamin B. Dunford, Rasmy Seying & Jennifer Pryce, Jury Decision Making: 45 Years of Empirical Research

on Deliberating Groups, 7 PSYCHOL., PUB. POL’Y & L. 622 (2001).

Some journals focus on publishing empirical work arising out of studies of the

legal system. They include the Journal of Legal Studies (a publication of the University

of Chicago Law School); Judicature (American Judicature Society); Jurimetrics: The Journal of Law, Science, and Technology (Section of Science & Technology Law, American

Bar Association); Law & Social Inquiry (American Bar Foundation); and Law & Society

Review (Law & Society Association).

26 See PARTRIDGE & LIND, supra note 21, at 1–3. For earlier analyses that proved to

be less conclusive, see JERRY GOLDMAN, AN EVALUATION OF THE CIVIL APPEALS MANAGEMENT PROGRAM: AN EXPERIMENT IN JUDICIAL ADMINISTRATION (1977), and Jerry

Goldman, The Civil Appeals Management Plan: An Experiment in Appellate Procedural Reform, 78 COLUM. L. REV. 1209 (1978).

27 PARTRIDGE & LIND, supra note 21, at 23–24. Exceptions were made for excluded cases, consolidated cases, and cases related to an earlier appeal. Id. at 24–28.

28 Id. at 5–8.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 9 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1129

come information from the two sets of cases and data from surveys of

both sets of attorneys, researchers attempted to sever any association

with other known or unknown factors (such as the identity of the

judges, the type of case, and any local rules or practices) that might

affect the outcome.29 The researcher could then identify the differences in the outcomes of the two sets of cases, apply the appropriate

statistical tests, and determine whether any differences are attributable to the mandatory conference.

The CAMP research experience suggests that experimental research, for all its benefits, may not be a sport for the short-winded.

The CAMP studies encompassed two separate data gathering operations and nonetheless yielded somewhat inconclusive results: FJC researchers suggested almost a decade after the program began that “a

balanced decision for most other courts of appeals would be to institute CAMP-like programs but not to go full speed.”30 Later experimental evaluation of a CAMP-like program in another circuit

produced a precise measure of the savings in judicial resources attributable to the program,31 giving the policymakers information from

which to make a judgment about whether a program that does the

work of approximately one appellate judge is justified.32 That study

took between three and four years from the time of the court’s request for research until the Center provided final results to the

court.33

Another form of experimental research uses a quasi-experimental

research design. In a quasi-experimental design, the researcher generally does not attempt to control the assignment of cases but does

attempt to manipulate variables to create two or more distinct groups

29 Id. at 21–22.

30 Id. at 11.

31 See JAMES B. EAGLIN, THE PRE-ARGUMENT CONFERENCE PROGRAM IN THE SIXTH

CIRCUIT COURT OF APPEALS (1990). That report evaluated a program begun in 1982.

Id. at 1. By 1990, variations of the CAMP program had been implemented in the

Second, Eighth, Ninth, and District of Columbia Circuits. Id. at 41. Currently, there

are court-staffed mediation or settlement conference programs in all of the courts of

appeals except for the Federal Circuit. JUDITH A. MCKENNA, LAURAL L. HOOPER &

MARY CLARK, CASE MANAGEMENT PROCEDURES IN THE FEDERAL COURTS OF APPEALS

26–32 (2000).

32 See EAGLIN, supra note 31, at 41–42 (standing for the proposition that the request was made in 1995).

33 See id. at 1. The Center provided interim results to the court on an ongoing

basis. The first request was in early 1985, and the Center communicated final results

to the court in February 1989. E-mail from James B. Eaglin, Research Division Federal Judicial Center, to Thomas E. Willging (Jan. 14, 2002).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 10 28-MAY-02 12:35

1130 notre dame law review [vol. 77:4

for comparing the effects of a treatment.34 For example, in studying

offers of judgment under Rule 68, an FJC researcher surveyed attorneys in two distinct sets of cases, those that had settled and those that

had been tried.35 Manipulating those variables allowed the researcher

to draw clear comparisons, but not make causal attributions.36

Experimental research enables the researcher to assess whether

the treatment caused the behavior that was predicted at the outset of

the experiment. In the opinion of many researchers, an experiment

is the only research design that so enables the researcher.37 Others

disagree and assert that statistical methods can sometimes be used to

generate causal inferences based on an analysis of multivariate relationships, that is, relationships involving three or more variables.38

Putting this debate to one side, one of the leading legal-social researchers has captured the consensus that “[w]ithout doubt, the most

powerful and reliable way to investigate the impact of a legal rule,

procedure, program, or other intervention is to conduct a controlled

experiment.”39

Professor Rosenberg’s summary of experimental field research

showed the methodology to have been rarely used outside the laboratory. In addition to the CAMP experiment, Rosenberg described four

other research projects that qualified as experiments.40 Two of the

34 VOGT, supra note 18, at 230–31.

35 JOHN E. SHAPARD, LIKELY CONSEQUENCES OF AMENDMENTS TO RULE 68, FEDERAL

RULES OF CIVIL PROCEDURE 3–5 (1995).

36 Id. at 5.

37 See FED. JUDICIAL CTR., supra note 20, at 15 (“Experiments are designed to test

the cause-and-effect relationship [between two or more variables].”); VOGT, supra

note 18, at 35 (“Most people who use the term ‘causal conclusion’ believe that an

experiment . . . is the only design from which researchers can properly infer cause.”).

38 See VOGT, supra note 18, at 35, 184.

39 Rosenberg, Procedure-Impact Studies, supra note 3, at 14.

40 For other examples of experimental research and evaluations of programs in

the legal system, see id. at 17–19 (New Jersey experiment testing mandatory pretrial

conferencing for cost and quality differences); id. at 20 (Ontario experiment testing

mandatory pretrial settlement conference for impact on settlement rates); id. at

22–23 (Kentucky experiment with impact of judicial case management on the cost,

time, and quality of justice); and id. at 23–25 (North Carolina experiment assessing

impact of court-annexed arbitration program on cost, delays, and other factors).

Apparently using a restrictive definition of “true” experiments, Professor Walker

added two simulated laboratory experiments to the list. Walker, supra note 8, at 70 &

n.15 (citing Thomas D. Rowe, Jr. & Neil Vidmar, Empirical Research on Offers of Settlement: A Preliminary Report, LAW & CONTEMP. PROBS., Autumn 1998, at 13, 13); id. at

n.16 (citing SAUL M. KASSIN, AN EMPIRICAL STUDY OF RULE 11 SANCTIONS (1985)). For

some reason, Professor Walker did not classify his own laboratory experiments within

his definition of “true” experiments. See id. at 69 n.6. See generally JOHN THIBAUT &

LAURENS WALKER, PROCEDURAL JUSTICE: A PSYCHOLOGICAL ANALYSIS (1975) (reporting
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 11 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1131

studies involved state courts, and one involved a Canadian court.

None took place before 1960. Since 1988, the frequency of experimental field research appears to have increased somewhat but remains relatively infrequent in comparison with the nonexperimental

research approaches discussed in Part II.41

Significantly, most or all of the experimental field research reported above relates to testing case management approaches or programs, such as the use of alternative dispute resolution, pretrial

conferences, and appellate conferencing programs. None of the experimental studies focused on the workings of a single procedural

rule, such as the class action, discovery, or sanctions rules. While the

Rule 68 offer of judgment studies dealt with a single rule, that rule

works like a settlement program. Creating experiments to test rules

that are an integral part of the litigation process may raise issues that

do not occur when an entire program is applied to or withheld from

experimental and control groups. For example, to apply or not apply

the results of a series of laboratory experiments testing assumptions and hypotheses

related to adversarial and inquisitorial systems of justice). Perhaps the distinction is

that the latter set of experiments did not directly involve participants in the legal

system. For further discussion of the extent of laboratory experiments relating to the

legal system, see supra note 25.

41 A thorough account of experimental and nonexperimental research in relation to civil rulemaking would exceed the scope of this Article. Recent research products of the FJC and the RAND Corporation’s Institute for Civil Justice suggest that use

of experimental methods has increased since 1988 but remains a modest portion of

the rulemaking-related research described in Part II. Note that all of the following

research relates to broad case management programs, such as mediation and offers of

judgment, rather than specific procedures, such as motions, sanctions, or other components of the litigation process. See, e.g., EAGLIN, supra note 31; JAMES S. KAKALIK,

TERENCE DUNWORTH, LAURAL A. HILL, DANIEL MCCAFFREY, MARIAN OSHIRO, NICHOLAS

M. PACE & MARY E. VAIANA, AN EVALUATION OF MEDIATION AND EARLY NEUTRAL EVALUATION UNDER THE CIVIL JUSTICE REFORM ACT 21 (1996) (describing random assignment to experimental and control groups in the Southern District of New York and

the Eastern District of Pennsylvania); E. ALLAN LIND, ARBITRATING HIGH-STAKES CASES:

AN EVALUATION OF COURT-ANNEXED ARBITRATION IN A UNITED STATES DISTRICT COURT

(1990) (reporting results of experimental study of arbitration in a single district

court); SHAPARD, supra note 35, at 3–5 (describing survey design for attorneys involved

in federal litigation in all Federal districts); DONNA STIENSTRA, MOLLY JOHNSON & PATRICIA LOMBARD, REPORT TO THE JUDICIAL CONFERENCE COMMITTEE ON COURT ADMINISTRATION AND CASE MANAGEMENT: A STUDY OF THE FIVE DEMONSTRATION PROGRAMS

ESTABLISHED UNDER THE CIVIL JUSTICE REFORM ACT OF 1990, at 215–54 (1997)

(describing experimental field study of the CJRA Early Assessment [Mediation] Program in the Western District of Missouri); see also Joshua D. Rosenberg & H. Jay

Folberg, Alternative Dispute Resolution: An Empirical Analysis, 46 STAN. L. REV. 1487,

1492–95 (1994) (describing the random assignment method used to study early neutral evaluation program in the Northern District of California).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 12 28-MAY-02 12:35

1132 notre dame law review [vol. 77:4

a modified class action, sanctions, or a disclosure rule to every other

case seems to require intruding into the litigation process in an extraordinary manner and imposing novel demands on judges and litigators. In addition, concerns about ethical and legal fairness may

inhibit experimental research. Such concerns will be addressed in

Part IV.42

2. Nonexperimental Observational Research

Nonexperimental observational research designs call for a researcher to observe subjects, often in a natural setting.43 Most empirical legal research is of this type, whether the subjects be cases,

attorneys, or litigants. Specifying the empirical questions, formulating

research hypotheses, and creating written protocols add systematic

rigor to the enterprise. One type of observational research involves

examining legal documents or other legal proceedings and using a

research protocol to ask the same questions regarding each case file

or legal document. For example, in the FJC’s empirical study of special masters, researchers examined docket sheets in a sample of cases

in which the court appeared to have considered appointing a special

master. In those cases, researchers reviewed the docket sheets and

materials in the court’s file to determine whether an appointment was

made, under what authority, for what purpose, and so forth.44

Observational research differs from experimental research in that

any testing of the effect of a treatment does not take place under conditions that the researcher controls. For that reason, nonexperimental observational designs generally cannot yield findings of causal

relationships among variables. Differences in the outcomes of various

types of cases, for example, may be caused by variables unrelated to

the type of case, such as the location of the court, the experience of

the attorneys, and, typically, a host of variables that are difficult to

identify and control.

Observational research, however, under certain conditions may

be used to generate findings that apply to an entire population of

cases, judges, attorneys, or other participants in the legal system. If a

sample of subjects has been selected randomly from the population,

the researcher can use statistical methods to calculate the extent to

42 See also infra text accompanying notes 383–93.

43 See VOGT, supra note 18, at 197–98.

44 See, e.g., THOMAS E. WILLGING ET AL., SPECIAL MASTERS’ INCIDENCE AND ACTIVITY:

REPORT TO THE JUDICIAL CONFERENCE’S ADVISORY COMMITTEE ON CIVIL RULES AND ITS

SUBCOMMITTEE ON SPECIAL MASTERS 81–83 (2000) [hereinafter FJC SPECIAL MASTERS’

STUDY].
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 13 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1133

which the findings pertain to the population as a whole. For example,

in the special master study mentioned above, Center researchers selected a random sample of a national population of civil cases that

contained docket entries referring to special masters.45 Based on that

sample, the researchers were able to state with precise confidence

levels the estimated minimum and maximum rates at which federal

judges considered appointing a special master in various types of cases

during the study period.46

3. Case Studies

The case study represents another type of research design that

legal-empirical researchers sometimes use. The assumption is that

“the example (the ‘case’) is in some way typical of the broader phenomenon,”47 but that is only an assumption. In addition, it is difficult

to generalize a case study to the broader universe of cases.48

A case study should not be confused with the study of published

opinions in a given case. An empirical case study will generally involve

examining the documents in the litigation (including, but not limited

to, published opinions), interviewing attorneys and perhaps judges,

and examining all available sources of information about the case.49

Case studies can advance knowledge of the legal process by providing

an intensive and detailed look at the processes used by attorneys,

judges, and courts to manage litigation. Similarities and differences

in approaches uncovered during case studies might provide hypotheses that researchers can explore further by using quantitative research

designs.

Some examples of case studies in civil litigation include research

on class action litigation. In one such study, researchers selected a set

of ten state and federal class action cases on the basis of their case type

(mass tort or consumer cases involving small individual losses),

whether they had been certified and resolved as class actions, whether

they had been substantially terminated, and whether the participants

45 Id.

46 See id. at 17–21 (providing a table showing incidence by the nature of suit and

text discussing confidence intervals); see also id. at 83–92 (providing a statistical discussion and a table showing confidence intervals for all nature of suit classifications).

47 VOGT, supra note 18, at 34.

48 See id. at 34–35.

49 See, e.g., S. ELIZABETH GIBSON, CASE STUDIES OF MASS TORT LIMITED FUND CLASS

ACTION SETTLEMENTS & BANKRUPTCY REORGANIZATIONS 9–11 (2000) (describing the

methods used to study the cases selected); Linda S. Mullenix, Beyond Consolidation:

Postaggregative Procedure in Asbestos Mass Tort Litigation, 32 WM. & MARY L. REV. 475,

485–87 (1991) (discussing the use of comparative case studies in asbestos litigation).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 14 28-MAY-02 12:35

1134 notre dame law review [vol. 77:4

were willing to speak on the record with researchers.50 In another

study, the object was to compare bankruptcy and limited-fund class

action approaches to resolving mass tort litigation.51 In these studies,

the small number of cases would limit the power of statistical tests,

and/or the absence of a control group would preclude making causal

statements. Including comparable cases may add a systematic element

and allow the researcher to identify and discuss similarities and differences in policy-relevant variables, such as the approach used by the

court to review a proposed class settlement.

Case studies may in some instances be the only research design

available for examining a phenomenon that occurs infrequently. In

those circumstances, a series of case studies may represent a powerful

research design. For example, a series of case studies recently examined mass tort settlement class actions.52 At the outset of that

study, researchers had been able to identify only five recent class actions that could be described as mass tort settlement class actions.53

Accordingly, that study should be seen as presenting data about the

universe of mass tort settlement class actions during the 1990s and not

just about a representative sample of such cases.

B. Research Methods

1. Surveys

Typically, the method of choice in both experimental and observational studies is a written survey, using an instrument such as a questionnaire that is administered systematically to the research subjects,

who may be judges, lawyers, litigants, or other participants in the legal

system.54 Or, as mentioned above, the subject of the research may be

the case and its documents. Administering the survey may also take

the form of in-person interviews, telephone interviews, or examining

50 See HENSLER ET AL., supra note 16, at 138–39 (describing case selection for

study of ten consumer and mass torts class action cases).

51 See GIBSON, supra note 49.

52 See JAY TIDMARSH, MASS TORT SETTLEMENT CLASS ACTIONS (1998).

53 Id. at 20–24

54 See, e.g., Elizabeth C. Wiggins, Thomas E. Willging & Donna Stienstra, The Federal Judicial Center’s Study of Rule 11, FED. JUDICIAL CTR. DIRECTIONS, Nov. 1991, at 3,

28–35 [hereinafter FJC Rule 11 Study] (conducting a survey of judges’ experiences

with frivolous litigation and their opinions regarding Rule 11 sanctions); Thomas E.

Willging, Donna Stienstra, John Shapard & Dean Miletich, An Empirical Study of Discovery and Disclosure Practice Under the 1993 Federal Rule Amendments, 39 B.C. L. REV. 525,

528–29 (1998) [hereinafter FJC Disclosure & Discovery Study] (conducting a survey of

attorneys about experiences in terminated federal cases and opinions about proposed

reforms).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 15 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1135

the case file and documents. Administering a written questionnaire

may involve using electronic media, such as the Internet or facsimile

transmission, to distribute the survey and record responses.

2. Focus Groups

Judges and other legal policymakers often need reliable information about the needs and interests of litigants. Rarely do non-attorney

litigants participate in formal hearings on proposed rules changes. To

formulate hypotheses or to reach tentative qualitative conclusions

about a subject, researchers sometimes employ focus groups, collections of six to twelve individuals to discuss a particular subject through

a structured interview conducted by an experienced researcherfacilitator.55 For example, when the Advisory Committee asked the

Center to develop illustrative plain language notices for use in class

actions, the Center used focus group techniques to identify typical potential class members’ understandings of class actions,56 legal forms,

legal terminology, and other aspects of notice.57 Focus groups might

also be useful in gathering information about the impact of court procedures on litigants or jurors.58

3. Literature Reviews

One additional method deserves comment even though it does

not involve conducting original empirical work. A literature review

consists of a “systematic survey and interpretation of the research findings (the ‘literature’) on a particular topic. . . .”59 Providing background data on a subject that has been studied empirically in

laboratory experiments and otherwise, such as the effects of jury size

on jury decisionmaking, gives the policymaker the perspectives of social scientists without the need for a major study. For example, to

support the Advisory Committee’s recent examination of discovery,

55 See VOGT, supra note 18, at 114.

56 Participants were screened with the goal of obtaining a group that would have

the capacity to understand such notices and would be challenged by the task. For

example, each member of the group had no less than a high school education and no

more than a college education. Those who had been involved in litigation or were

working for a law firm were excluded.

57 For an overview of the Center’s work, see the FJC’s website, http://www.fjc.gov.

“Class Action Notices” information can be found at this website by accessing the “Current FJC Activities” page.

58 See generally RICHARD A. KREUGER & MARY ANNE CASEY, FOCUS GROUPS: A PRACTICAL GUIDE FOR APPLIED RESEARCH (3d ed. 2000) (presenting a guide and reference

manual for the use of focus groups in research purposes).

59 VOGT, supra note 18, at 163.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 16 28-MAY-02 12:35

1136 notre dame law review [vol. 77:4

the Center conducted a thorough review of empirical research on the

subject.60

C. Research Setting

Field research refers to research that takes place in a “naturally

occurring” setting “that is not a laboratory or library.”61 A field experiment is one type of field research, and observational research is often

also referred to as field research.

D. Sampling

In most types of empirical research, the goal is to study a sufficiently large sample of a population to enable the researcher to make

statistically sound inferences about the total population from which

the sample was drawn (for example, a “survey”).62 However, in some

observational field research, the researcher may need to select cases

or other objects of study other than randomly. For example, the random selection of cases might produce an insufficient number of opportunities to observe the particular object of interest within the time

and budget allocated for the research. Researchers refer to a sample

selected on the basis of available resources as a convenience sample to

distinguish it from a random sample. Field studies based on a convenience sample do not allow the researcher to generalize, that is, to

make statements about the entire population under consideration.

Often, the point of such empirical research is simply to observe and

describe systematically the phenomenon of interest. Such descriptive

research may also generate hypotheses for later testing.

The Center’s research on class actions is a good example of observational field research based on a convenience sample.63 The study

60 See Judith A. McKenna & Elizabeth C. Wiggins, Empirical Research on Civil Discovery, 39 B.C. L. REV. 785, 787 (1998). Another example is the Center’s literature

review summarizing both social science and legal materials for the Mass Tort Working

Group, which the Chief Justice created at the behest of the Advisory Committee. E.g.,

THOMAS E. WILLGING, APPENDIX C: MASS TORTS PROBLEMS AND PROPOSALS: A REPORT

TO THE MASS TORTS WORKING GROUP, in REPORT OF THE ADVISORY COMMITTEE ON

CIVIL RULES AND THE WORKING GROUP ON MASS TORTS TO THE CHIEF JUSTICE OF THE

UNITED STATES AND TO THE JUDICIAL CONFERENCE OF THE UNITED STATES, reprinted in

187 F.R.D. 328 (1999).

61 VOGT, supra note 18, at 111.

62 See, e.g., id. at 286.

63 See Thomas E. Willging, Laural L. Hooper & Robert J. Niemic, An Empirical

Analysis of Rule 23 To Address the Rulemaking Challenges, 71 N.Y.U. L. REV. 74 (1996)

[hereinafter NYU Empirical Analysis] (surveying class action case files in terminated

cases in four federal districts). The original version of the study, with a complete set
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 17 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1137

could have been based on a sample of class actions filed nationwide,

but considerable time, logistical support from clerks’ offices and computer systems administrators, travel, and personnel resources would

have been required to study cases spread across ninety-four federal

districts. Center researchers instead chose to examine cases in four

districts that exhibited high levels of class action activity. Recognizing

that these courts may not be representative—indeed, they were selected precisely because they showed the promise of having an uncharacteristically high number of class actions—the researchers had

to caution the reader that “[e]ach district should be viewed as a separate entity and the data from the four districts should be viewed as

descriptive—four separate snapshots of recent class action activity.”64

By focusing on four courts with high reported rates of filing, Center

researchers and the Advisory Committee traded the ability to generalize about class actions for the ability to collect data and report their

findings within approximately one year from the date the Advisory

Committee requested the study.

E. Measurement

Empirical research can be further divided into quantitative and

qualitative studies. The Center’s class action study described above

presents a good example of a quantitative observational field study.

Researchers examined approximately four hundred class action case

files in cases denominated as class actions that terminated within a

two-year period and systematically recorded information about numerous variables.65 Similarly, the Center conducted an extensive

study of Rule 11 sanctioning activity in five federal district courts that

were selected because their computerized dockets facilitated data collection on Rule 11 cases.66

Qualitative field research often uses the case study design to examine in depth the manifestation of a particular phenomenon in a

single case or set of cases. Interviewing judges, attorneys, and other

participants in the legal system is a typical method for eliciting qualitative information.

of figures, tables, and appendices was published as THOMAS E. WILLGING, LAURAL L.

HOOPER & ROBERT J. NIEMIC EMPIRICAL STUDY OF CLASS ACTIONS IN FOUR FEDERAL

DISTRICT COURTS: FINAL REPORT TO THE ADVISORY COMMITTEE ON CIVIL RULES (1996)

[hereinafter FJC EMPIRICAL STUDY OF CLASS ACTIONS].

64 NYU Empirical Analysis, supra note 63, at 84.

65 See FJC EMPIRICAL STUDY OF CLASS ACTIONS, supra note 63, at 180–85.

66 See FJC Rule 11 Study, supra note 54, at 2.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 18 28-MAY-02 12:35

1138 notre dame law review [vol. 77:4

In short, empirical researchers draw from a host of research approaches that can assist rulemakers in determining whether and how

to craft new rules to address identifiable problems. More than one

design and method can be used at a time. Often complementary

methods achieve different purposes. For example, in studying special

masters, the Center was asked to report on the incidence of the use of

special masters and to quantify the frequency with which special masters played various pretrial and post-trial roles.67 In addition to these

quantitative questions, the Committee was interested in knowing

more about the quality of special masters’ activities in performing

those roles.68 For the quantitative aspects of the study, surveying a

nationwide sample of the cases seemed appropriate; for the qualitative

aspects, interviews with judges and lawyers in a targeted subset of the

sample seemed more appropriate.69 The final product weaves together elements of survey and case study methods to present both a

quantitative overview of all cases and a closer qualitative look at selected cases.70

F. Commentators’ Calls for Research

In contrast to the variety of designs and methods available for

studying the legal system in action, empirically sophisticated commentators appear to have focused on a small subset of available designs

and methods. As noted in the Introduction, one commentator explicitly called for the Advisory Committee to perfect the civil rulemaking

process by adopting a “program of restricted field experiments” that

would enable the Committee “to predict the impact of proposed

changes to the Federal Rules of Civil Procedure.”71 Implementing

true experiments “in a small number of U.S. district courts” would,

according to this commentator, reduce the costs of rulemaking when

67 See FJC SPECIAL MASTERS’ STUDY, supra note 44, at 2–3, 15–17.

68 See id.

69 See id. at 13–15.

70 See id. at 2–3, 15–17. Along similar lines, the Center’s Rule 11 study combines

a quantitative field study in selected districts with a nationwide survey of judges’ opinions about the need for, and operation of, Rule 11. FJC Rule 11 Study, supra note 54,

at 2. The RAND study of class actions combines quantitative estimates of the incidence of class actions with case studies and field interviews. See HENSLER ET AL., supra

note 16, at 49–68, 138–39.

71 Walker, supra note 8, at 67. Professor Walker uses the term “restricted” in a

geographical sense to make clear that such experiments need not be conducted on an

unrestricted or national basis. See id. at 75–76. The proposal would not apply to

“amendments intended only to clarify existing Rules.” Id. at 76.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 19 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1139

one takes into account “the cost of subjecting persons to rules of dubious quality.”72

Another commentator decried the “ignorance” of the rulemaking bodies, specifically their “studied indifference to empirical questions.”73 While less explicit or universal about the research designs to

be employed, this commentator advocated “empirical evidence on the

operation of the Rules or proposed amendments” and put a premium

on knowledge that “concerns alternative reform strategies and their

likely impact.”74 In 1995, the Standing Committee’s Subcommittee

on Long Range Planning recommended that “[e]ach Advisory Committee should ground its proposals on available data and develop

mechanisms for gathering and evaluating data that are not otherwise

available, and should use these data to decide whether changes in existing rules should be proposed.”75

Amended Rule 11 (the 1983 version) and amended Rule 26(a)

(the 1993 version) were the culprits that led to Professor Burbank’s

call for a moratorium on rulemaking.76 Each of those amendments

dramatically revised national rules and precipitated major changes in

civil litigation. With no equivalent national rules in operation at the

time of the Rule 26(a) amendment,77 his call for a study of the impact

of the proposed rule amounts to a call for experimental field research. This is true because the only direct way to study a rule’s impact is to put it into effect, either on an experimental basis or across

the board.

In either the Rule 11 or Rule 26(a) contexts, conducting a controlled field experiment of the type described earlier in this Article

72 Id. at 76.

73 Burbank, supra note 6, at 841; see also Thomas E. Baker, An Introduction to Federal Court Rulemaking Procedure, 22 TEX. TECH L. REV. 323, 335 (1991) (advocating that

“more should be done to encourage and to utilize empirical work in judicial

rulemaking”).

74 Burbank, supra note 6, at 841–42.

75 SELF-STUDY, supra note 12, at 699. The Long Range Plan adopted by the Judicial Conference did not expressly comment on the use of empirical research in the

rulemaking process. The report recommended that “[r]ules should be developed

exclusively in accordance with the time-tested and orderly process established by the

Rules Enabling Act.” JUDICIAL CONFERENCE OF THE U.S., LONG RANGE PLAN FOR THE

FEDERAL COURTS, reprinted in 166 F.R.D. 53, 118 (1995).

76 See generally Burbank, supra note 6 (discussing rulemaking in the context of

amendments to Rules 11 and 26(a)).

77 Three federal district courts had on their own initiative adopted local rules

providing for disclosure in civil cases. See Mullenix, supra note 6, at 813–20.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 20 28-MAY-02 12:35

1140 notre dame law review [vol. 77:4

would be difficult if not impossible.78 Ideally, studying the impact of

an inchoate proposed rule would require experimental testing of the

proposal in districts that are randomly selected, not self-selected. Developing reliable information concerning the impact of the proposed

sanctions and initial disclosure rules would seem to have depended on

adopting one or both of the rules in one or more districts on an experimental basis. A control group using the standard procedure in

the same districts would have been needed to provide a basis for comparison. To remove the effect of variation among judges, each judge

would have to have been asked to apply both experimental and control procedures to cases selected randomly, without knowing the merits or procedural context.

One could, of course, have surveyed judges and lawyers and have

asked them to predict the impact of the proposed rule on hypothetical future cases, but such responses would necessarily have produced

subjective responses with limited internal and external validity. In the

context of initial disclosure, studying an analogue of the proposed

rule in a local district or a state court might be the most one could

hope for.79

Thus, there seems be a disconnect between the calls for empirical

research and the range of designs available to researchers. Commentators have concentrated their efforts at prodding the civil rulemakers

to undertake experimental studies of proposed amendments to the

Federal Rules of Civil Procedure.80 This is not to say that these commentators rigidly adhere to a call for controlled experimental work in

all contexts. Each has also shown an appreciation for empirical research designs that are not experimental81 and would undoubtedly

78 See supra text accompanying notes 20–39 (describing a controlled field

experiment).

79 See Mullenix, supra note 6, at 813–20 (discussing three district courts that implemented local rules requiring informal discovery before formal discovery).

80 See, e.g., Stephen B. Burbank & S. Jay Plager, Foreword: The Law of Federal Judicial

Discipline and the Lessons of Social Science, 142 U. PA. L. REV. 1 (1993); Linda S. Mullenix, The Pervasive Myth of Pervasive Discovery Abuse: The Sequel, 39 B.C. L. REV. 683

(1998) (relying on nonexperimental empirical research); Laurens Walker, A Comprehensive Reform for Federal Civil Rulemaking, 61 GEO. WASH. L. REV. 455 (1993) [hereinafter Walker, Comprehensive Reform]; Laurens Walker, Avoiding Surprise from Federal Civil

Rule Making: The Role of Economic Analysis, 23 J. LEGAL STUD. 569 (1994) [hereinafter

Walker, Avoiding Surprise].

81 See, e.g., Burbank & Plager, supra note 80, at 16 (1993) (finding a descriptive

quantitative and qualitative field study to be “the first rigorous study of the experience

under the 1980 Act”); id. at 19 (finding the “number and variety of methods employed were impressive” even though none were experimental); Mullenix, supra note

80, at 683 (1998) (relying on nonexperimental empirical research); Walker, Compre-
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 21 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1141

find the use of nonexperimental empirical research approaches to be

an improvement over then-current practices. Nonetheless, at least

one commentator quite explicitly has posited experimental research

designs as the gold standard for rulemakers to employ.82 We now

turn to a description of the empirical research that rulemakers have

commissioned and used in the last decade or so.

II. WHAT TYPES OF EMPIRICAL RESEARCH HAVE THE CIVIL

RULEMAKERS SOUGHT AND USED, AND HOW OFTEN?

The last decade or so has witnessed dramatic changes in the

rulemaking process. The 1988 amendments to the Rules Enabling

Act required that Advisory Committee and Standing Committee meetings “shall be open to the public,”83 after “sufficient notice.”84 Responding to recommendations from judges and scholars that “rules

changes be predicated on a sounder empirical basis,” the advisory

committees increased “their requests for assistance from the Federal

Judicial Center to conduct research on litigation practices and the impact of the rules.”85 As this Part reveals, the Advisory Committee on

Civil Rules, in particular, has established a pattern of continuing consultation with the FJC and other empirical researchers about empirihensive Reform, supra note 80, at 476–89 (1993) (calling for rulemaking to obtain adequate information to assess the costs and benefits of proposed rules changes by using

social science methods, apparently not limited to controlled experiments); Walker,

Avoiding Surprise, supra note 80, at 593 (1994) (suggesting as an adequate interim

measure that “economic analysts could furnish an acceptable basis for civil rule making by grounding their generalizations on systematic observations, a standard less demanding than hypothesis testing”).

82 Walker, Avoiding Surprise, supra note 80, at 593 (suggesting that “the committee

should either conduct limited field experiments incorporating the proposed changes,

or the committee should wait for change until private research ventures can be

completed”).

83 Rules Enabling Act of 1988, Pub. L. No. 100-702, 102 Stat. 4648, (codified at 28

U.S.C. § 2073(c)(1) (1994)).

84 Id. § 2073(c)(2).

85 Peter G. McCabe, Renewal of the Federal Rulemaking Process, 44 AM. U. L. REV.

1655, 1680 (1995). Mr. McCabe, Assistant Director for Judges Programs, Administrative Office of the United States Courts, and Secretary to the Judicial Conference

Standing Committee, listed five areas in which the Advisory Committee requested FJC

research: a study of Rule 11 before the 1993 amendments, a study on “the use and

operation of protective orders under Rule 26(c), offers of settlement under Rule 68,

consensual settlement of class actions under Rule 23, and the effect of mandatory

disclosure under the 1993 amendments to Rule 26. Id. at 1680–81. He also noted

that the Advisory Committee on Criminal Rules “considered the results of the Federal

Judicial Center’s study on cameras in the courtroom before approving amendments

to Rule 53.” Id. at 1681.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 22 28-MAY-02 12:35

1142 notre dame law review [vol. 77:4

cal questions.86 Such consultations occur before the Committee

proposes rules changes, while it reviews and hears comments on proposals that have been made, and while it deliberates about those proposals. Alongside the increased openness and interest in empirical

findings, some experienced participants in the process expressed concern, perhaps even alarm, that “the openness of the process . . . may

encourage factional politics.”87 This Part presents research studies;

Part III discusses patterns observed in those research studies and also

considers aspects of the relationship between rulemaking and legislative activity.

Table 1 lists research projects conducted by the Center at the request of the Advisory Committee, including the rule in question, the

type of research conducted, the date of the request, and the date of

the final report. This Part then summarizes the background and purpose of each project and describes links between research findings

and proposed rule changes. Relying primarily on official Committee

notes and published minutes, the discussion concentrates on links between findings and proposals that the rulemakers identified in the

course of their deliberations. Each project is presented in the same

format. Note that a considerable amount of Center research dealing

with procedural matters lies beyond the scope of this Article because

it was not the subject of a request by the Advisory Committee (though

often the subject of a request by another Judicial Conference

committee).88

86 See, e.g., infra notes 89, 101, 119, 149.

87 Carrington, supra note 1, at 165; see also Mullenix, supra note 6, at 830–36.

88 See, e.g., JOE S. CECIL & THOMAS E. WILLGING, COURT-APPOINTED EXPERTS: DEFINING THE ROLE OF EXPERTS APPOINTED UNDER FEDERAL RULE OF EVIDENCE 706

(1993); LAURAL L. HOOPER, JOE S. CECIL & THOMAS E. WILLGING, NEUTRAL SCIENCE

PANELS: TWO EXAMPLES OF PANELS OF COURT-APPOINTED EXPERTS IN THE BREAST IMPLANTS PRODUCT LIABILITY LITIGATION (2001); MOLLY TREADWAY JOHNSON & CAROL

KRAFKA, ELECTRONIC MEDIA COVERAGE OF FEDERAL CIVIL PROCEEDINGS (1994); MCKENNA ET AL., supra note 31; ROBERT J. NIEMIC, MEDIATION AND CONFERENCE PROGRAMS

IN THE FEDERAL COURTS OF APPEALS (1997); ELIZABETH PLAPINGER & DONNA STIENSTRA,

ADR AND SETTLEMENT IN THE FEDERAL DISTRICT COURTS (1996).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 23 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1143

TABLE 1. TIMING AND TYPE OF RESEARCH REQUESTED OF THE FEDERAL

JUDICIAL CENTER BY THE ADVISORY COMMITTEE ON CIVIL RULES

Federal Civil Date of Date of

Rule Involved Research Approaches Request Report Est. Yrs.

12(b)(6): Observational field study reviewing a random November Spring 0.5

Motion To sample of terminated 1985 cases in two 1988 1989

Dismiss for district courts and comparison with studies

Failure To conducted in 1975 and 1978

State a Claim

56: Summary Observational field study based on random 1989–1990 1991 1.5

Judgment sample of docket entries and filings in six (est.)

district courts selected for convenience,

comparing rates and activities with data from

1975 and 1986 studies

11: Signing Observational field study reviewing docketed January May 1991 1.5

of Pleading; Rule 11 activity in terminated cases in five 1990

Sanctions district courts selected for convenience

11: Signing Observational field study surveying all district Spring May 1991 1.0

of Pleading; judges regarding their experiences with Rule 1990

Sanctions 11 sanctions

11: Signing Observational field study surveying a sample Spring Summer 0.3

of Pleading; of attorneys and district judges regarding 1995 1995

Sanctions their experiences with the 1993 amendments

to Rule 11

26(c): Observational field study surveying activity November April & 1.5–2.0

Protective identified from electronic docket records in 1992 October

Orders three district courts selected for convenience 1994

68: Offer of Quasi-experimental field study surveying October April 1994 0.5–1.5

Judgment attorneys in four sets of two hundred cases 1993 & 1995

(contract, tort, civil rights, other), one

hundred of which settled and one hundred

of which went to trial

47(a): Observational field study surveying a random Spring October 0.5

Attorney Voir sample of one hundred and fifty district 1994 1994

Dire judges; comparison with similar FJC survey

conducted in 1976–1977

23: Class Observational field study surveying all April Fall 1995 1.5

Actions identifiable class actions terminated in a 1994

convenience sample of four districts during

a two year period between 1994 and 1996

23(c)(1): Observational field study of pilot group of March August 1.5

Class Action nonlawyer FJC employees; four focus groups 2000 2001

Notices composed of persons with high school or

college education

26–37: (1) Observational field study surveying 2000 October September 1.0

Discovery attorneys in a random national sample of 1996 1997

1000 closed civil cases;

(2) Literature review of empirical studies

53: Special (1) Observational field study surveying October October 1.0–1.5

Masters docket entries for a 136 case sample 1998 1999 &

randomly generated from a national April 2000

population of approximately 445,000

electronic docket sheets;

(2) Observational field study interviewing

judges and attorneys in a non-randomly

selected subset of the above sample
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 24 28-MAY-02 12:35

1144 notre dame law review [vol. 77:4

Federal Civil Date of Date of

Rule Involved Research Approaches Request Report Est. Yrs.

26, 34, and Observational field study surveying October Pending N/A

37: convenience sample of magistrate judges to 1999

Electronic identify candidates for approximately twenty

Discovery civil case studies of cases with significant

discovery of computer-based evidence; case

studies to include analysis of filed documents

and interviews of judges and attorneys

23: Class (1) Observational field study surveying December Pending N/A

Actions docket sheet activity to determine rate of 2001

(Effects of class action filings before and after the

Amchem and Amchem and Ortiz decisions;

Ortiz) (2) Observational field study surveying

attorneys in class actions to determine the

perceived effects of Amchem and Ortiz

A. Rule 12(b)(6) Study

1. Committee Request

The Advisory Committee requested this study at about the time

commentators began to call attention to the need for empirical work

to support the rulemaking process.89 In November 1988, the Committee reviewed and discussed a draft proposal to abrogate Rule

12(b)(6).90 The purpose of the proposed change was to reduce or

eliminate motions that were unproductive in the sense of not contributing to the disposition of the case.91 One commentator had opined,

tongue-in-cheek, that Rule 12(b)(6) “was last effectively used during

the McKinley administration” and that “it is in a sense a revolving

door device, rarely dispositive.”92 In the reporter’s note accompanying the proposed abrogation of Rule 12(b)(6), Professor Carrington

89 See generally Symposium, Empirical Studies of Civil Procedure, Part I, LAW & CONTEMP. PROBS., Summer 1988, at 1 (reviewing the contemporary condition of the empirical study of civil procedure); Symposium, Empirical Studies of Civil Procedure, Part II,

LAW & CONTEMP. PROBS., Autumn 1988, at 1 (reporting on more recent empirical

studies of particular procedural devices, and considering the relation between empirical study and substantive consequences of procedure). For an overview of the articles

in both parts of the symposium, see Paul D. Carrington, Foreword: The Scientific Study of

Legal Institutions, LAW & CONTEMP. PROBS., Summer 1988, at 1, 1–11; see also Rosenberg, Procedure-Impact Studies, supra note 3, at 17–29 (emphasizing the importance of

empirical research on the functioning of procedural rules); Walker, supra note 8, at

72–77 (emphasizing the need for a systematic plan for data collection to provide information for guiding the future development of the rules).

90 See THOMAS E. WILLGING, USE OF RULE 12(B)(6) IN TWO FEDERAL DISTRICT

COURTS 1 (1989).

91 Id. at 2.

92 ARTHUR R. MILLER, THE AUGUST 1983 AMENDMENTS TO THE FEDERAL RULES OF

CIVIL PROCEDURE: PROMOTING EFFECTIVE CASE MANAGEMENT AND LAWYER RESPONSIBILITY 8 (1984).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 25 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1145

suggested that the Committee “consider whether additional data

might be gathered that would illuminate the questions raised by the

proposal.”93 The Committee in turn asked the FJC to conduct such a

study.

2. Research Methods

FJC researchers examined docket sheets and case files in approximately three hundred terminated cases in each of two federal district

courts: the District of Maryland and the Eastern District of Pennsylvania.94 The courts were selected based in part on convenience95

and not as representative of all district courts.

3. Findings and Linkage with Rules Proposals

In those two courts, Rule 12(b)(6) motions led to final disposition of three percent of all cases, a decline from the six percent rate

found in 1975 in those same courts.96 Though not a large percentage

of the cases, the number of cases was sufficient to persuade the Committee that Rule 12(b)(6) continued to serve a useful purpose. The

Committee considered the Center’s data at its April 1989 meeting and

decided not to change Rule 12(b)(6).97

B. Rule 56 Summary Judgment Study

1. Background

During the early and mid 1980s, commentators expressed concern that Rule 56, as interpreted, might be inhibiting the utility of

summary judgment because of uncertainty about how Rule 56 should

be applied.98 Whether the 1986 Supreme Court decisions in a trilogy

93 WILLGING, supra note 90, at 2.

94 Id. at 7.

95 In addition to their geographical convenience, these two courts had been part

of an earlier FJC study of motions practice. See PAUL R.J. CONNOLLY & PATRICIA A.

LOMBARD, JUDICIAL CONTROLS AND THE CIVIL LITIGATIVE PROCESS: MOTIONS 4 n.10

(1980). Having baseline data on Rule 12(b)(6) motions enabled the researchers to

compare results within the same two courts over a period of time. See WILLGING, supra

note 90, at 5–7.

96 See WILLGING, supra note 90, at 9. The study also found that Rule 12(b)(6)

motions led to the disposition of individual defendants in an additional two percent

of the cases. Id. at 8.

97 See id. at 3.

98 See, e.g., JOE S. CECIL & C. R. DOUGLAS, SUMMARY JUDGMENT PRACTICE IN THREE

DISTRICT COURTS 1 (1987); William W Schwarzer, Summary Judgment Under the Federal

Rules: Defining Genuine Issues of Material Fact, reprinted in 99 F.R.D. 465 (1984).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 26 28-MAY-02 12:35

1146 notre dame law review [vol. 77:4

of cases cured that perceived inhibitory effect was of concern.99 If

those decisions had not facilitated summary judgment practice, the

case for revising Rule 56 might be strengthened.100

2. Committee Request

Around 1989 or 1990, building on an earlier request for data that

predated the 1986 Supreme Court decisions,101 the Committee asked

the Center for data on trends in the filing, disposition, and appeal of

summary judgment motions.

3. Research Methods

Center researchers examined docket entries and filings in six federal district courts. Three districts were selected because they had

been included in previous studies; the other three districts were chosen because of the perception that they restricted summary judgment

practice.102

4. Findings and Linkage with Rules Proposals

The principal finding was that the filing of summary judgment

motions in nonprisoner civil cases increased between 1975 and 1986,

but did not exhibit any statistically significant increases between 1986

and 1989.103 Without expressly referring to these findings, the Committee proceeded to publish a proposed rule revision in August

1990.104 As stated in the proposed Committee notes, the purpose of

the proposed changes was

to enhance the utility of the summary judgment procedure as a

means to avoid the time and expense of discovery, preparation for

trial, and trial itself as to matters that . . . can have but one outcome—while at the same time assuring that parties are not deprived

of a fair opportunity to show that a trial is needed to resolve such

matters.105

99 CECIL & DOUGLAS, supra note 98, at 2–3.

100 See id.

101 The product of that request was the study by Cecil & Douglas. Id.

102 Joe. S. Cecil, Trends in Summary Judgment Practice: A Summary of Findings, FED.

JUDICIAL CTR. DIRECTIONS, Apr. 1991, at 11, 11.

103 See id. at 12–13.

104 See COMM. ON RULES OF PRACTICE AND PROCEDURE OF THE JUDICIAL CONFERENCE

OF THE U.S., PRELIMINARY DRAFT OF PROPOSED AMENDMENTS TO THE FEDERAL RULES OF

CIVIL PROCEDURE AND THE FEDERAL RULES OF EVIDENCE, reprinted in 137 F.R.D. 53, 141

(1990).

105 Id., reprinted in 137 F.R.D. 53, 146.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 27 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1147

5. Aftermath

Proposed changes in Rule 56 were recommended by the Advisory

Committee and forwarded to the Judicial Conference by the Standing

Committee. The Judicial Conference, however, rejected the proposal

at its September 1992 meeting. Judge Keeton, the chair of the Standing Committee reported that

some members of the Conference had argued that the summary

judgment rule was working well in its present form and that judges

had become familiar with the language of the rule and the current

case law. [He also] detected a criticism by some Conference members that too many changes were being proposed in the rules.106

Judge Pointer, the chair of the Advisory Committee on Civil

Rules, added that “some members seemed not to like the case law on

Rule 56 and might not have wanted to enshrine it in the rule.”107

C. Rule 11 Sanctions Study

1. Background

From the time of its adoption in 1983, revised Rule 11 raised an

intense controversy about the exact nature of the duty imposed by the

amended rule, and whether the rule was being applied in a way that

would chill creative advocacy, have a disproportionate impact on civil

rights litigants, or generate extensive satellite litigation.108 Much of

the controversy appeared to arise from amended Rule 11’s mandate

that the court faced with a violation of the rule “shall impose . . . an

appropriate sanction.”109

Motivated by the controversy and by conflicting interpretations of

the operation of Rule 11,110 Center researchers undertook two studies

of Rule 11 before being asked by the Advisory Committee to conduct

additional studies. The first Center-generated study examined the

purposes and standards that federal judges brought to the administration of amended Rule 11. Social psychologist Saul Kassin employed

106 Comm. on Rules of Practice and Procedure of the Judicial Conference of the

U.S., Meeting Minutes, Dec. 17, 1992, 1992 WL 739926, at *1.

107 Id.

108 For an extensive bibliography of the books, monographs, and articles discussing Rule 11, see COMM. ON THE RULES OF PRACTICE AND PROCEDURE OF THE JUDICIAL

CONFERENCE OF THE U.S., CALL FOR COMMENTS, reprinted in 131 F.R.D. 335, 350–59

(1990) [hereinafter CALL FOR RULE 11 COMMENTS].

109 FED. R. CIV. P. 11.

110 See THOMAS E. WILLGING, THE RULE 11 SANCTIONING PROCESS 15 (1989) (stating that “the issues [surrounding Rule 11] have evolved from automatic support or

opposition to the rule to more refined questions about its operation”).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 28 28-MAY-02 12:35

1148 notre dame law review [vol. 77:4

an experimental design in which he presented ten hypothetical scenarios based on ten recently published cases to sitting district judges.111

Overall, Kassin found “a good deal of interjudge disagreement over

what actions constitute a violation of the rule, only partial compliance

with the desired objective standard,” and “a continued neglect of alternative, nonmonetary” sanctions.112 He also found that a “clear majority of respondents believed that deterrence is the primary purpose

of Rule 11 sanctions” and that “judges who embrace a compensatory

rationale are the most likely to grant a Rule 11 motion for attorneys’

fees.”113

The second Center-generated Rule 11 study was a primarily qualitative field study, based on interviews with judges and attorneys regarding Rule 11 sanctions proceedings in which they had been

involved.114 Among other findings, that study reported that “little evidence was found that [Rule 11] sanctions have a chilling effect on

creative advocacy or unpopular causes”;115 that “[s]atellite litigation

occurs primarily in cases involving large compensatory sanctions

awards”;116 that “legitimate complaints from lawyers about surprise

and lack of due process can be accommodated without elaborate satellite evidentiary hearings”;117 and that “Rule 11 has begun to achieve

its goal of deterring frivolous filings.”118

2. Committee Request

In 1990, the Advisory Committee initiated a thorough review of

Rule 11. As one component of its review process, the Committee

asked the Center to conduct an empirical study of the current operation of the rule. The Committee also issued a public call for comments in which it asked, among other things, for information about

the amount of satellite litigation generated by Rule 11, the extent to

which Rule 11 activity has been concentrated in specific types of cases

or on particular types of litigants, and the amount of judicial variation

in sanctioning practices.119

111 SAUL M. KASSIN, AN EMPIRICAL STUDY OF RULE 11 SANCTIONS 11–15 (1985).

112 Id. at xi.

113 Id. at x.

114 See WILLGING, supra note 110, at 17.

115 Id. at 8.

116 Id. at 3.

117 Id. at 7.

118 Id. at 11.

119 See CALL FOR RULE 11 COMMENTS, supra note 108, reprinted in 131 F.R.D. 335,

345–50.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 29 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1149

3. Research Methods

The Center conducted an observational field study of Rule 11 activity in five federal district courts by identifying cases with references

to Rule 11 or sanctions in the court’s electronic docketing system—a

compilation of all the docket entries for all pending and recently

closed cases. Researchers examined and coded the files in each district, producing five separate snapshot-like descriptions of Rule 11 activity in those five districts.120

The Center also conducted a survey of all district judges about

their experiences and opinions relating to Rule 11. The Center reported to the Advisory Committee the results of both studies in May

1991.

4. Findings and Linkage with Rules Proposals

On June 13, 1991, the new chair of the Advisory Committee, District Judge Sam C. Pointer, Jr., in a memorandum to the chair of the

Standing Committee, indicated that the FJC study, written comments,

public testimony, and various articles and reports lend support to the

propositions, among others, that Rule 11:

(1) “has tended to impact plaintiffs more frequently and severely

than defendants,”

(2) “has too rarely been enforced through non-monetary sanctions,” and

(3) has caused litigants and judges to spend a “not . . . insignificant

amount of time to deal with Rule 11 motions, the great majority

of which were not granted.”121

The above findings each have counterparts in specific findings

detailed in the FJC study.122 The Committee’s findings, in turn, can

be linked with specific amendments. This is not to assert that the

Center’s findings are the sole cause or even a primary cause of related

amendments to the rules. The Committee had multiple sources, in

addition to the Committee members’ own experiences with and opin120 See FJC Rule 11 Study, supra note 54, at 2.

121 Id. at 4–5 (quoting Letter from the Honorable Sam C. Pointer, Jr. to the Honorable Robert E. Keeton, Chairman, Standing Committee (June 13, 1991) (as revised

after the Standing Committee’s July 18–20 meeting)).

122 See id. at 3–4. In addition, the finding on non-monetary sanctions may have

been linked with Saul Kassin’s finding that a “clear majority of [judge] respondents

believed that deterrence is the primary purpose of rule 11 sanctions,” KASSIN, supra

note 111, at x, and that disparities in treatment of the same case may have reflected

differences in judges’ “beliefs about the purposes served” by imposing sanctions. Id.

at 29–32.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 30 28-MAY-02 12:35

1150 notre dame law review [vol. 77:4

ions about Rule 11. In many, if not all, instances, the Center’s findings were corroborated by the testimony of witnesses at a public

hearing, by other empirical studies,123 or by other sources.

The Committee’s overall findings relating to the 1983 rule’s impact on plaintiffs appear to have directly supported the 1993 amendments to Rule 11—specifically the reference in Rule 11(b)(2) to

“claims, defenses, and other legal contentions”124 and the listing in

Rule 11(c)(1)(A) of “the challenged paper, claim, defense, contention, allegation, or denial.”125 Also, the addition of a separate section

(amended Rule 11(b)(4)) to deal with denials of factual contentions

seems attributable to the finding that Rule 11 sanctions were more

frequently sought against and imposed on plaintiffs.

The Committee’s finding regarding non-monetary sanctions led

to amended Rule 11(c)(2) which imposes a number of constraints on

awarding monetary sanctions in the form of attorneys’ fees and costs

to the party seeking sanctions.126 Most importantly, Rule 11(c)(2)

clarified that deterrence, not compensation, was the primary purpose

of the rule.

The Committee’s finding regarding the time required for satellite

litigation has a direct counterpart in the safe harbor created under

Rule 11(c)(1) for litigants who withdraw contentions after receiving

notice of a proposed Rule 11 motion.127

123 See, e.g., STEPHEN B. BURBANK, RULE 11 IN TRANSITION: THE REPORT OF THE

THIRD CIRCUIT TASK FORCE ON FEDERAL RULE OF CIVIL PROCEDURE 11 (1989) (reporting the findings and recommendations of the task force on the actual implementation of Rule 11); Lawrence C. Marshall, Herbert M. Kritzer & Frances Kahn Zemans,

The Use and Impact of Rule 11, 86 NW. L. REV. 943, 985–86 (1992) (revealing that research supported by American Judicature Society is substantially similar to that of the

Center).

124 FED. R. CIV. P. 11(b)(2).

125 Id. 11(c)(1)(A). Before the 1993 amendments, Rule 11 referred in general

terms to “a pleading, motion, or other paper.” FED. R. CIV. P. 11(c)(1)(A), reprinted in

28 U.S.C.A. FEDERAL RULES OF CIVIL PROCEDURE RULES 1–11 (West 1992) (Rule

amended 1993).

126 Fed. R. Civ. P. 11(c)(2); see also BURBANK, supra note 123, at 36–45 (discussing

the findings on sanctions).

127 See FED. R. CIV. P. 11(c)(1)(A). Judge Pointer made that connection in his

testimony to Congress in support of the 1993 amendments to Rule 11: “[W]e believe

the FJC studies amply support our conclusion that there has been an excessive and

unproductive amount of Rule 11 activity. To be sure, the ‘safe harbor’ will reduce the

risks to a litigant for initially including a questionable claim or defense.” Proposed

Amendments to the Federal Rules of Civil Procedure: Hearings Before the Subcomm. on Courts

and Administrative Practice of the Senate Comm. on the Judiciary, 103d Cong. 9 (1993)

[hereinafter Hearings] (testimony of the Hon. Sam C. Pointer, Jr.).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 31 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1151

In addition to the findings mentioned above, the Center found

that the “majority of those targeted by Rule 11 activity had an opportunity to oppose the imposition of sanctions; however, judges sometimes

failed to provide procedural safeguards when acting sua sponte.”128

The amended rule addressed that deficiency by adding a requirement

of “notice and a reasonable opportunity to respond” as a precondition

to imposing sanctions.129

The Center’s survey of judges led to findings that both supported

and undermined the Committee’s proposed amendments.130 Judge

Pointer did not directly cite findings based on the survey. We summarized the findings in these terms:

Most judges find the rule moderately effective as a deterrent but

have found other case management devices more useful in deterring groundless litigation. In addition, a sizeable minority have

seen some negative impact on the conduct of litigation. Yet a great

majority of judges believe that overall Rule 11 has had a positive

effect on litigation in the federal courts and wish to retain the 1983

language of the rule.131

The Center’s report on the survey cautioned the reader to

“[r]ecall that the judges were surveyed before the Advisory Committee

proposed the new revisions, so this last finding does not reflect the

judges’ preference for the 1983 language over the proposed revised

language.”132 In addition, the survey found that “[m]ost judges do

not find groundless litigation to be a problem in counseled cases in

their districts.”133 On the crucial point of whether Rule 11 should

remain mandatory or be made discretionary, survey results provided

ambiguous answers. One could emphasize survey findings that supported the rulemakers’ efforts to deal with satellite litigation by loosening the mandatory aspects of Rule 11, or one could emphasize

findings that supported retaining the status quo. The Standing Committee decided to loosen the standard by changing the operative language of Rule 11(c) from “shall” to “may.” The Advisory Committee

had proposed retaining the mandatory term “shall.”134

128 Hearings, supra note 127, at 3.

129 FED. R. CIV. P. 11(c).

130 See FJC Rule 11 Study, supra note 54, at 28.

131 Id.

132 Id.

133 Id.

134 REPORT ON ISSUES AND CHANGES TO FEDERAL RULES OF CIVIL PROCEDURE FROM

SAM C. POINTER, JR., CHAIRMAN ADVISORY COMMITTEE ON CIVIL RULES, TO HONORABLE

ROBERT E. KEETON, CHAIRMAN STANDING COMMITTEE ON RULES OF PRACTICE AND PROCEDURE (May 1, 1992), reprinted in 146 F.R.D. 519, 522–25 (1993).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 32 28-MAY-02 12:35

1152 notre dame law review [vol. 77:4

In dissenting from the Supreme Court’s transmittal of the 1993

amendments to Rule 11 to Congress, Justice Scalia cited the FJC survey as showing that “80% of the district judges believe Rule 11 has had

an overall positive effect and should be retained in its present

form.”135 In testifying before Congress in support of the proposed

amendments, Judge Pointer referred to Justice Scalia’s opposition and

referred to the FJC survey as finding that “the great majority of district

judges believe that Rule 11 . . . has been a valuable tool, albeit less

effective than some of the other management techniques available to

the courts.”136

5. Outcome and Aftermath

Amended Rule 11 went into effect on December 1, 1993. To reverse some of the amendments to Rule 11, the House of Representatives included provisions to that effect in the Attorney Accountability

Act of 1995, which it passed.137 That Act would have made Rule 11

sanctions mandatory, required that sanctions be adequate to compensate injured parties, eliminated the “safe harbor,” and made Rule 11

applicable to discovery.138 In passing the bill, the House subcommittee responsible for drafting it relied heavily on the same portions of

the FJC survey of judges that Justice Scalia had cited.139 Dissenting

members of the House asserted that “[b]ecause it is of such recent

vintage, the Federal Judicial Center has not had time to study how the

revised Rule 11 is working.”140

After the House’s action in 1995, the Advisory Committee asked

the Center to update its findings on Rule 11 and to “elicit judges’

current views based on their experience with the 1993 amendments.”141 The Center identified representative samples consisting of

148 federal district judges and 1130 federal trial attorneys. Questionnaires solicited opinions about the major provisions of the Attorney

135 Amendments to the Federal Rules of Civil Procedure, 146 F.R.D. 507, 509

(1993) (Scalia, J., dissenting).

136 Hearings, supra note 127, at 9.

137 H.R. 988, 104th Cong. (1995).

138 See id. § 4. Rule 11(d) excluded discovery activity from the terms of Rule 11,

leaving discovery sanctions to be governed by the discovery rules, including Rule 37.

FED. R. CIV. P. 11(d) advisory committee’s note.

139 See supra text accompanying notes 135–37; see also H.R. REP. NO. 104-62, at

9–10 (1995).

140 H.R. REP. NO. 104-62, at 9–10.

141 JOHN SHAPARD, GEORGE CORT, MARIE CORDISCO, THOMAS WILLGING, ELIZABETH

WIGGINS & KIM MCLAWIN, REPORT OF A SURVEY CONCERNING RULE 11, FEDERAL RULES

OF CIVIL PROCEDURE (1995) [hereinafter FJC 1995 RULE 11 SURVEY].
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 33 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1153

Accountability Act of 1995 as described above. Responses “suggest

that a majority of attorneys and judges generally oppose the [House’s]

proposed changes to Rule 11.”142 The only exception is that a “majority of judges and defendants’ attorneys, and a near majority of plaintiffs’ attorneys, believe that the purpose of Rule 11 should encompass

compensation of parties injured by violations of Rule 11 as well as

deterrence of such violations.”143

No changes have been made in Rule 11 since the 1993 amendments, either by the Advisory Committee or by Congress.

D. Rule 26(c) Protective Orders Study

1. Background and Committee Request

In May of 1990, the Subcommittee on Courts and Administrative

Practice of the Senate Judiciary Committee held hearings concerning

the extent to which courts were permitting parties to restrict access to

information obtained through discovery and relevant to public health

and safety by issuing protective orders allowing discovery documents

to be sealed or otherwise kept private. Around the same time, several

bills concerning protective orders and sealed settlement agreements

were introduced in the House of Representatives.144 FJC staff monitored the hearings and prepared memoranda describing unresolved

empirical questions that seemed central to the debate.145

In the hearings and in the media, members of Congress and interested groups of citizens raised concerns about the extent to which

federal courts issued protective orders under Rule 26(c) that prevented the public from learning about hazardous substances and

142 Id. at 2.

143 Id.

144 See, e.g., Federal Court Settlements Sunshine Act of 1991, H.R. 3803, 102d

Cong.; Federal Sunshine in Litigation Act of 1990, H.R. 2017, 101st Cong.; Act

Prohibiting Court Orders Which Limit the Availability of Information, H.R. 129, 101st

Cong. (1989); see also FEDERAL COURTS STUDY COMM., REPORT OF THE FEDERAL COURTS

STUDY COMMITTEE 102–03 (1990) (advocating the use of protective orders to protect

“sensitive materials”); Richard L. Marcus, The Discovery Confidentiality Controversy, 1991

U. ILL. L. REV. 457, 459 (arguing that current discovery rules are sufficient to deal

with current problems); Arthur R. Miller, Confidentiality, Protective Orders, and Public

Access to the Courts, 105 HARV. L. REV. 427, 462–63 (1991) (discussing certain reform

proposals dealing with changes in the discovery regime).

145 See, e.g., Memorandum from Elizabeth Wiggins, Joe Cecil, and James Pettler to

William B. Eldridge (Oct. 27, 1992) (on file with the author) [hereinafter Memorandum] (describing the above background and defining the empirical questions).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 34 28-MAY-02 12:35

1154 notre dame law review [vol. 77:4

products that were the subject of litigation.146 Some also raised concerns about the extent to which the parties accomplished the same

result by stipulating that product-related information would not be

disclosed after discovery or settlement.147

Developments in Congress, as well as the Center’s work and interest in the subject, were communicated to the Chair of the Advisory

Committee.148 At its November 1992 meeting, the Advisory Committee invited the Center to conduct an empirical study.149 The Committee noted a caution that even “[i]f a study fails to find widespread

difficulties with protective orders,” it will nevertheless be “difficult to

be confident that there are no problems.”150 The “first tentative results” of the Center’s work were presented to the Committee at its

April 1994 meeting.151 Center researchers presented an interim report to the Committee at its October 1994 meeting.152

2. Research Methods

As in the Rule 11 study, Center researchers identified cases involving protective order activity by examining codes in the court’s

electronic docketing system that referred to protective order motions,

stipulations, or orders.153 They then obtained more detailed information about a random sample of cases that involved protective order

activity from each district by recording information from docket

sheets and case files.154 Tracking the Committee’s interests, researchers collected information about the incidence of protective order activity, the use of stipulated agreements compared with contested or

uncontested motions, the number of motions granted and denied,

the stated objectives of protective orders, the frequency with which

146 E.g., Court Secrecy: Examining the Use of Secrecy and Confidentiality of Documents by

Courts in Civil Litigation: Hearing Before the S. Subcomm. on Courts and Administrative Practice of the Comm. on the Judiciary, 101st Cong. 124–25 (1990) (statement of Diane

Weaver).

147 E.g., id. at 155 (statement of Paul K. McMasters).

148 See Memorandum, supra note 145, at 4.

149 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S. 5 (Nov.

12–14, 1992).

150 Id.

151 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Comments on Proposed Rules (Apr. 28–29, 1994), available at http://www.uscourts.gov/

rules/Minutes/cv4-28.htm (last visited Feb. 12, 2002).

152 See ELIZABETH C. WIGGINS & MELISSA J. PECHERSKI, PROTECTIVE ORDER ACTIVITY

IN THREE FEDERAL JUDICIAL DISTRICTS: INTERIM REPORT TO THE ADVISORY COMMITTEE

ON CIVIL RULES 1 (1994) (on file with the author).

153 Id.

154 Id.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 35 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1155

protective orders are modified or dissolved, and the types of cases in

which protective orders are granted.155

3. Findings and Linkage with Rules Proposals

In October 1993, the Advisory Committee published for public

comment a preliminary draft of proposed changes to Rule 26(c).156

The thrust of the proposed changes was “to dispel any doubt that a

court has the power to modify or vacate a protective order” and to set

out standards to guide courts in exercising that power.157 One theme

that emerged from the Committee discussion of the public comments

was that there was a “continuing paucity of systematic empirical evidence about the use, modification, and effects of protective orders.”158 At its April 1994 meeting, the Committee deferred its

proposal and expressed to Congress the hope that “a pending Federal

Judicial Center survey of several district courts on the use of protective

orders would provide helpful empirical data on current practices.”159

At that same meeting, the Committee received “the first tentative results” of the FJC study and noted that it showed “a substantial rate of

protective order activity, more often involving contested motions than

stipulations” and “suggest[ed] the need to examine the common belief that ‘most’ protective orders result from agreement among the

parties.”160

The FJC presented its interim report at the Committee’s October

1994 meeting. As gleaned from the minutes of the Committee’s discussion of that report,161 the following findings are of interest:

• “there was protective order activity in a range of 4.7% to 10.0%

of all cases” in the three districts studied;

• “[m]ost protective order activity is initiated by motion, not by

stipulation of the parties; the highest figure for initiation by

party stipulation was 26%”;

155 Id. at 1–2.

156 COMM. ON RULES OF PRACTICE AND PROCEDURE OF THE JUDICIAL CONFERENCE OF

THE U.S., PROPOSED RULES, reprinted in 150 F.R.D. 323, 383 (1993).

157 Id. at 387 (Committee Note).

158 Civil Rules Advisory Comm., supra note 151.

159 Letter from the Honorable Patrick E. Higginbotham, Committee on Rules of

Practice and Procedure of the Judicial Conference of the United States, to the Honorable Herb Kohl, U.S. Senate (Aug. 25, 1994) (on file with the author).

160 Civil Rules Advisory Comm., supra note 151.

161 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Rule

26(c) (Oct. 20–21, 1994), available at http://www.uscourts.gov/rules/Minutes/cv10-

20.htm (last visited Feb. 12, 2002).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 36 28-MAY-02 12:35

1156 notre dame law review [vol. 77:4

• “the rate of hearings on motions was highly variable: in the District of Columbia, it was 12%, in Eastern Michigan, 59%, and in

Eastern Pennsylvania, 2%”;

• approximately 40% of the motions resolved in the three districts

were granted in whole or in part, and an approximately equal

percentage were denied [while the balance were not ruled on];

• contract, civil rights, and “other statutes” comprised “large portions” of the cases in which “an order was entered to restrict access to discovery materials”; “personal injuries accounted for 8%

or 9% of the total, depending on the district”; and

• protective orders “were modified or dissolved, whether by court

order or agreement in very few of the cases.”162

At its October 1994 meeting, the Committee revised its proposed

rule, adding to Rule 26(c)(1) “an express provision recognizing and

confirming the common practice of entering protective orders on

stipulation by the parties.”163 The Standing Committee adopted the

Advisory Committee’s recommendation and transmitted it to the Judicial Conference of the United States with a recommendation that it be

sent to the Supreme Court for submission to Congress. After intense

lobbying by public interest groups favoring public access to discovery

materials involving public health and safety,164 the Judicial Conference voted to strike the language in proposed Rule 26(c)(1) that

would have expressly permitted stipulated protective orders.165 The

Judicial Conference then voted to return the matter to the Advisory

Committee for its consideration without any directive as to whether or

not stipulated protective orders should be permitted.166 The Advisory

Committee recommended, and the Standing Committee approved,

publishing for comment the version returned by the Judicial Confer162 Id.

163 Id.

164 See, e.g., Richard J. Vangelisti, Proposed Amendment to Federal Rule of Civil Procedure 26(c) Concerning Protective Orders: A Critical Analysis of What It Means and How It

Operates, 48 BAYLOR L. REV. 163, 165–66 (1996) (“Under intense pressure from right

of access advocates, the Judicial Conference rejected the proposed amendment.”); id.

at 183–84 (further describing the intensity of the lobbying effort, including reference

to front page attention in the New York Times followed by prominent nationwide media coverage); Michael McCauley, Proposed Rule Changes Threaten To Increase Court Secrecy, NAT’L B. ASS’N MAG., Feb. 1996, at 31, 36 (describing the proposed changes,

soliciting comments, and offering an “an activist kit about the proposed secrecy

rules”). These interest groups do not appear to have participated in the rulemaking

process prior to this stage.

165 Reporter’s Note, Rule 26(c), in Advisory Comm. on Civil Rules, Agenda Book,

§ III.B (Washington, D.C., Apr. 18–19, 1996) (on file with author).

166 Id.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 37 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1157

ence, including the provision on stipulated protective orders.167 At its

October 1996 meeting, the Advisory Committee noted the “substantial

controversy” concerning the protective order proposal and held it “for

further study in conjunction with the broader study of discovery issues

to be launched over the next year.”168

Linkages between the results of the FJC study and the rules proposals remain unclear. After the FJC report, the Committee added a

controversial reference to stipulated protective orders in the proposed

rule. While the Committee expressed surprise that the FJC had found

such a modest percentage of protective orders entered by stipulation,

perhaps the quantification of that activity called the Committee’s attention to an aspect of the current practice not covered by the original draft. The Committee change, ironically, appeared to do little

more than authorize a practice that FJC showed to have been used in

a substantial minority of cases involving protective orders.

4. Outcome and Aftermath

The Advisory Committee conducted an extensive review of discovery rules between 1996 and 1999. The Committee decided not to

include changes to Rule 26(c) in the amendments that went into effect on December 1, 2000. The Committee reasoned that “richly experienced lawyers, from all fields of practice, find no need to change

present protective-order practice.”169 The Committee acknowledged

that “some members of Congress see a need.”170 Congress has not

adopted any related legislation.

E. Rule 68 Offers of Judgment Study

1. Background

Rule 68 permits “a party defending against a claim” to serve an

offer on the adverse party “to allow judgment to be taken against the

defending party for the money or property or to the effect specified in

167 COMM. ON RULES OF PRACTICE AND PROCEDURE OF THE JUDICIAL CONFERENCE OF

THE U.S., PROPOSED RULES, reprinted in 163 F.R.D. 91, 140–41 (1995).

168 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Chairman’s Remarks (Oct. 17–18, 1996), available at http://www.uscourts.gov/rules/Minutes/cv10-1796.htm (last visited Feb. 12, 2002).

169 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Rule

26(c) (Mar. 16–17, 1998), available at http://www.uscourts/gov/rules/Minutes/

10348civilminutes.htm (last visited Feb. 12, 2002). The Committee also cited the

FJC’s “sophisticated and helpful study.” Id. at 39.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 38 28-MAY-02 12:35

1158 notre dame law review [vol. 77:4

the offer.”171 If the adverse party does not accept the offer and does

not obtain a judgment “more favorable than the offer, the offeree

must pay the costs incurred after the making of the offer.”172 By its

terms, the rule is not available to parties making claims. Rule 68, part

of the original 1938 Federal Rules of Civil Procedure, appears to have

been rarely used, at least until the 1980s. Around that time, interest

in promoting settlement increased,173 and proposals to expand the

reach of Rule 68 were introduced as part of the rulemaking process.174 These proposals would have broadened the scope of Rule 68

by including claimants’ offers to settle and by defining “costs” to include attorneys’ fees. After considerable debate and criticism, the

proposals were shelved.175

2. Committee Request

In 1992, Judge William W. Schwarzer, then Director of the FJC,

published an article proposing changes in Rule 68. Judge Schwarzer

summarized the “principal objections” to the 1983 and 1984 proposals

as “that fee-shifting offers of judgment could have a devastating im171 FED. R. CIV. P. 68.

172 Id. The Supreme Court has interpreted the term “costs” to include the liability

the defending party might otherwise have to pay the opposing party’s attorneys’ fees

under some fee-shifting statutes, including the Civil Rights Attorneys’ Fee Awards Act

of 1976, which define the award of fees as part of the “costs” of the action. See Marek

v. Chesny, 473 U.S. 1, 9 (1985).

173 See FED. R. CIV. P. 16(c) advisory committee’s note (1983 amendment) (discussing then clause 7, now clause 9: “settlement should be facilitated at as early a stage

of the litigation as possible”). See generally D. MARIE PROVINE, SETTLEMENT STRATEGIES

FOR FEDERAL DISTRICT JUDGES (1986) (reporting on techniques of civil case resolutions without trial from the perspective of trial judges).

174 In 1983, the Committee published for comment a proposal to amend Rule 68

that would have mandated cost-shifting, including attorneys’ fees whenever a plaintiff

or defendant made an offer to settle a claim that was not accepted and that turned

out to be more favorable to the adverse party than the terms of a judgment obtained

through litigation. COMM. ON RULES OF PRACTICE AND PROCEDURE OF THE JUDICIAL

CONFERENCE OF THE U.S., PROPOSED RULES, reprinted in 98 F.R.D. 353, 361–63 (1983).

In the face of criticism, including challenges to the committee’s authority to impose

fee-shifting under the Rules Enabling Act, 28 U.S.C. § 2072 (1994), the Committee

published for comment a revised rule that it described as “purely procedural” and

that did “not provide for attorneys’ fee shifting but authorize[d] imposition of a sanction based on the creation of unnecessary delay and needless increase in the cost of

litigation.” COMM. ON RULES OF PRACTICE AND PROCEDURE OF THE JUDICIAL CONFERENCE OF THE U.S., PROPOSED RULES, reprinted in 102 F.R.D. 423, 424 (1984).

175 See Thomas D. Rowe, Jr. & Neil Vidmar, Empirical Research on Offers of Settlement:

A Preliminary Report, LAW & CONTEMP. PROBS., Autumn 1998, at 13, 15 and sources

cited therein.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 39 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1159

pact on plaintiffs . . . and that they could circumvent the statutory

provisions for attorney fees in civil rights cases.”176 The Advisory Committee reviewed Judge Schwarzer’s proposal at its November 1992 and

May 1993 meetings and at its October 1993 meeting considered a proposal to adopt “a sanction that provides for limited attorney fee shifting.”177 The proposed mechanism would “shift reasonable post-offer

fees, but subtract the benefit that results from the difference between

offer and judgment and limit the maximum award to the amount of

the judgment”—a proposal that came to be known as the “capped

benefit-of-the-judgment approach.”178 The Committee noted that the

Judicial Conference’s Court Administration and Court Management

Committee had endorsed a proposal under a Senate bill that would

adopt the same provision as a statute.179

At the October 1993 meeting, in addition to debating the merits

of the proposed revision to Rule 68 and the intricacies of the Rules

Enabling Act, the Committee debated whether to ask the FJC to study

the merits of the proposal by surveying attorneys in terminated civil

cases. By a vote of seven to two, the Committee defeated a motion

resolving “not ask the Federal Judicial Center to undertake the proposed survey.”180 The Committee then voted unanimously to “recommend that the Federal Judicial Center undertake two surveys,

including one focusing on the use of Rule 68 in statutory fee-shifting

cases.”181 In the course of its deliberations, the Committee expressly

noted that “the question of allocating responsibility between legislation and the Rules Enabling Act process is difficult” and that there

“may be substantive elements to attorney fee shifting in this setting

that counsel action by Congress.”182

176 William W Schwarzer, Fee-Shifting Offers of Judgment—An Approach To Reducing

the Cost of Litigation, 76 JUDICATURE 147, 147–48 (1992).

177 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Rule 68

(Oct. 21–23, 1993), available at http://www.uscourts.gov/rules/Mintues/civ10-21.htm

(last visited Feb. 12, 2002).

178 Id. at 15. In other words, in calculating post-offer attorneys’ fees the offering

party would have to offset the difference between the offer and the more favorable

judgment. In addition, a plaintiff would only have to pay the opposing party’s postoffer attorneys fees up to the amount of any judgment that was less than the unaccepted offer; a plaintiff would not be at risk of having to pay anything beyond the

amount of the judgment. If the plaintiff were to receive no judgment, there would be

no liability for fees.

179 See id. at 14.

180 Id. at 18–19.

181 Id. at 19.

182 Id. at 18.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 40 28-MAY-02 12:35

1160 notre dame law review [vol. 77:4

Professor Thomas Rowe, an Advisory Committee member from

1994 to 2000, conducted a computer-simulated study of the effects of

different versions of Rule 68 on the negotiating behavior of attorneys

and law students.183 There was no formal Committee request for his

study.

3. Research Methods

FJC researchers sent questionnaires to approximately 2000 attorneys from a sample of 800 federal civil cases that had terminated

about one year earlier. Half of the cases had settled and the other

half had been tried.184 Approximately 55% of those attorneys responded.185 Those questionnaires sought the attorneys’ opinions

about Rule 68 and their specific recollections and estimates from the

terminated case of the cost of litigation, settlement options, and possible application of variations of Rule 68 proposals. The sample concentrated on the approximately 30% of civil cases in which the parties

were likely to have faced a choice between settlement and trial and

focused on four types of cases: contract, tort, civil rights, and other.

Within those four groups, researchers selected 100 cases that ended in

trial and 100 that ended in settlement. A separate questionnaire was

used for civil rights cases to test the effects of Rule 68 in that type of

case.

In Professor Rowe’s study, a simulated civil rights claim arising

out of a mistaken address drug bust provided the framework for attorneys to respond to three alternative versions of Rule 68. Participants

were volunteers who were randomly selected from the roles of the national Inns of Court. One hundred seventy attorneys, approximately

8.5% of those solicited, participated.186

4. Findings and Linkage with Rules Proposals

A threshold finding goes to the issue of whether the incentives in

Rule 68 to settle cases are worth pursuing. Researchers found sub183 Thomas D. Rowe, Jr. & David A. Anderson, One-Way Fee Shifting Statutes and

Offer of Judgment Rules: An Experiment, 36 JURIMETRICS J. 255, 263 (1996) (discussing

alternative interpretations of Rule 68 in connection with the Civil Rights Attorneys’

Fees Awards Act); see also David A. Anderson & Thomas D. Rowe, Jr., Empirical Evidence on Settlement Devices: Does Rule 68 Encourage Settlement?, 71 CHI.-KENT L. REV. 519,

525–26 (1995) (discussing how proposed changes would impact the likelihood of

settlement).

184 See SHAPARD, supra note 35, at 3–5 (describing the research methods).

185 Id. at 5.

186 Rowe & Anderson, supra note 183, at 263.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 41 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1161

stantial opportunities for rulemakers to accomplish much in terms of

reducing the costs of litigation by improving incentives to settle. The

study found that the median expenses of sampled civil cases that went

to trial was $35,000 per party compared to median expenses of

$10,000 per party for sampled civil cases that settled.187 Moreover,

“about 40% of all tried cases could have settled, and 15%–20% very

likely would have settled if the parties simply had engaged in more

negotiation.”188 As to the timing of settlement, the findings suggested

that “a fee-shifting offer-of-judgment rule might lead the parties in

about 60% of the cases that would settle anyway to settle more quickly

and might save up to 50% of expenses in half of those cases.”189

Professor Rowe’s study of alternative versions of Rule 68 also produced useful threshold information. Rowe and Anderson concluded

that “the interpretation of Rule 68’s influence on § 1988 attorneys’

fees can have a significant effect on litigation bargaining.”190 Interpreting “Rule 68 ‘costs’ to include post-offer attorneys’ fees normally

paid by plaintiffs under § 1988” would produce the most impact:

“Under this construction, plaintiffs would accept offers 31% below

their no-Rule-68 bottom line, and defendants would make offers 9%

below their no-Rule-68 bottom line.”191

As to the related question of impact of existing Rule 68 on civil

rights plaintiffs, the FJC study found that “risk aversion is common

among civil rights plaintiffs,”192 and that “existing Rule 68 may exacerbate the risk aversion of civil rights plaintiffs . . . [but] only in limited

circumstances.”193

As to attorneys’ opinions about proposals to amend Rule 68, almost three-fourths of the attorneys who responded to the FJC survey

“favored amending Rule 68 to permit offers from both parties and to

include more significant incentives.”194 Surprisingly, “[e]ven in civil

rights cases, in which Rule 68 is often said to operate entirely to the

benefit of defendants, almost 50% of plaintiffs’ attorneys favored retaining the basic approach of the existing rule, while only 27% favored abolishing the current rule.”195 A key finding, however, is that

187 SHAPARD, supra note 35, at 6–8.

188 Id. at 2.

189 Id. at 12.

190 Rowe & Anderson, supra note 183, at 273.

191 Id.

192 SHAPARD, supra note 35, at 16.

193 Id. at 17.

194 Id. at 2.

195 Id. at 3.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 42 28-MAY-02 12:35

1162 notre dame law review [vol. 77:4

“attorneys have strong opinions on both sides of the issue.”196 The

strength of the opposing attorneys’ opinions may have contributed to

the decision to abandon the proposals. The Committee also faced

serious uncertainty about the viability of these exceedingly complex

proposals as well as serious concerns about its power under the Rules

Enabling Act to enact fee-shifting amendments.197

5. Outcome and Aftermath

At its March 20–21, 1997 meeting, the Advisory Committee concluded that “it may prove wise to defer further consideration [of Rule

68 proposals] pending developments in Congress.”198 Ending a fifteen-year saga on as cheery a note as possible, the Committee’s reporter predicted that “Rule 68 may yet provide the occasion for

exploring means of cooperating with Congress in matters that involve

the Civil Rules but that may best be addressed through the exercise of

congressional power to make substantive law.”199 Loosely stated, the

ball is in Congress’s court and the Committee’s research and debates

are available to assist Congress. None of the congressional proposals

to amend Rule 68, most notably those in the “Contract with America”

proposals, have been adopted.200

F. Rule 47(a) Attorney Participation in Voir Dire Study

1. Background

Rule 47(a) currently provides that “[t]he court may permit the

parties or their attorneys to conduct the examination of prospective

jurors or may itself conduct the examination.”201 In the event that the

court conducts the examination, it has the choice of permitting the

parties “to supplement” the court’s examination or to “submit to the

196 Id.

197 For a thorough discussion of the questions and concerns the Committee’s reporter had about the proposals, see generally Edward H. Cooper, Rule 68, Fee Shifting

and the Rulemaking Process, in REFORMING THE CIVIL JUSTICE SYSTEM 108–49 (Larry

Kramer ed., 1996).

198 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S. (Mar.

20–21, 1997), available at http://www.uscourts.gov/rules/Minutes/cv3-97.htm (last

visited Feb. 11, 2002).

199 Id.

200 See Thomas D. Rowe, Jr., Indemnity or Compensation? The Contract with America,

Loser-Pays Attorney Fee Shifting, and a One-Way Alternative, 37 WASHBURN L.J. 317, 317

(1998) (“The early weeks of the newly Republican Congress in 1995 may turn out to

have been a high-water mark for the idea of English-style, loser-pays attorney fee shifting in America.”).

201 FED. R. CIV. P. 47(a).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 43 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1163

prospective jurors such additional questions of the parties or their attorneys as it deems proper.”202 A 1977 FJC study found that “approximately three-fourths of federal district judges conduct voir dire

examination without oral participation by counsel.”203

From time to time, legislation had been introduced in Congress

to establish a right for attorneys to participate in voir dire. At its October 1994 meeting, the Advisory Committee called attention to continuing efforts of senators and representatives to enact such legislation

and to Judicial Conference opposition to such efforts.204 Citing the

need for attorneys to develop information to justify the use of peremptory challenges—a need created by the Supreme Court in a series

of cases attempting to control challenges made on the basis of racial,

ethnic, or gender stereotypes205—the Committee drafted proposed

rules that would permit attorney participation in voir dire.206 During

the course of its deliberations, the Committee indicated that without

its action, “the lawyers who have addressed the Committee will return

to Congress to renew longstanding efforts to secure legislation.”207

2. Committee Request

The Committee’s chair requested that the Center update its 1977

survey of district judges’ practices in conducting voir dire. The Committee requested the study early in 1994, and the Center presented its

results at the Committee’s October 1994 meeting.

202 Id.

203 GORDON BERMANT, CONDUCT OF THE VOIR DIRE EXAMINATION: PRACTICES AND

OPINIONS OF FEDERAL DISTRICT JUDGES 6 (1977).

204 See Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Rule

47(a) (Oct. 20–21, 1994), available at http://www.uscourts.gov/rules/Minutes/cv10-

20.htm (last visited Feb. 11, 2002).

205 Id. Attorneys argue that cases such as Batson v. Kentucky, 476 U.S. 79 (1986)

(holding that it is a violation of the Equal Protection Clause for prosecution to exclude jurors based on race), and J.E.B. v. Alabama ex rel. T.B., 511 U.S. 127 (holding

that it was a violation of the Equal Protection Clause for a state to use peremptory

strikes to exclude jurors on the basis of gender), spurred the need for greater

participation.

206 See Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Rule

47(a) (Apr. 20, 1995), available at http://www.uscourts.gov/rules/Minutes/mincv4.htm (last visited Feb. 11, 2002).

207 Id.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 44 28-MAY-02 12:35

1164 notre dame law review [vol. 77:4

3. Research Methods

Center staff mailed written questionnaires to a randomly selected

sample of 150 federal district judges, 124 (83%) of whom completed

and returned it.208

4. Findings and Linkage with Rules Proposals

The 1994 FJC study found that 59% of federal district judges permitted attorneys to participate directly in voir dire proceedings.209

The rate of attorney participation had more than doubled since the

1977 FJC study, a change that had occurred without any change in

Rule 47(a). Significantly and surprisingly, the survey also found that

district judges who permitted direct attorney participation in voir dire

spent approximately the same amount of time selecting jurors as

judges who did all of the questioning.210 The study also found a host

of mechanisms that judges use to control voir dire, ranging from general admonitions regarding improper questions to specific limits on

time and manner of attorney questioning.211

The Committee cited all of the above findings during the meeting at which it decided to proceed with a proposal to amend Rule

47(a).212 The Committee’s published proposal also cited those findings in the proposed Committee note.213 The Committee inferred

from the study’s findings that “judges who permit party participation

have found little difficulty in controlling potential misuses of voir

dire” and concluded that “the problems that have been perceived in

some state-court systems of party participation can be avoided by making clear the discretionary power of the district court to control the

behavior of the party or counsel.”214 Consistent with those findings,

the Committee draft started with the proposition that “[t]he court

208 Memorandum from John Shapard & Molly Johnson to the Advisory Committee

on Civil Rules on Survey Concerning Voir Dire 1 (Sept. 26, 1994) (on file with author

and with the Administrative Office of the U.S. Court’s Rules Support Office).

209 Id.

210 Id. at 2. Notably, the median amount of time spent on voir dire increased

from less than one hour in 1977 to between one and two hours in 1994. Id.

211 See id. at 5–6.

212 Civil Rules Advisory Comm., supra note 204.

213 COMM. ON RULES OF PRACTICE AND PROCEDURE OF THE JUDICIAL CONFERENCE OF

THE U.S., PROPOSED RULES, reprinted in 163 F.R.D. 139, 145 (1995). The Committee

did present both sides of the participation finding: “Although a recent survey shows

that a majority of district judges permit party participation, the power to exclude is

often exercised.” Id., reprinted in 163 F.R.D. 139, 145 (1995) (citing Shapard & Johnson, supra note 207).

214 Id., reprinted in 163 F.R.D. 139, 145 (1995).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 45 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1165

shall conduct the voir dire examination of prospective jurors.”215 The

proposed rule continued by providing that “the court shall also permit

the parties to orally examine the prospective jurors to supplement the

court’s examination within reasonable limits of time, manner, and

subject matter, as the court determines in its discretion.”216

5. Outcome and Aftermath

Prior to publication for public comment, the Committee faced an

unprecedented attempt to have the Judicial Conference mandate that

the proposals not be published, an attempt that was rejected by the

Judicial Conference “[a]fter spirited discussion.”217 At the public

hearings, “[a]lmost all of the many federal judges who commented on

the proposal spoke in opposition,” voicing as a “common theme, . . .

the fear that they will lose control if they lose the unlimited right to

deny any lawyer participation in voir dire.”218 The Committee decided to shelve the proposed rule change and to encourage the FJC,

“[which] seems receptive—to put voir dire on its educational agenda

for new judges and for judge workshops.”219 The Committee noted

that there “may be some room for systematic experimentation to test

the information provided by the FJC survey of federal judges.”220

Neither the Advisory Committee nor the FJC has taken action on the

latter proposal.

The Committee reported at its October 6, 1997 meeting that the

Committee “did urge the Federal Judicial Center to frame its sessions

for new judges to stress the importance of party participation. This

has been done. Judge Higginbotham, the former chair of this Committee has spoken on the topic at several meetings.”221

215 Id., reprinted in 163 F.R.D. 139, 144–45 (1995).

216 Id., reprinted in 163 F.R.D. 139, 145 (1995).

217 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S. (Nov.

9–10, 1995), available at http://www.uscourts.gov/rules/Minutes/min-cv11.htm (last

visited Feb. 11, 2002).

218 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Rule

47(a) (Apr. 18–19, 1996), available at http://www.uscourts.gov/rules/Minutes/cv4-

1896.htm (last visited Feb. 11, 2002).

219 Id. at 5.

220 Id.

221 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Agenda

Items (Oct. 6–7, 1997), available at http://www.uscourts.gov/rules/Minutes/cv10-

97.htm (last visited Feb. 11, 2002).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 46 28-MAY-02 12:35

1166 notre dame law review [vol. 77:4

G. Rule 23 Class Actions Study

1. Background

Advisory Committee activity concerning Rule 23 class actions has

followed multiple paths in varying phases over the past decade. For

clarity and simplicity, we present this complex set of proposals within

the framework of these phases, which have roughly followed the tenure of chairs of the Committee.

2. Phase One

After almost a generation following amendments to Rule 23 in

1966, momentum for change began to build in the late 1980s.222 The

catalyst for Committee action appeared to have been a March 1991

directive from the Judicial Conference that the Committee “study

whether Rule 23, F.R.C.P., should be amended to accommodate the

demands of mass tort litigation.”223 Judge Sam C. Pointer, Jr. chaired

the Advisory Committee at that time and had also served on an American Bar Association Section of Litigation committee that had studied

class actions and issued an extensive report in 1986.224 The recommendations included eliminating the three subdivisions of Rule

23(b), treating notice and opt-out provisions of Rule 23(c) on a caseby-case basis, authorizing pre-certification rulings on motions to dismiss or for summary judgment, facilitating early judicial management

of class actions, and permitting appellate review of the certification

ruling by permission of the court of appeals.225 The Committee

drafted a proposed rule revision that incorporated many, if not all, of

the recommendations contained in that report.226

222 For a brief history of the interlude between 1966 and Committee action to

review Rule 23, see FJC EMPIRICAL STUDY OF CLASS ACTIONS, supra note 63, at 78–81.

223 JUDICIAL CONFERENCE OF THE U.S., REPORT OF PROCEEDINGS 33 (1991). The

Judicial Conference’s action in turn originated in a report of the Ad Hoc Committee

on Asbestos Litigation, appointed by the Chief Justice of the United States in September 1990. AD HOC COMM. ON ASBESTOS LITIGATION, REPORT 37 (1991). The Ad Hoc

Committee’s first recommendation was that the Judicial Conference “[e]ndorse in

principle the proposition that Congress should enact a national legislative scheme to

come to grips with the impending disaster relating to resolution of asbestos personal

injury disputes.” Id. at 27.

224 ABA SEC. OF LITIG., REPORT AND RECOMMENDATIONS OF THE SPECIAL COMMITTEE

ON CLASS ACTION IMPROVEMENTS, reprinted in 110 F.R.D. 195 (1986).

225 Id., reprinted in 110 F.R.D. 195, 199–200 (1986).

226 See Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., 3–5

(Nov. 12–14, 1992). “The current draft is based in large part on a 1986 report of the

Litigation Section of the American Bar Association.” Id. at 4.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 47 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1167

3. Phase Two

In June 1993, the Standing Committee deferred action on the

Committee’s proposal, citing the press of other business227 and the

anticipated appointment of new members to the Committee. Judge

Patrick Higginbotham assumed the chair of the Advisory Committee

at its October 1993 meeting and scheduled a series of intensive meetings and symposiums designed to educate the Committee on all aspects of contemporary experience with Rule 23. The Committee

identified and called on selected practitioners, academics, and judges

to recount their experiences with the current rule and to point out

areas that needed attention. Conferences were held at Southern

Methodist University, the University of Pennsylvania, the University of

Alabama, and New York University. The Committee asked the FJC to

conduct empirical research and later encouraged researchers from

RAND to address the subject as well, yielding reports that will be highlighted below. Judge Higginbotham shepherded a series of proposals

through extensive revisions and into publication228 before his term

expired in 1996.

4. Phase Three

Judge Paul Niemeyer assumed the Advisory Committee chair in

October 1996 and presided over a period of public comment that

yielded more than 1500 pages of written statements and testimony

and over 700 pages of transcripts of oral testimony.229 This round of

revision produced a single rule change, creating a procedure to petition a court of appeals to allow an interlocutory appeal of a decision

granting or denying class certification.230 At Judge Niemeyer’s request, the Chief Justice appointed a working group to consider fur227 See discussion supra Parts II.C (Rule 11 Sanctions), II.D (Rule 26(c) Protective

Orders), and II.H.1 (Rule 26(a)(1) Protective Orders).

228 See COMM. ON THE RULES OF PRACTICE AND PROCEDURE OF THE JUDICIAL CONFERENCE OF THE U.S., PROPOSED RULES, reprinted in 167 F.R.D. 535, 559–66 (1996).

229 The proceedings are compiled into four volumes. ADVISORY COMM. ON CIVIL

RULES, JUDICIAL CONFERENCE OF THE U.S., WORKING PAPERS OF THE ADVISORY COMMITTEE ON CIVIL RULES ON PROPOSED AMENDMENTS TO CIVIL RULE 23 (1997) [hereinafter

WORKING PAPERS]. Volume one covers proposed rules and minutes of committee

meetings as well as appendices reproducing the Center’s empirical study and Professor Edward Cooper’s article entitled Challenges to the Rulemaking Process. Volume two

contains written comments submitted to the Committee. Volume three contains transcripts of hearings held in Philadelphia, Dallas, and San Francisco. Volume four consists of written statements of witnesses who testified at the hearings.

230 See FED. R. CIV. P. 23(f).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 48 28-MAY-02 12:35

1168 notre dame law review [vol. 77:4

ther actions relating to mass torts, including the possibility of further

changes in the class action rules.231

5. Phase Four

Judge Niemeyer also began the process of further revisions in

Rule 23 that remain in process. Judge David F. Levi became the Committee chair in 2001. In that year, the Standing Committee published

several proposed changes for comment,232 and the Advisory Committee scheduled hearings in San Francisco, Washington, and Dallas between November 30, 2001 and February 4, 2002.

6. Phase Five

In June 2001, the Advisory Committee and the Standing Committee decided to defer publication of several proposed Rule 23 changes

so that the Advisory Committee could consider them further.233 The

proposal dealt with overlapping and competing class actions. The Advisory Committee sponsored a conference at the University of Chicago

on October 22–23, 2001 to consider those pending proposals along

with those published for comment.

7. Committee Requests

There were no requests for empirical study during Phase One.

The Committee chair opted at that time to proceed on the basis of the

Committee’s collective experience and the Section of Litigation report. At the outset of Phase Two, however, the Committee expanded

the scope of its review and undertook to develop a research program

on class actions, “working initially with the Federal Judicial Center.”234

The Committee’s schedule called for interim reports at the University

231 See ADVISORY COMM. ON CIVIL RULES, JUDICIAL CONFERENCE OF THE U.S., REPORT

OF THE ADVISORY COMMITTEE ON CIVIL RULES AND THE WORKING GROUP ON MASS

TORTS TO THE CHIEF JUSTICE OF THE U.S. AND TO THE JUDICIAL CONFERENCE OF THE

UNITED STATES (1999), reprinted in 187 F.R.D. 293 (1999). The report recommended,

among other things, that the Chief Justice appoint an Ad Hoc Committee on Mass

Torts to recommend legislation, rule changes, and case management approaches.

Id., reprinted in 187 F.R.D. 293, 296 (1999). No action was taken on this recommendation, leaving consideration of class action rules changes with the Advisory Committee.

232 COMM. ON RULES OF PRACTICE AND PROCEDURE OF THE JUDICIAL CONFERENCE OF

THE U.S., PROPOSED RULES, reprinted in 201 F.R.D. 560, 586–718 (2001) [hereinafter

2001 PROPOSED RULES].

233 Standing Comm., DRAFT Minutes, June 2001, at 19 (on file with author).

234 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Rule 23

(Apr. 28–29, 1994), available at http://www.uscourts.gov/rules/Minutes/cv4-28.htm

(last visited Feb. 11, 2002). Judge Higginbotham appointed a research subcommittee
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 49 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1169

of Pennsylvania in February 1995, and at New York University in April

1995, with a final report to be prepared in time for the Committee’s

deliberations at the University of Alabama in November 1995.

During Phase Four, in March 2000, the Advisory Committee’s

subcommittee on class actions asked the FJC to develop illustrative

forms of notice to a class communicating the required information in

plain language. The Center provided interim reports to the subcommittee at its January 2001 meeting and to the Committee at its April

2001 meeting. Notices were posted on the FJC’s web site in August

2001 in conjunction with the Committee’s publication of proposed

revisions to Rule 23(c).

Phase Five has included informal discussion about potential research projects relating to competing and overlapping class actions

and settlements. In December 2001, the Committee asked the Center

to gather data on class action filing trends and to survey lawyers to

gather data about the effects of Amchem Products, Inc. v. Windsor 235

and Ortiz v. Fibreboard Corp. 236 on class actions in federal courts.

8. Research Methods

Professor Edward Cooper, the Committee’s reporter, developed

an extensive “wish list” of empirical questions to guide the Center’s

research effort.237 The topics on the list were: Individual Actions and

Aggregation: Routine Class Actions; Race to File; Representatives:

Who? Whence? Why?; Time of Certification; Certification Disputes;

Plaintiff Classes; Defendant Classes; Issues Classes and Subclasses; Notice; Opt-Outs and Opt-Ins; Individual Member Participation; Settlement; Trial; Small Claims Classes; Fee-Recovery Ratios; Overlapping

Classes; Counterclaims and Discovery; and Res Judicata.

To address this wide-ranging list of issues, the Center staff proposed to study all cases that had terminated within the most recent

two-year period in four district courts that had high levels of activity

and two district courts with medium levels of activity. Major unanticipated difficulties in identifying terminated class action cases in the

consisting of Circuit Judge Anthony Scirica, District Judge David Doty, Professor

Thomas Rowe, and Professor Edward Cooper.

235 521 U.S. 591 (1997) (holding that a class certification failed to satisfy the requirements of Rule 23).

236 527 U.S. 815 (1999) (holding that “applicants for contested certification”

under Rule 23 need to demonstrate that the class action fund is limited “by more than

the agreement of the parties”); see also infra Part II.K.

237 An evolved version of Professor Cooper’s empirical questions can be found at

Edward H. Cooper, Rule 23: Challenges to the Rulemaking Process, 71 N.Y.U. L. REV. 13,

42–52 (1996).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 50 28-MAY-02 12:35

1170 notre dame law review [vol. 77:4

first two districts led to a modification of the proposed study, limiting

it to four districts with high levels of class action activity.238 With that

modification, Center reports were presented as scheduled.

Data from a 1987–1990 district court study of judicial time spent

in managing various aspects of different types of civil and criminal

cases were used to identify the amount of judicial time various types of

class actions typically took.239 A case study of class activity in the

breast implant litigation was proposed for the indefinite future.240

9. Findings and Linkage with Rules Proposals

Rather than review the voluminous findings in the study, this Section will document the findings referred to in the proposed Committee notes to revisions of Rule 23 and in discussions of empirical data

during Committee deliberations. The Committee note to the 1996

proposed revision described in phases two and three, above, stated

generally that the FJC study “provided much useful information that

has helped shape these amendments.”241

A key concept in the proposed revision required distinguishing

among claims that could stand on their own, claims that required aggregation to support their viability, and claims where “the probable

relief to individual class members” might not justify “the costs and

burdens of class litigation.”242 Informing this approach was the FJC’s

finding that “median individual recovery figures . . . ranged from $315

to $528 . . . far below the level that would be required to support

individual litigation.”243

238 See FJC EMPIRICAL STUDY OF CLASS ACTIONS, supra note 63, at 197–98.

239 See id. at 22–23, 95–97, 199.

240 This aspect of the proposal led to two Center studies of settlement class actions. See generally GIBSON, supra note 49 (studying mass tort limited fund class action

settlements); TIDMARSH, supra note 52 (investigating five cases in which Rule 23 was

used to obtain settlements of tort class actions). For a full study of the breast implant

litigation, see generally HOOPER ET AL., supra note 88.

241 COMM. ON RULES OF PRACTICE AND PROCEDURE OF THE JUDICIAL CONFERENCE OF

THE U.S., supra note 228, reprinted in 167 F.R.D. 535, 560 (1996).

242 Id., reprinted in 167 F.R.D. 535, 559 (1996).

243 Id., reprinted in 167 F.R.D. 535, 561 (1996); see also FJC EMPIRICAL STUDY OF

CLASS ACTIONS, supra note 63, at 84–85. At the low end of the scale, Committee discussions sometimes focused on anecdotes like “an overcharge of 2 cents a month

imposed by a telephone company for 12 months on 2,000,000 customers.” Civil Rules

Advisory Comm., supra note 218. The Committee in that instance considered the FJC

data which found nine of 150 certified class actions in which individual recoveries

were valued at less than $100, three of which were valued at less than $25, with the

lowest being $16. “But it was responded that very small claims cases do in fact exist.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 51 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1171

The 1996 proposals included a provision to certify class actions

“for purposes of settlement, even though the requirements of subdivision [23](b)(3) might not be met for purposes of trial.”244 During

deliberations on that proposal, this author specified for the Committee that the FJC study found that “of 150 certified classes, 60 were

certified only for settlement.”245

Regarding notice and the opportunity to opt out, the Committee

note reported that “the Federal Judicial Center study suggests that notices of settlement do not always provide the clear and succinct information that must be provided to support meaningful decisions

whether to object to the settlement or . . . whether to request exclusion.”246 In the 2001 proposed revision,247 the Committee states that

the notice to the class “must concisely and clearly describe in plain,

easily understood language” the relevant rights and consequences that

flow from class certification.248 In conjunction with the publication of

that proposed revision, the Center has published on its web page illustrative securities and product liability notices written in plain

language.249

Regarding the timing of the class certification decision, both the

1996 and 2001 proposals contain language designed to alter the “as

soon as practicable” provision in the current rule. Both versions cite

the FJC study’s uncovering of “many cases in which it was doubtful

whether determination of the class-action question was made as soon

At least in some parts of the country very small claims classes are filed in state courts

and removed.” Id.

244 COMM. ON RULES OF PRACTICE AND PROCEDURE OF THE JUDICIAL CONFERENCE OF

THE U.S., supra note 228, reprinted in 167 F.R.D. 535, 559 (1996).

245 Civil Rules Advisory Comm., supra note 197. In Amchem v. Ortiz, 521 U.S. 591

(1997), the Supreme Court concluded that settlement is relevant to class certification

after citing the FJC study for the proposition that “the ‘settlement only’ class has become a stock device.” 521 U.S. at 618; see also FJC EMPIRICAL STUDY OF CLASS ACTIONS,

supra note 63, at 146 (“A large number of these cases were settlement classes which

were certified simultaneously with the preliminary approval of a proposed

settlement.”).

246 COMM. ON RULES OF PRACTICE AND PROCEDURE OF THE JUDICIAL CONFERENCE OF

THE U.S., supra note 228, reprinted in 167 F.R.D. 535, 564 (1996); see also FJC EMPIRICAL

STUDY OF CLASS ACTIONS, supra note 63, at 132–34.

247 See supra Part II.G.5.

248 COMM. ON RULES OF PRACTICE AND PROCEDURE OF THE JUDICIAL CONFERENCE OF

THE U.S., PROPOSED RULES, reprinted in 201 F.R.D. 586, 604–05 (2001). The Committee note references the FJC study. Id., reprinted in 201 F.R.D. 586, 607–08 (2001).

249 Judge David Levi’s memorandum transmitting the proposed revisions refers

the reader to the FJC’s illustrative notices. Id., reprinted in 201 F.R.D. 586, 594 (2001).

The illustrative notices can be found at http://www.fjc.gov under Current FJC Activities: Class Action Notices (last visited Jan. 28, 2002).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 52 28-MAY-02 12:35

1172 notre dame law review [vol. 77:4

as practicable after commencement of the action.”250 The FJC study

found that overall “approximately two out of three cases in each of the

four districts had rulings on either a motion to dismiss, a motion for

summary judgment, or a sua sponte dismissal order” before any ruling

on class certification.251 The proposed Committee note indicates that

“[t]he party opposing the class may prefer to win dismissal or summary judgment as to the individual plaintiffs without certification and

without binding the class that might have been certified.”252 To conform with the practice found in the FJC study, a practice the Committee deemed to be valuable, the Committee proposed the substitution

of “at an early practicable time” for “as soon as practicable.”253

The FJC study found a handful of cases in which there was no

notice and hearing before the court approved a classwide settlement.254 Without referring directly to the FJC findings, the proposed

1996 revision added a hearing requirement to Rule 23(e) “to confirm

the common practice of holding hearings as a part of the process of

approving dismissal or compromise of a class action.”255

In proposing a discretionary interlocutory appeal as new Rule

23(f), the Committee cautioned that “[p]ermission to appeal should

be granted with restraint” because “[t]he Federal Judicial Center

study supports the view that many suits with class-action allegations

present familiar and almost routine issues that are no more worthy of

immediate appeal than many other interlocutory rulings.”256

In Phase Four, the Committee proposed a new set of rules dealing with appointment of counsel and awarding attorneys’ fees. In the

Committee note to proposed Rule 23(g), the Committee referred to

the FJC study’s finding on the time between the filing of the action

and rulings on certification to support the proposition that setting

250 2001 PROPOSED RULES, supra note 232, reprinted in 167 F.R.D. 560, 607–08

(2001); see id., reprinted in 167 F.R.D. 560, 564–65 (2001).

251 FJC EMPIRICAL STUDY OF CLASS ACTIONS, supra note 63, at 32.

252 2001 PROPOSED RULES, supra note 232, reprinted in 167 F.R.D. 560, 608 (2001).

253 Id., reprinted in 167 F.R.D. 560, 604 (2001). The Committee note discusses at

considerable length reasons judges may have for deferring the decision on class certification. Id. at 607–10.

254 FJC EMPIRICAL STUDY OF CLASS ACTIONS, supra note 63, at 146–47.

255 COMM. ON RULES OF PRACTICE AND PROCEDURE OF THE JUDICIAL CONF. OF THE

U.S., PROPOSED RULES, reprinted in 167 F.R.D. 523, 565 (1996) [hereinafter 1996 PROPOSED RULES]. The 2001 proposals contain a hearing requirement in Rule

23(e)(1)(C), which “confirms and mandates the already common practice of holding

hearings as part of the process.” 2001 PROPOSED RULES, supra note 232, reprinted in

167 F.R.D. 560, 619 (2001).

256 FED. R. CIV. P. 23(f) advisory committee’s note; see also NYU Empirical Analysis,

supra note 63, at 87–92.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 53 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1173

aside time for appointing counsel “should not often present

difficulties.”257

In proposing a new rule regarding procedures and standards for

awarding attorneys’ fees, Rule 23(h), the Committee cites the RAND

report for the proposition that “[f]ee awards are a powerful influence

on the way attorneys initiate, develop, and conclude class actions.”258

The Committee also cited the same report for the proposition that a

“fundamental focus is the result actually achieved for class

members.”259

10. Aftermath

Of the 1996 proposals, the interlocutory appeal provision of Rule

23(f) was the only change adopted. It went into effect on December

1, 1998. Proposals relating to factors affecting the certification of class

actions, including settlement class actions, have been deferred but not

discarded. In December 2001, the Committee asked the Center to

conduct research relating to the need to revive proposals to clarify the

criteria for certifying settlement class actions. Proposals published for

comment in August 2001 are pending hearing and further review by

the Advisory Committee and the Standing Committee. They include

the proposals discussed above dealing with the timing of certification,

the clarity of notice, the review of settlements, the appointment of

class counsel, and the award of fees. None of these appear to be controversial. The Phase Five proposals remain under Committee

consideration.

H. Rules 26 to 37 Discovery Study

1. Background

In December 1993, what one commentator described as “perhaps

the most contested discovery changes in the history of the Federal

Rules” took effect.260 Most of the controversy centered on the disclosure rules.261 Because Congress had adopted the Civil Justice Reform

Act of 1990, which called for a five-year period of exploratory use of

257 2001 PROPOSED RULES, supra note 232, reprinted in 167 F.R.D. 560, 640 (2001).

258 Id., reprinted in 167 F.R.D. 560, 646 (2001).

259 Id., reprinted in 167 F.R.D. 560, 650 (2001).

260 Jeffrey W. Stempel, Politics and Sociology in Federal Civil Rulemaking: Errors of

Scope, 52 ALA. L. REV. 529, 549 (2001).

261 See id. The amendments barely escaped congressional rejection. See Richard

L. Marcus, Discovery Containment Redux, 39 B.C. L. REV. 747, 767–68 (1998) and

sources cited therein. For a summary of related discovery rules changes proposed

and adopted in 1993, see id. at 764–68.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 54 28-MAY-02 12:35

1174 notre dame law review [vol. 77:4

various cost and delay reduction approaches as adopted by local district courts, the Committee had good reason not to propose making

the 1993 Rule 26(a)(1) disclosure rules mandatory across all districts.262 Instead, the 1993 Rule permitted courts, by local rule, to exempt all or particular types of cases from disclosure requirements or

to modify the information to be disclosed.263 The result was what

came to be referred to generally as a “balkanization” of federal discovery practice.264

In 1996, the American College of Trial Lawyers recommended to

the Committee a proposal to limit the scope of discovery to matters

relevant to the claims and defenses in a case.265 Rule 26(b)(1) at that

time permitted discovery into all matters relevant to the “subject matter” of the litigation.266

2. Committee Request

At his first meeting as Chair of the Advisory Committee in October 1996, Judge Paul V. Niemeyer gave priority to the issue of discovery revision. He announced that “[t]he time may have come to

consider changes [in discovery rules] more fundamental than those

made in recent years.”267 Judge Niemeyer appointed a discovery subcommittee,268 outlined a possible three-layered approach to discovery,

including the American College of Trial Lawyer’s proposal,269 laid out

a plan for gathering information and convening a conference the following September,270 and discussed with FJC staff the type of research

the Center might be able to provide.271 The FJC research results

would be presented at the September 1997 conference proposed by

Judge Niemeyer.

262 See FED. R. CIV. P. 26(a) advisory committee’s note.

263 See id.

264 See generally Marcus, supra note 261, at 770–72.

265 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., New

Business (Apr. 18–19, 1996), available at http://www.uscourts.gov/rules/Minutes/

CV4-1896.htm (last visited Feb. 5, 2002).

266 FED. R. CIV. P. 26(b)(1) (amended 2000).

267 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Chairman’s Remarks (Oct. 17–18, 1996), available at http://www.uscourts.gov/rules/Minutes/CV10–1796.htm (last visited Feb. 5, 2002).

268 Id.

269 See Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Discovery (Oct. 17–18, 1996), available at http://www.uscourts.gov/rules/Minutes/

CV10-1796.htm (last visited Feb. 5, 2002).

270 See id.

271 See id.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 55 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1175

3. Research Methods

The Center’s approach to research on discovery was two-pronged.

To capture the rich history of empirical work on discovery, Center

researchers reviewed the empirical literature on the subject.272 To

document current activity, the Center, working with the discovery subcommittee then chaired by District Judge David F. Levi, proposed to

survey attorneys in 1000 recently terminated federal cases, seeking information about the amount and cost of discovery in those cases, the

type and costs of any problems faced during various discovery events,

and the perceived effects of initial disclosure, expert disclosure, and

other aspects of the 1993 amendments. In addition, the Center and

the subcommittee decided that the questionnaire should solicit the

attorneys’ opinions about the need for discovery reform and the most

promising approaches to any needed reforms.273

4. Findings and Linkage with Rules Proposals

As with the class action findings, the results of the discovery research are too detailed to be included here. In general, the survey

found that attorneys reported:

• discovery expenses were typically quite modest, representing

about 3% of the amount at stake in the litigation;274

• discovery occurred in 85% of the cases studied and discovery

problems occurred in almost half of those cases; expenses associated with such problems represented about 4% of overall litigation expenses;275

• high levels of discovery problems and expenses were associated

with high stakes, complex, contentious, or discovery-laden litigation (but did not occur in all such cases);276

• initial disclosure was being widely used and in cases in which it

had any effect, the process worked as intended to reduce ex272 The product of that review was published as part of the September 1997 Boston College Symposium Conference on Discovery Rules. See McKenna & Wiggins,

supra note 60, at 787–807.

273 The product of the survey research was initially published by the Center as

THOMAS E. WILLGING ET AL., FED. JUDICIAL CTR., DISCOVERY AND DISCLOSURE PRACTICE,

PROBLEMS AND PROPOSALS FOR CHANGE: A CASE-BASED NATIONAL SURVEY OF COUNSEL

IN CLOSED FEDERAL CIVIL CASES (1997) [hereinafter FJC DISCOVERY PRACTICE,

PROBLEMS, AND PROPOSALS], and a later version was included in the FJC Disclosure &

Discovery Study, supra note 54.

274 FJC DISCOVERY PRACTICE, PROBLEMS & PROPOSALS, supra note 273, at 4.

275 Id. at 2–5.

276 Id. at 4.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 56 28-MAY-02 12:35

1176 notre dame law review [vol. 77:4

pense and delay without interfering with fair resolution of

cases;277 and

• the majority of attorneys wanted a uniform national rule governing disclosure.278

a. Initial Disclosure

For district judges, the proposal to remove the right of local districts to opt out of the requirements of Rule 26(a)(1) represented the

most controversial aspect of the proposed changes. In the debate

over whether to repeal, modify, or extend the then-current rule to the

national level, the Advisory Committee considered several alternatives.

The first was the version of Rule 26(a)(1) that had been adopted in

1993, which required disclosure of witnesses or documents that were

“relevant to disputed facts alleged with particularity in the pleadings.”279 The Committee noted that “initial studies . . . find [the 1993

amendments] effective. The Federal Judicial Center study is the most

recent and detailed.”280 More specifically, Committee members observed that the FJC study showed that “initial disclosure . . . rather

often succeeds in reducing cost or delay, or promoting settlement, or

leading to better outcomes.”281

Despite the FJC findings on the effectiveness of the 1993 amendments, the Committee responded to concerns expressed by judges

and attorneys in districts that had opted out of the 1993 amendments.

In the interests of uniformity and avoiding a battle among judges, lawyers, and members of Congress like the one that accompanied the

1993 amendments,282 the Committee adopted a “middle-ground proposal” that was expressly designed to “eliminate the ‘heartburn’ that

arises from requiring disclosure of the identity of unfavorable witnesses and documents.”283 Thus, the Committee adopted a version

that limited the required disclosures to witnesses and documents that

the disclosing party “may use to support its claims or defenses.”284

277 Id. at 2.

278 Id. at 3.

279 FED. R. CIV. P. 26(a)(1)(A), (B), reprinted in FEDERAL CIVIL JUDICIAL PROCEDURES AND RULES 132 (West 1999) (Rule amended 2000).

280 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Disclosure (Mar. 16–17, 1998), available at http://www.uscourts.gov/rules/Minutes/

0398civilminutes.htm (last visited Feb. 5, 2002).

281 Id.; see also FJC Disclosure & Discovery Study, supra note 54, at 562–65 (discussing

the perceptions of initial disclosure’s effects).

282 See supra notes 260–65 and accompanying text.

283 Civil Rules Advisory Comm., supra note 280.

284 FED. R. CIV. P. 26(a)(1)(A) & (B).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 57 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1177

At the meeting in which the discovery proposals were approved in

their final form, Judge Levi, then chair of the discovery subcommittee,

noted that he had been “surprised by the support expressed for initial

disclosure”285 and “learned that disclosure is practiced more widely

than we had thought.”286 He argued for “a uniform national procedure to enforce national substantive law.”287 Judge Levi observed that

the Committee had “heard opposition from many judges, but they

have not had the information we have had . . . . When empirical work

can be done, we have had it done.”288 In the end, the Committee

accepted the compromise “middle-ground” version, and it went into

effect on December 1, 2000 with little apparent fanfare.289

FJC data from the survey of attorneys and from other sources supported the Committee’s decision to seek a uniform national rule.290

The Committee cited prior FJC data to support the empirical proposition that “[a] striking array of local regimes in fact emerged for disclosure and related features introduced in 1993.”291 The Committee

observed specifically that “[l]awyers surveyed by the Federal Judicial

Center ranked adoption of a uniform national disclosure rule second

among proposed rules changes (behind increased availability of

judges to resolve discovery disputes) as a means to reduce litigation

expenses without interfering with fair outcomes.”292

The Center provided a less controversial form of assistance to the

Committee in helping to determine which broad categories of cases

should be excluded from the Rule 26(a)(1) disclosure requirements.293 These “low-end exclusions” represented the types of cases

in which discovery was unlikely. By surveying local rules promulgated

285 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Motion:

Rule 26(b)(1) (Apr. 19–20, 1999), available at http://www.uscourts.gov/rules/Minutes/0499civilminutes.htm (last visited Feb. 11, 2002).

286 Id.; see also FJC Disclosure & Discovery Study, supra note 54, at 559–61 (discussing

the prevalence of initial disclosure activity).

287 Civil Rules Advisory Comm., supra note 285.

288 Id.

289 An archive of rules submitted for Supreme Court approval can be found at

http://www.uscourts.gov/rules/archive.htm (last visited Feb. 11, 2002).

290 FJC Disclosure & Discovery Study, supra note 54, at 588–91.

291 FED. R. CIV. P. 26 advisory committee’s notes (2000 amendment) (discussing

purposes of amendments) (citing D. STIENSTRA, FED. JUDICIAL CTR., IMPLEMENTATION

OF DISCLOSURE IN UNITED STATES DISTRICT COURTS, WITH SPECIFIC ATTENTION TO

COURTS’ RESPONSES TO SELECTED AMENDMENTS TO FEDERAL RULE OF CIVIL PROCEDURE

26 (1998)).

292 Id. (citing FJC DISCOVERY PRACTICE, PROBLEMS, AND PROPOSALS, supra note 273,

at 44–45).

293 For a listing of the categories excluded, see FED. R. CIV. P. 26(a)(1)(E).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 58 28-MAY-02 12:35

1178 notre dame law review [vol. 77:4

under Rule 16 and under the 1993 disclosure rules and listing all exclusions, the FJC gave the Committee’s reporter systematically-collected information about district courts’ practices in excluding cases.

Using case filing statistics, the Center was able to estimate the percentage of all civil cases—approximately 33%—likely to be affected by the

exclusions.294

b. Scope of Discovery

For members of the bar, the most controversial and contentious

aspect295 of the Advisory Committee’s proposals to revise the discovery

rules involved narrowing the scope of discovery from matters “relevant

to the subject matter involved in the pending action”296 to matters

“relevant to the claim or defense of any party,”297 subject to expansion

by court order to allow discovery of any matter relevant to the subject

matter.298 The Committee supported its position, in part, by referring

to the FJC study’s finding that “[n]early one-third of the lawyers surveyed in 1997 by the Federal Judicial Center endorsed narrowing the

scope of discovery as a means of reducing litigation expense without

interfering with fair case resolutions.”299 Of course, as Professor

Thomas Rowe, a member of the Committee, observed in citing the

same data, “‘two-thirds [of the attorneys surveyed] did not express this

view.’”300 Moreover, the lawyers surveyed by the FJC ranked narrowing the scope of discovery fifth out of six types of reforms presented

for their consideration.301 Had the proposed rule been in effect at

the time of the survey, attorneys estimated that it would have reduced

litigation expenses by about 12%.302

294 See FED. R. CIV. P. 26 advisory committee’s notes (2000 amendment) (subdivision (a)(1)) (referring to the FJC estimate that the exemptions amount to about onethird of all civil filings).

295 See supra notes 260–65 and sources cited therein.

296 FED. R. CIV. P. 26(b)(1), reprinted in FEDERAL CIVIL JUDICIAL PROCEDURE AND

RULES 133 (West 1999) (Rule amended 2000).

297 FED. R. CIV. P. 26(b)(1).

298 Id.

299 FED. R. CIV. P. 26 advisory committee’s notes (2000 amendments) (subdivision

(b)(1)); see also FJC Disclosure & Discovery Study, supra note 54, at 584–87 (discussing

changes likely to reduce discovery problems and expenses).

300 Stempel, supra note 260, at 578; see also infra note 325 and sources cited

therein.

301 FJC Disclosure and Discovery, supra note 54, at 586–87.

302 Id. at 584–85 tbl.35.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 59 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1179

c. Duration of Depositions

The Committee amended Rule 30(b)(2) to limit depositions to

“one day of seven hours”303 unless the parties stipulated or the court

ordered otherwise.304 The FJC study found that the typical deposition

in the surveyed cases took three hours and that three-quarters of the

cases had depositions that averaged five hours or less.305 In response

to a discovery subcommittee request, the FJC examined its survey

database further and found that for those relatively few attorneys

(sixty-nine of 572, or 12%) who reported that depositions lasted too

long, the median length of the deposition was seven hours.306 Ten

percent of the overlong depositions lasted twenty-four hours or

more.307 Also, in response to a subcommittee request made during

the course of its drafting, FJC staff examined local district court rules

limiting the length of depositions and found that the length of depositions in those districts did not differ from those in districts without

such local rules.308 The Committee adopted the limit of seven hours,

a number tailored to fit below a level that some attorneys had found

problematic.

d. Number of Depositions

The Advisory Committee considered but decided against lowering the presumptive number of depositions below the ten per side

permitted by Rule 30(a)(2)(A). The Committee noted that the “FJC

study shows that most cases involve far fewer depositions than

[ten].”309 In deciding not to lower the limit, the Committee observed

that it had no indication that ten depositions per side is too many.310

303 FED. R. CIV. P. 30(d)(2).

304 Id.

305 FJC Disclosure & Discovery Study, supra note 54, at 571.

306 Memorandum from Thomas E. Willging to the Discovery Subcommittee 3

(Dec. 22, 1997) (on file with author).

307 Id.

308 MARIE CORDISCO LEARY & THOMAS E. WILLGING, NUMERICAL AND DURATIONAL

LIMITS ON DISCOVERY EVENTS AS ADOPTED IN FEDERAL LOCAL RULES AND STATE PRACTICES 10–11 (1998) (on file with author).

309 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Number

of Depositions (Mar. 16–17, 1998), available at http://www.uscourts.gov/rules/Minutes/0398civilminutes.htm (last visited Feb. 5, 2002); see also FJC Disclosure & Discovery

Study, supra note 54, at 571 (Table 24 shows that 75% of cases had seven or fewer

depositions).

310 See Civil Rules Advisory Comm., supra note 309; see also FJC Disclosure & Discovery Study, supra note 54, at 573 (Table 25 indicates that 4% of attorneys reported there

were too many depositions in surveyed cases).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 60 28-MAY-02 12:35

1180 notre dame law review [vol. 77:4

The Committee also commented that the “fact that most cases are

completed with far fewer [than ten] depositions tends to support the

conclusion that the stated limit has not encouraged parties to take

more depositions than they otherwise would.”311

e. Discovery Cutoffs

In its study of courts participating in the evaluation of the Civil

Justice Reform Act of 1990 policies, RAND found that reported lawyer

work hours on a case were significantly lower and the time to disposition of a case was significantly shorter in districts with smaller median

numbers of days to the discovery cutoff.312 RAND asserted that these

findings supported the policy of adopting shorter discovery cutoffs to

reduce costs and delay.313 In its survey, the FJC was unable to replicate those findings despite using case-specific measures of discovery

cutoffs and case duration.314

The Committee considered and decided not to adopt any of several alternatives for setting a date after which discovery would be cut

off. The Committee made reference to RAND and FJC data during its

deliberations, but its decision seemed to hinge primarily on the difficulty of setting firm trial dates to accompany any discovery cutoff and

the belief that an early discovery cutoff made little sense without an

early firm trial date.315

I. Rule 53 Use of Special Masters Study

1. Background

Rule 53 authorizes courts to appoint special masters in exceptional cases. The framework of the rule “appears to contemplate the

traditional activity of a special master in holding evidentiary hearings

and issuing reports with factual findings to facilitate a trial.”316 Although no empirical data existed until recently, it was apparent to

sophisticated observers that the functions for which courts appointed

special masters had grown far beyond the limited reach of Rule 53.317

311 Civil Rules Advisory Comm., supra note 309, at 37.

312 See Kakalik et al., supra note 16, at 666–67.

313 See id. at 680–81.

314 See FJC Disclosure & Discovery Study, supra note 54, at 557–59.

315 See Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Discovery Time Cut-off (Mar. 16–17, 1998), available at http://www.uscourts.gov/rules/

Minutes/0398civilminutes.htm (last visited Feb. 5, 2002). .

316 FJC SPECIAL MASTERS’ STUDY, supra note 44, at 1.

317 See, e.g., Edward H. Cooper, Civil Rule 53: An Enabling Act Challenge, 76 TEX. L.

REV. 1607 (1998).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 61 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1181

At the suggestion of two local district court Civil Justice Advisory

Groups in 1993, the Advisory Committee began to consider revising

the rules to accommodate various uses of special masters.318 The

Committee debated various approaches, including possible amendments to Rules 16 or 53, over a span of four meetings before deciding

that there was “no apparent need for imminent action.”319

2. Committee Request

At its December 1998 meeting, Judge Niemeyer appointed a subcommittee to “make a recommendation whether Rule 53 reform

should be pursued.”320 The Committee noted the need to “form a

clear picture of the seeming wide variety of present practices,”321 and

Judge Niemeyer indicated that it would be “appropriate to ask the

Federal Judicial Center to undertake any study that can be designed

in consultation with the Subcommittee.”322 The Center and the subcommittee designed a research plan that would produce preliminary

data on the incidence of special master activities for the October 1999

meeting and a final report for the April 2000 meeting.

3. Research Methods323

This study was based on cases in which special master appointments were considered. For the first phase of the study, a sample of

136 cases in which special master activity was recorded on a docket

sheet was extracted from a population that included approximately

445,000 terminated civil cases. Incidence of appointment was estimated based on this sample. Information from case files in the sample of cases was collected to respond to questions relating to the

circumstances of the appointment, the nature of the special masters’

duties, the type of person appointed, and other issues related to the

operation of Rule 53.

In Phase Two, researchers conducted telephone interviews of

judges and attorneys in a portion of the sample of cases. These interviews were designed to address qualitative questions regarding how

318 Id. at 1608 n.6.

319 Id. at 1609.

320 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Copyright Rules: Related Rules 65, 81 (Nov. 12–13, 1998), available at http:/

www.courts.gov/rules/Minutes/1198civilminutes.htm (last visited Feb. 5, 2002).

321 Id.

322 Id.

323 For a discussion of the methods used, see FJC SPECIAL MASTERS’ STUDY, supra

note 44 at 13–15, 81–92.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 62 28-MAY-02 12:35

1182 notre dame law review [vol. 77:4

the rule worked in practice, the goals of the appointment, their effectiveness, problems encountered, and the need, if any, for a rule

change.324

4. Findings and Linkage with Rules Proposals

The threshold question before the Committee was whether to

proceed with revising Rule 53. Though appointing a special master

was rarely raised, occurring in less than three cases per 1000, the raw

number of cases, more than 600 cases per year, impressed the Committee as “not a rare or inconsequential event.”325 Moreover, the

“wide range of activities from pretrial through trial and on to post-trial

work . . . suggests there is at least room to expand Rule 53, which

focuses only on trial uses.”326 The Committee concluded that the FJC

data “suggest a need to update Rule 53 to cover pretrial and post-trial

activity”327 and decided to proceed with revising Rule 53 “while the

FJC goes on with [Phase 2 of] its study.”328

The Committee drafted a revised rule that clarified the authority

of judges to appoint pretrial and posttrial masters, citing the FJC study

as confirming the premise for the revision.329 In the Committee note,

the drafters start with the proposition that “Rule 53 is revised to reflect

changing practices in using masters”330 that “[a] study by the Federal

Judicial Center documents the variety of responsibilities that have

come to be assigned to masters.”331

The Center report addressed the question of how appointments

at the pretrial and post-trial stages came to pass under a rule that did

not provide for them. The answer was that “[c]onsent and acquies324 For a listing of the research questions, see id. at 12–13.

325 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Rule 53

Subcommittee 37 (Oct. 14–15, 1999), available at http://www.uscourts.gov/rules/

Minutes/1099mnCV.pdf (last visited Feb. 5, 2002); see also FJC SPECIAL MASTERS’

STUDY, supra note 44, at 3, 15–21 (discussing the incidence of the special master

appointment).

326 Civil Rules Advisory Comm., supra note 325, at 37; see also FJC SPECIAL MASTERS’ STUDY, supra note 44, at 4–5, 52–64 (discussing why special master rules are

working despite their limits).

327 Civil Rules Advisory Comm., supra note 325, at 38.

328 Id.

329 2001 PROPOSED RULES, supra note 232, reprinted in 201 F.R.D. 560, 670 (2001)

(stating that “[t]he Federal Judicial Center did a study that . . . confirmed that special

masters often are used for purposes not clearly contemplated by Rule 53”).

330 Id., reprinted in 201 F.R.D. 560, 693 (2001).

331 Id., reprinted in 201 F.R.D. 560, 694 (2001) (citing FJC SPECIAL MASTERS’ STUDY,

supra note 44).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 63 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1183

cence appear to be the driving forces.”332 The proposed Rule

53(a)(1) would provide that “a court may appoint a master . . . (A)

[to] perform duties consented to by the parties.”333

The proposed revision calls for the order appointing a special

master to specify “the circumstances, if any, in which the master may

communicate ex parte with the court or a party.”334 In presenting the

proposal to the Standing Committee, the Advisory Committee’s reporter “pointed out that the Federal Judicial Center’s study of masters

in the district courts had revealed that ex parte communication between a master and either the court or the parties are the focus of

continuing concern but may be very beneficial in certain circumstances.”335 He went on to say that ex parte communication was “the

single most difficult problem cited by interviewees” in the FJC

study.336

Despite the lack of empirical data, the Committee proposed a major change in special master practice, namely the removal of authority

to appoint a special master to review facts and present a report in a

jury trial. Center research found, somewhat ironically, that special

masters were infrequently appointed for trial purposes, the original

core purpose of Rule 53 appointments.337 In none of the cases studied intensively in Phase Two did the appointments contemplate testimony before a jury by a special master.338 Seven cases were studied,

four of which involved bench trials and the other three of which involved court-appointed experts under Federal Rule of Evidence

706.339 The Committee’s rationale, however, deemphasized any empirical issues. The Committee concluded that the “practice intrudes

on the jury’s province with too little offsetting benefit” and that the

trial master’s evidence taking would be insulated from the jury and

perhaps even from a reviewing court.340

332 FJC SPECIAL MASTERS’ STUDY, supra note 44, at 5.

333 2001 PROPOSED RULES, supra note 232, reprinted in 201 F.R.D. 560, 683 (2001).

The proposed Committee note addresses the question of acquiescence by urging

courts to “be careful to avoid any appearance of influence that may lead a party to

consent to an appointment that would otherwise be resisted.” Id., reprinted in 201

F.R.D. 560, 695 (2001).

334 Id., reprinted in 201 F.R.D. 560, 686 (2001).

335 Standing Comm., supra note 233, at 27.

336 Id.

337 Id.

338 See FJC SPECIAL MASTERS’ STUDY, supra note 44, at 24–25.

339 Id.

340 2001 PROPOSED RULES, supra note 232, reprinted in 201 F.R.D. 560, 696–98

(2001).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 64 28-MAY-02 12:35

1184 notre dame law review [vol. 77:4

5. Aftermath

The proposed rule is pending. Written comments were due by

February 15, 2002. No one commented on Rule 53 at the first hearing, which was held in San Francisco on November 30, 2001. Apparently, the proposed rule has not generated any controversy. The

Advisory Committee was expected to review the proposed rule at its

May 2002 meeting. If approved by the Advisory Committee, the

Standing Committee, and the Judicial Conference at their 2002 meetings, and if approved by the Supreme Court and transmitted to Congress by May 1, 2003, the proposed amendment would go into effect

on December 1, 2003, unless Congress intervenes.341

J. Rules 26, 34, and 37 Electronic Discovery Study

1. Background

In October 1999, the Discovery Subcommittee of the Advisory

Committee identified “electronic discovery”—that is, the discovery of

computer-based evidence such as e-mail, databases, and Internet activity—as an area of growing concern for practitioners and judges

alike.342 Several proposals had been floated in the legal literature to

refine the definition of “document” found in Rule 34 or to specifically

address problems associated with electronic discovery in Rule 26.

During 2000, the subcommittee held two “mini-conferences” with

practitioners, judges, and technical experts to explore the problems

further and determine whether amendment of the rules was either

necessary or desirable.343

2. Committee Request

There being no consensus on the subcommittee or from the

mini-conferences as to whether amendment of the rules was necessary, the subcommittee asked the FJC to conduct case studies to determine whether existing discovery rules helped, hindered, or had no

341 See 28 U.S.C. § 2074(a) (1994).

342 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Discovery 23–28 (Oct. 14–15, 1999), available at http://www.uscourts.gov/rules/Minutes/

1099mnCV.pdf (last visited Feb. 5, 2002).

343 Richard Marcus, reporter for the subcommittee, detailed the status of the discussion and listed more than thirty ideas for rule amendments relating to electronic

discovery derived from the legal literature, the mini-conferences, and other sources.

Richard L. Marcus, Confronting the Future: Coping with Discovery of Electronic Material,

LAW & CONTEMP. PROBS., Spring/Summer 2001, at 253, 274–79. These preliminary

ideas for rules amendments have not been placed on the subcommittee or committee’s agenda for formal consideration as of January 2002.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 65 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1185

effect on the discovery of computer-based evidence, as distinct from

discovery overall.344

3. Research Methods

FJC researchers surveyed magistrate judges during the summer of

2000 to identify recently closed cases in which electronic discovery was

prominent, in which the documents filed with the court were publicly

available for study, and in which the participants would likely be willing to be interviewed about their discovery experience. The survey

also gathered informal data on the frequency and type of problems

associated with electronic discovery. Approximately twenty cases were

identified as candidates for study. Researchers have studied and

coded discovery-related documents filed in these cases to identify

types of evidence sought, types of problems encountered, and rules

implicated. They have also developed interview protocols to elicit

data from both judges and counsel related to specific issues identified

by the subcommittee.

4. Findings and Linkage with Rules Proposals

As of January 2002, this project was still underway, and there are

no findings to report. Center staff expect to present a preliminary

report and analysis to the subcommittee in the autumn of 2002.

K. Class Actions—Study of the Effects of Amchem and Ortiz

1. Background

As part of its 1996 proposals to amend Rule 23, the Advisory Committee proposed to add as Rule 23(b)(4) a settlement class action,

that is, one in which “the parties to a settlement request certification

under subdivision (b)(3) for purposes of settlement, even though the

requirements of subdivision (b)(3) might not be met for purposes of

trial.”345 While the proposal was pending, the Supreme Court decided the case of Amchem Products, Inc. v. Windsor,

346 in which it reversed in part a Third Circuit opinion holding that the fact of

settlement was irrelevant to determining whether a class should be

certified under Rules 23(a) and 23(b)(3).347

“Settlement is relevant to a class certification,” the Court held, in

that a district court confronted with a request for settlement-only class

344 See Civil Rules Advisory Comm., supra note 342, at 28.

345 1996 PROPOSED RULES, supra note 255, reprinted in 167 F.R.D. 523, 559 (1996).

346 521 U.S. 591 (1997).

347 Id. at 609–11.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 66 28-MAY-02 12:35

1186 notre dame law review [vol. 77:4

certification “need not inquire whether the case, if tried, would present intractable management problems . . . for the proposal is that

there be no trial.”348 The Court went on to hold that “other specifications of the Rule—those designed to protect absentees by blocking

unwarranted or overbroad class definitions—demand undiluted, even

heightened, attention in the settlement context.”349 The Court acknowledged the Advisory Committee’s proposal to amend Rule 23 to

“expressly authorize settlement class certification,” noted the “voluminous public comments—many of them opposed to, or skeptical of,

the amendment,” and concluded that the Court would “consider the

certification at issue under the Rule as it is currently framed.”350 Issues relating to the validity of the proposed amendment were necessarily left for another day.

After the Court’s decision, the Committee discussed the “difficulties and opportunities” presented by Amchem and by consensus decided to “defer further consideration of settlement classes.”351 In

part, the Committee wished to “to gain the benefit of greater

experience.”352

After Amchem, the Supreme Court decided Ortiz v. Fibreboard

Corp.,

353 which further restricted the ability of parties to settle mass

tort litigation on a global basis by interpreting narrowly Rule

23(b)(1)(B)’s applicability to limited fund settlements.354 One commentator concluded, for example, that “the Supreme Court’s decision

in Ortiz raises serious doubts about whether the Court would ever uphold a limited fund class action settlement of mass tort claims.”355

2. Committee Request

Against this backdrop, the Advisory Committee faced the question of whether to revive its proposal to establish standards for settlement class actions that might be less restrictive than those imposed by

the Supreme Court. Anecdotal evidence supported conjecture that

the Rule 23 standards were having the effect of driving class actions

into state courts. To assist in its impending decision whether to rec348 Id. at 619–20.

349 Id. at 620.

350 Id. at 619.

351 Minutes, Civil Rules Advisory Comm., Judicial Conference of the U.S., Rule 23

(Oct. 6–7, 1997), available at http://www.uscourts.gov/rules/Minutes/cv10-97.htm

(last visited Feb. 5, 2002).

352 Id.

353 527 U.S. 815 (1999).

354 See id. at 864–65.

355 GIBSON, supra note 49, at 28.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 67 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1187

ommend reviving the settlement class issue, in December 2001 the

subcommittee on class actions asked the Center to examine empirically the possible effects of Amchem, Ortiz, and related developments in

federal class action litigation on the filing and disposition of class actions in the federal courts.

3. Research Methods

At the time of this writing, it is too soon to specify precisely the

research designs to be employed in carrying out this study. Two observational field designs are being given serious consideration. The

first would entail an electronic search of all docket entries for at least

two years before the Third Circuit decision in Amchem up to the end

of 2001 to identify class actions and determine any trends in the filing

of class actions in federal court in various types of cases. The second

design would involve surveying attorneys who have appeared in recently terminated class actions and ask about the effects of Amchem

and Ortiz on the ability to resolve on a global basis the issues that gave

rise to the litigation.

4. Findings and Linkage with Rules Proposals

There are no findings to date. The expectation is that the findings will be used to inform the threshold decision about whether or

not to revive the settlement class issue and draft a proposed rule.

Having detailed the major research efforts conducted at the request of the Advisory Committee, it seems appropriate to turn to an

analysis of the patterns revealed by these summary case histories.

III. PATTERNS AND PRACTICES IN USING EMPIRICAL RESEARCH IN

CIVIL RULEMAKING

This Part reviews the research summaries presented in Part II and

discusses some patterns that emerge. It summarizes the timing, purposes, and types of the research and examines its impact in relation to

its intended purposes. Special attention will be paid to the relationship of the research to the rulemaking process, as well as to related

legislative activity.

A. Type and Timing of Research

The first and clearest conclusion is that none of the research conducted was, or could reasonably have been, experimental. The closest

approximation was a retrospective quasi-experimental design for the

offer-of-judgment research, but even that design could not be consid-
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 68 28-MAY-02 12:35

1188 notre dame law review [vol. 77:4

ered experimental in the sense of imposing prior controls that would

enable the results to support causal inferences.356

Observational field studies using surveys were the most frequently

used research designs and methods. This combination was used in

eleven of the twelve completed studies, and the twelfth was a quasiexperimental field study that used survey methods. All twelve studies

were retrospective, that is, studying the effects of existing rules. A few

surveys (for example, discovery, Rule 68) included questions about

proposals. None focused on the prospective application of rules proposals that had not been adopted.

Typically the field studies started with electronic searches of

docket records to identify cases and records that were then studied in

depth. Three written surveys sought specific information about a responding attorney’s experience in a recently-closed case. Another

sought attorneys’ opinions at a general level, and three sought judges’

opinions at a general level. Telephone interviews of attorneys and

judges complemented case-file information in one study (special masters). Focus groups of potential class members supported the effort to

study and improve class action notices.

The timing of the research also followed a clear pattern.357 Typically, the Center presented written research reports, sometimes in a

preliminary form, to the Advisory Committee within a year from the

date the Committee requested the work. For the twelve completed

projects described in Table 1, the average time from request to first

report was one year. The average time from request to final report

was 1.2 years. In all but one instance, the final report was delivered

within 1.5 years; in the one instance, the final report took two years.

Further use of the data continued, of course, as long as the Committee had a proposed rule under consideration. The Committee frequently asked researchers to address questions that arose during the

drafting and hearing stages. Databases created during the research

process presented a ready resource for responding to new questions.

Why did the Committee not request experimental research? The

conclusion seems unassailable that the time available to the

rulemakers for most empirical studies of matters of interest does not

356 In that study, Center researchers surveyed attorneys in cases that had settled

and compared their responses to those of attorneys in cases that had gone to trial.

Though the analysis is comparative, the assignment of cases to the two conditions is

not controlled, the vantage point is retrospective, and one cannot determine causal

relationships using such an approach. The research, of course, does not purport to

address causal relationships. See generally SHAPARD, supra note 35, at 3–5 (describing

survey construction, sampling, and analysis).

357 See supra tbl.1.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 69 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1189

permit the use of controlled experiments on a prospective basis. We

have few examples from which to extract empirical data about the

time needed to conduct experimental research. Anecdotal information from the studies of CAMP-type procedures indicate that experimental research can consume between three and ten years.358 Even

then, assuming that the empirical issues present close questions—

which should be the case, if the research and the consequent suspension of equal protection of the laws are to satisfy ethical norms discussed below359—the results may be inconclusive and variable from

one court setting or local legal culture to another.

As the rulemaking process has evolved, generally a request for

research arises around the time the Chair of the Advisory Committee

or a critical mass of its membership have decided that a particular rule

is a prime candidate for amendment. Such intuitions generally arise

from personal experience, from the urgings of an interested constituency, or from a sense of urgency created by congressional action that

might preempt present and future rulemaking. Whatever the source,

requesting research makes sense, either to test the feasibility of and

support for a particular measure, or to accumulate the information

necessary to craft a workable rule. In the long run, having the ability

to obtain specialized research information helps the rulemakers develop a recognized expertise in drafting rules. Such a reputation may

in turn solidify congressional commitment to the Rules Enabling Act

process.360

With requests for research coming from the Committee at the

same time as active efforts to revise the rules or to draft competing

statutes are underway, the pressure mounts to conduct what Center

researchers like to call “quick and clean” (as opposed to “quick and

dirty”) research. An additional source of pressure flows directly from

the brief terms that Advisory Committee members and especially the

chair serve. In recent years, members of the Committee have been

appointed to three-year terms that may be renewed once. Chairs have

358 Recent examination of case management principle under the Civil Justice Reform Act adds another anecdote. While the study was not a controlled experiment, it

required many of the elements of an experiment, including complex design issues,

selecting courts for inclusion and for comparison, and monitoring the progress of

cases through the system. That study took approximately six years from the date of

the statute to the date of publication of the results of the study of ten pilot courts. See

generally JAMES S. KAKALIK, TERENCE DUNWORTH, LAURAL A. HILL, DANIEL MCCAFFERY,

JUST, SPEEDY, AND INEXPENSIVE? AN EVALUATION OF JUDICIAL CASE MANAGEMENT UNDER

THE CIVIL JUSTICE REFORM ACT (1996).

359 See infra notes 383–93 and accompanying text (outlining the ethics of

experimentation).

360 See infra notes 376–79.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 70 28-MAY-02 12:35

1190 notre dame law review [vol. 77:4

generally been appointed to a three-year term, with a possible oneyear extension.361 The term of the chair roughly equals the cycle for

rulemaking. Thus, “[a] Chair with a three-year term . . . can see a

project through only if it commences at the outset of his or her tenure.”362 To fit within that cycle, research must be reported to the

Committee within a year.

The discovery project, discussed in Part II.H, represents a good

example of the timing necessary to complete a rule revision within the

term of a single chair. Judge Niemeyer started the process in October

1996, and the revised rule went into effect on December 1, 2000. The

additional year beyond the ordinary three-year term afforded the

chair and the Committee barely enough time to explore the issues,

commission research, and conduct a conference before drafting and

publishing rules. Conducting an experiment could not possibly fit

into that time frame. If experimentation is to be considered, new approaches will have to be devised, as will be discussed in Part IV.

B. Underlying Research Questions and Purposes

Research requests appear to have centered around three broad

types of questions. These questions typically relate to the purposes of

the undertaking. The three broad types of questions concern issues

relating to the frequency of certain practices, to the presence or absence of support for rule revision, and to the mechanical workings of

the current rules.

First, on several occasions, rulemakers have approached the

Center at the outset of a proposed rule change, though rarely before

the reporter has drafted a proposed amendment for internal Committee consideration. In this context, the Committee usually wishes to

obtain data that will help it decide whether the subject should be addressed in earnest. The Committee has asked how often a rule such as

Rule 12(b)(6) has been used successfully, how summary judgment activity has changed (or not) over time, how much satellite litigation

accompanies Rule 11 activity, how much contested protective order

activity occurs, how often judges allow attorneys to participate in voir

dire, and how often special masters serve pretrial or post-trial roles. In

those instances, the research product may have helped the Committee

361 A self-study subcommittee of the long-range planning committee of the Judicial Conference recommended that the term for chairs of the advisory rules committees be extended to five years. SELF-STUDY, supra note 12, reprinted in 168 F.R.D. 679,

697 (1995). The recommendation “was discussed with the Chief Justice on December

13, 1995.” Id., reprinted in 168 F.R.D. 679, 681 (1995).

362 Id., reprinted in 168 F.R.D. 679, 697 (1995).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 71 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1191

decide whether to proceed and, if it decided to continue, how to define which provisions of a rule should be amended.

Occasionally, as in the Rule 12(b)(6) context, research results

may provide a reason not to proceed at all. More often, though, the

rulemaking process appears to have built momentum by the time the

research has been completed, as in the Rule 56 proposals, and empirical support for or against the Committee’s approach may have little

effect.

Considerations of various sorts, ranging from pressure from

outside interest groups to the introduction of proposed legislation by

members of Congress, may have an impact on the Committee’s need

for empirical research, as appears to have been the case in the studies

relating to protective orders and voir dire practices. In those instances, the Committee used the research to refine the issues. Using

empirical information about the workings of a rule can contribute to

demonstrating specialized knowledge and objectivity that in turn may

help the Committee withstand challenges to its exercise of Rules Enabling Act authority.

Occasionally, the Committee has decided after more extensive

work, at times with the direct intervention of the Judicial Conference,

to abandon an effort, as it did with Rule 68 offers of judgment, protective orders, and attorney participation in voir dire. Research probably

had some impact on the Rule 68 decision, but interest group activity

in opposition to Committee proposals seems to have dominated the

protective order and attorney participation in voir dire decisions.

These experiences suggest limits in the ability of empirical research

findings to insulate empirically grounded proposals from challenges

arising from other interests and values. Empirical data generally

ground specific proposals in reality, but data sometimes fail to illuminate the primary values at stake or the primary reason for taking action or not.

Giving prominence to nonempirically-supported values is to be

expected in a rulemaking process in which interest groups play an

important role. Even in those instances in which findings are rejected

for unquantified—and perhaps unquantifiable—reasons, empirical

research serves a purpose. When rulemakers have decided to depart

from an empirically-driven proposal, they generally have seen themselves as obligated to articulate reasons for doing so. For example, in

modifying the disclosure requirements of Rule 26(a)(1) despite empirical research validating their effectiveness, the Committee chose to
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 72 28-MAY-02 12:35

1192 notre dame law review [vol. 77:4

draft a proposal that might achieve a consensus and produce uniformity of national rules.363

Central to the purpose of conducting empirical research on disputed or doubtful propositions is the risk of turning up data that are

unwelcome. At times, such data may be incompatible with proposals

that the Committee developed before the research ended. Typically,

the research and the Committee’s deliberations proceed simultaneously. Research data may or may not move the Committee away from

a proposal.

On at least one occasion, opponents of a proposed, published

rule cited Center research in opposition to the Committee’s proposal.

That instance involved the so-called “just ain’t worth it” proposal regarding class actions, which would have called for judges to consider

“whether the probable relief to individual class members justifies the

costs and burdens of class litigation.”364 Numerous witnesses opposed

that provision. A small fraction of the opponents cited the FJC’s finding that the median recovery per class member ranged from $315 to

$528 to support their arguments that most class actions provided nontrivial recoveries to individual class members.365 The Committee

abandoned the proposal.

Second, the Committee has with increasing frequency asked the

Center to develop a nuanced examination of how a particular rule

operates in the civil litigation context. These requests seem designed

to assist the Committee in deciding how, not whether, to amend a

rule. Examples include the Center’s studies of Rule 11 sanctions, class

actions, discovery, and special masters.366 On each occasion, the reporter has specified thoughts about the direction in which reform

might proceed and provided a roadmap for the research. Given this

degree of coordination, it is not surprising that many, but certainly

not all, of the research findings fit into the rulemaking plan.367 Occasionally, as with the questions dealing with overlapping or duplicative

363 See supra notes 280–85 and accompanying text.

364 1996 PROPOSED RULES, supra note 255, reprinted in 167 F.R.D. 560, 559 (1996);

see also supra notes 242–44 and accompanying text (discussing the FJC finding regarding the size of claims).

365 For the reporter’s summary of those comments, see 1 WORKING PAPERS, supra

note 229, at 331–50. See, for example, the summary of Alan Black’s statement: “The

FJC study shows that if the ‘$2’ claims class exists, it is very rare.” Id. at 332.

366 See, e.g., FJC DISCOVERY PRACTICE, PROBLEMS, AND PROPOSALS, supra note 273;

FJC SPECIAL MASTERS’ STUDY, supra note 44; FJC Disclosure & Discovery Study, supra note

54.

367 Cf. Maurice Rosenberg, Federal Rules of Civil Procedure in Action: Assessing Their

Impact, 137 U. PA. L. REV. 2197, 2211 (1989) (discussing the importance of “the kind

of close working relationship between the researchers and the Rules Committee that
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 73 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1193

class actions, available research tools may not always be able to match

the sweep of the empirical questions.

In discussing rules proposals that stirred serious challenges, one

should not overlook a function that contextual research may play in

responding to such challenges. Empirical research may give the Committee objective data to cite in support of its proposals. The Committee’s use of Center data on initial disclosure to support the

amendments to the discovery rules that went into effect in 2000 represents a prime example, as discussed above.368 Though Rule 11

changes and class action proposals stirred heated debates, empirical

research presented answers to some objections from interest groups.

Third, sometimes the Committee has asked the Center to conduct research that appears to be designed to test the support for a

given proposal or a general approach to revising a rule. For example,

in asking the Center to survey all federal district judges to elicit their

views on Rule 11 sanctions, the Committee sought and obtained a

product that might have proved to be useful in steering its way

through a thicket of competing interests. At the least, no district

judge could argue that his or her views were not solicited.369 Similarly, in considering the question of whether to expand attorney participation in voir dire, the Committee was caught between competing

interests of district judges and influential members of Congress. Consulting the judges and documenting their practices seemed a necessary, but as it turned out not a sufficient, step in seeking Judicial

Conference support.370

Research on Rule 68 offers of judgment provides another example in which a specific proposal needed the test of outside review. Exploring the effects of a complicated formula in the context of an

already complex set of fee-shifting rules seemed indispensable to further action. A controlled field experiment may have been a viable way

to test the effects of the proposal. In the absence of such an experiment, uncovering attorneys’ strong views for and against the proposals

the discovery project had” and indicating that “[f]or the most productive results, that

type of relationship is essential”).

368 See supra Part II.H.4.a.

369 See supra notes 131–37, 141–44 and accompanying text (analyzing two surveys

of judges). Ironically, dissenting justices argued that the committee had ignored

those views. See supra note 135. In hindsight, however, the 1993 amendments appear

to have reflected accurately the views of most district judges on all but one point. See

FJC 1995 RULE 11 SURVEY, supra note 141, at 2 (listing the single exception as a majority preference for the purpose of Rule 11 sanctions to include compensation of parties injured by Rule 11 as well as deference).

370 See supra notes 209–22 and accompanying text (discussing voir dire findings

and outcome).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 74 28-MAY-02 12:35

1194 notre dame law review [vol. 77:4

may have provided the information the Committee needed. The

Committee had once before withdrawn a Rule 68 amendment after

intense partisan criticism,371 and the Center’s research helped corroborate the likelihood of a recurrence. We are left to speculate whether

the interest groups might have altered their position had there been

clear empirical data derived from a controlled experiment.

What impact has empirical research had? Do the rulemakers find

it useful? That itself is an empirical question, best addressed by an

objective observer outside the Center. The case summaries in Part II

shed considerable light on the issue by using references to empirical

work found in the minutes of meetings and in the official Committee

notes. Based on those objective references and my subjective impressions, the following represents a starting point for assessing the impact

of empirical research in the rulemaking process.

In general, empirical research appears to have had a substantial

impact on the drafting of amended rules. As noted earlier, empirical

research also has had an impact on Committee decisions on whether

to proceed, in whole or in part, with proposed rule revisions. Turning

to the fourteen instances in which the Advisory Committee requested

that the Center conduct research, six of the rulemaking efforts ended

with a Committee decision not to proceed with the proposed rulemaking.372 Of the remaining eight requests (involving five rules), four

studies provided data that were used in drafting amendments that

went into effect.373 Three studies and part of another relate to pending rules proposals regarding class actions, special masters, and electronic discovery.

For example, empirical research has had, in my opinion, a substantial impact in the discovery rule revision that took effect on December 1, 2000. The highest impact seemed to be in the area of

disclosure. The Center’s findings appeared to counter the misimpression, based on presentations from some interest groups, that disclosure rules were not working well. In addition, Center findings on

problems of discovery, which appeared to be concentrated in the 5%

of cases with the highest stakes and amount of discovery, dovetailed

with the Committee’s approach to identifying the relatively few cases

371 See Rowe & Vidmar, supra note 175, at 15; Roy D. Simon, Jr., The Riddle of Rule

68, 54 GEO. WASH. L. REV. 1, 10–16 (1985).

372 Those subjects were Rule 12(b)(6) motions to dismiss, Rule 56 (summary judgment), reexamination of 1993 amendments to Rule 11 sanctions, Rule 26(c) (protective orders), Rule 68 (offers of judgment), and Rule 47(a) (attorney participation in

voir dire). See supra Parts III.A, III.D, III.E, and III.F.

373 Those subjects were Rule 11 (two studies), Rule 23(f) (part of one study), and

Rules 26–37. See supra Parts III.C and III.F.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 75 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1195

that might require judicial management of discovery. In addition,

Center data on the working of discovery limitations provided an ongoing resource that the reporters and members called on to inform decisions about where to draw the line. This was particularly true

regarding the length of depositions. The Committee also used Center

data on the numbers of depositions and the absence of reported

problems with the number of depositions as part of its decision to

leave existing limits intact.

Center research has had a notable impact in certain other areas,

sometimes confirming a direction the Committee had already chosen

and perhaps contributing data that refined proposals already formulated. For example, in the Rule 11 area, Center findings seemed for

the most part to confirm conventional wisdom about satellite litigation, monetary sanctions, and disparate impact on plaintiffs. Data on

the absence of hearings in a substantial number of cases may have led

to more explicit hearing requirements, a matter that otherwise might

have been left unstated. In the attorney voir dire, class action, and

special master areas, Center data appear to have had an identifiable

effect, as described in Part II.

Of the projects in which Center research has delved into the context of how a rule operates or a proposed rule might operate, only the

study of Rule 68 offers of judgment might be considered to have had a

low impact. That, however, seems to be a hindsight judgment based

on knowledge that the Committee abandoned the proposal. The FJC

study did provide the Committee with information supporting its decision not to proceed in the face of strong opposing forces on both

sides, a contribution that seems significant.

The Advisory Committee has been selective in requesting empirical research. During the time span covered by this Article, numerous

changes recommended by the Committee went into effect without

special empirical study.374 Clearly, the Committee has exercised judg374 Such changes include nontrivial revisions to: Rule 4 regarding service of process (120 S.Ct. 50, 72–74 (1999); 113 S.Ct. 88, 92–105 (1992)); Rule 5 regarding certificate of service, filing by facsimile, and filing of disclosures (120 S.Ct. 50, 74 (1999);

116 S.Ct. 74, 82 (1995); 111 S.Ct. 579, 683–85 (1990)); Rule 9 regarding admiralty

provisions (117 S.Ct. 76, 85 (1996)); Rule 12(a) regarding presentation of motions to

dismiss (120 S.Ct. 50, 74 (1999); 113 S.Ct. 88, 280–82 (1992)); Rule 15 regarding

amended pleadings (111 S.Ct. 579, 689–90 (1996)); Rule 16 regarding pretrial conferences (113 S.Ct. 88, 284–88 (1992)); Rule 26 regarding initial disclosures (113

S.Ct. 88, 293–314 (1992)); Rule 43(a) regarding form of taking testimony (116 S.Ct.

74, 83–84 (1995)); Rule 44(a)(2) regarding proof of foreign records (111 S.Ct. 579,

701–10 (1990)); Rule 45 regarding subpoenas (111 S.Ct. 579, 711–22 (1990)); Rule

47(b) regarding alternate jurors (111 S.Ct. 579, 730–31 (1990)); Rule 50(a) regarding

judgment as a matter of law (111 S.Ct. 579, 733–38 (1990)); Rule 54(d) regarding
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 76 28-MAY-02 12:35

1196 notre dame law review [vol. 77:4

ment in calling on the FJC’s limited resources. No explicit criteria

have been developed for distinguishing rulemaking proposals that

need careful study and those that do not. Nor does there appear to

be a need to establish criteria that might crimp the development of an

evolving process. Suffice it to say that by reviewing the research requests at a general level, the primary factors appear to be the paucity

of empirical information about a subject, and a need to know more

about the empirical context in which a complex rule operates.375

C. A Note on Legislation

To the degree that basing procedural rules on empirical analysis

is an important value for our civil justice system, the experiences described in this Article strongly suggest that the rulemaking process has

a decided advantage over the legislative process. Congressional deference to the rulemaking process appears to have been the dominant

mode for dealing with the proposals discussed in this Article. Of the

fourteen rulemaking instances examined in this Article, ten involved

legislative activity at some stage of the rulemaking process.376 In only

one instance during the period covered by this Article (1988–2001)

did Congress adopt legislation—the Private Securities Litigation Reform Act377—that altered the operation of an existing rule. In that

instance, Congress created special class action rules, but limited the

operation of the new rules to the subject matter of private securities

litigation.378

Of course, empirical analysis of the questions deemed most important by the rules committees should not be and does not appear to

have been the sole determinant of whether Congress might defer to

attorneys’ fees (113 S.Ct. 88, 384–86 (1992)); Rule 83(a) regarding local rules (115

S.Ct. 71, 83–84 (1994)); Rule B regarding attachment and garnishment (120 S.Ct. 50,

77–81 (1999)); and Rule C regarding in rem actions (120 S.Ct. 50, 77–81 (1999); 111

S.Ct. 579, 760–62 (1990)).

375 Those implicit criteria and others may be gleaned from Cooper, supra note

237, and Cooper, supra note 317.

376 There was no evidence that Congress had conducted or commissioned any

empirical examination of the issues that were pending simultaneously before Congress and the Advisory Committee. A closer look at the legislative process might reveal such research, but none appears to have been communicated to the rulemakers

and none was uncovered by this author in the course of conducting research for the

Advisory Committee on many of the subjects discussed in Part II above.

377 Pub. L. No. 104-67, 109 Stat. 737 (1995) (codified as amended at 15 U.S.C.

§ 77a (1994)). The statute modified the conditions for management and certification

of class actions in securities litigation, altering the application of Rule 23 to such

cases.

378 Id.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 77 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1197

new or amended rules. For many reasons, including a congressional

commitment to the Rules Enabling Act process that it created, Congress might be expected to defer to the rulemaking process, at least in

areas that fall fairly clearly on the “procedural” side of the long recognized if difficult-to-apply “procedural-substantive” distinction within

the rulemaking process.379 Any deference shown by Congress cannot

be attributed solely to the rulemaking body’s capacity to commission

and use empirical research. Congress, no less than the rules committee, might be expected to rely on personal experiences and impressions communicated by colleagues, witnesses, and others. As

discussed above,380 Congress came close to rejecting the proposed

amendments to Rule 11 sanctions, which were based on empirical

data, and to Rule 26 regarding initial disclosure in 1993, which were

not based on empirical data. In those instances, the proposed congressional action did not appear to turn on the presence or absence of

empirical research.

IV. EXPERIMENTAL RESEARCH IN THE RULEMAKING PROCESS

To recapitulate, commentators have called for more experimental research to support the rulemaking process.381 Dramatic increases

in empirical research to support the rulemaking process have occurred in the past decade or so,382 but none of that empirical work

involved experimental research. Yet, experimental research indisputably affords the most powerful instrument for examining legal procedures, the only design that many researchers rely on to support causal

inferences. This Part discusses some of the limits of experimental research before presenting a proposal for a revision of the Rules Enabling Act designed to make the rulemaking process more open to

experimental research.

379 See generally Stephen B. Burbank, The Rules Enabling Act of 1934, 130 U. PA. L.

REV. 1015 (1982) (examining the development of rulemaking under the Rules Enabling Act).

380 See supra notes 137–44 and accompanying text.

381 See supra Part II.F, notes 69–77 and accompanying text.

382 In 1988, Professor Laurens Walker counted thirty empirical studies related to

rulemaking that had been conducted during the first fifty years of the Federal Rules

of Civil Procedure, a rate of slightly more than one study every other year. Walker,

supra note 8, at 69–70. During the twelve years since then, the Center itself has conducted at least twelve studies directly related to the federal rules, a rate of one per

year, and certainly other research institutions have contributed a considerable number of studies. For example, see the studies supported by RAND and the American

Judicature Society in supra notes 16, 41, 123, and 358; see also sources cited supra note

25 (discussing laboratory and other research).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 78 28-MAY-02 12:35

1198 notre dame law review [vol. 77:4

Aside from the time and cost required for experimental research,

matters of principle limit the judicial branch’s willingness and ability

to use these designs. A fundamental premise of our legal system—

embedded in the Equal Protection Clause, expressly in the Fourteenth Amendment and implicitly in the Fifth Amendment, as well as

in our collective notions of fundamental fairness383—is that like cases

will be treated in like manner. To examine closely the conditions

under which legal experimentation might legitimately take place,

Chief Justice Warren Burger, serving in his capacity as Chairman of

the Board of the FJC, appointed a blue-ribbon Advisory Committee on

Experimentation in the Law.384 The Chief Justice directed the Committee to examine “the propriety, value and effectiveness of controlled experimentation for evaluating innovations in the justice

system.”385

The Center’s Advisory Committee produced a comprehensive

analysis of the conditions under which experimentation might be considered and the terms under which it might be implemented. Without going into the intricacies of the ethical analyses, it is useful to

restate the preconditions posited by the Committee before a judicial

body would be justified in authorizing an experiment involving a component of the legal system:

(1) The current rule or practice “must either need substantial improvement or be of doubtful effectiveness” and should not simply be an experiment “for experiment’s sake.”386 Amendments

to conform the rules to contemporary practices or to ordain the

best practices for judges to follow probably would not qualify

under that standard.387 Nonetheless, such rule changes might

benefit from empirical research to identify and examine the

practices in a litigation context and search for potential side

effects. If the practice has been adopted in many courts, there

is less concern about delaying a good idea but greater concern

383 See Rosenberg, supra note 3, at 16–17. This is not to say, of course, that our

legal system, based as it is on federalism, does not tolerate differences between state

and federal procedural rules that allow like cases to be treated differently.

384 See FED. JUDICIAL CTR., supra note 20, at 79.

385 Id.

386 Id. at 11.

387 See, for example, the 2001 proposals to change Rule 23 (c)(1)(A)’s treatment

of the timing of class certification activity. The Committee note describes the change

as “consistent with the reality that courts generally make certification decisions after

the deliberation required for a sound decision, as shown by the Federal Judicial

Center figures” and also as “consistent with best practice.” 2001 PROPOSED RULES,

supra note 232, reprinted in 201 F.R.D. 560, 592 (2001); see also supra notes 250–54

(discussing “as soon as practicable” findings).
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 79 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1199

about denying the benefits of that idea in an unknown number

of cases.

(2) There must be “significant uncertainty about the value of the

proposed innovation.”388 Just as in medical research, one

would not withhold a known successful treatment from a sick or

dying patient, one should not withhold a known beneficial legal

procedure from a litigant. For example, once the Second Circuit determined that CAMP produced beneficial results without

countervailing detriments, the court adopted the program.389

Some courts determined that the research results from the

CAMP experiment sufficed to justify adoption of similar programs. In the Sixth Circuit, an earlier evaluation of an existing

program had “proved inconclusive,” and the court decided to

pursue an experiment to test the benefits of the program in

relationship to its costs.390 When a procedure is mandated by

the Constitution or by a statute, adopting a rule to implement

that procedure need not be tested by experiment.

(3) There must be “no other practical means to resolve uncertainties about the effectiveness of the proposed innovation.”391 In

other words, “if essential information can be obtained satisfactorily through simulation or other forms of research that do not

directly affect the operation of the justice system, considerations of ethics . . . militate against a program experiment.”392

For example, obtaining information about the operation of the

discovery and disclosure systems by surveying counsel or judges

in recently terminated cases might be seen as a less intrusive

alternative to employing experimental designs.393

(4) Finally, “the experiment must seriously be intended to inform a

future choice between retaining the status quo or implementing the innovation.”394 If adoption of the proposed rule appears not to be feasible because of considerations of cost,

political opposition, or other impediments, conducting an experiment should not be used to temporize or otherwise delay

an inevitable decision to adopt or reject the rule. On the other

hand, if conducting a needed experiment requires several years

of study, opposition to the merits of the proposed innovation

388 FED. JUDICIAL CTR., supra note 20, at 11.

389 EAGLIN, supra note 31, at 1.

390 Id. & n.1 (adopting Local Rule 18).

391 FED. JUDICIAL CTR., supra note 20, at 11.

392 Id.

393 See supra notes 272–74 (discussing FJC research methods in discovery and disclosure study).

394 FED. JUDICIAL CTR., supra note 20, at 11.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 80 28-MAY-02 12:35

1200 notre dame law review [vol. 77:4

should not require the proponents to draw back from an experiment to test the merits.

Thus, the conditions under which rulemakers might employ experimental research limit the likelihood of its use. In addition to being costly and time-consuming, experimental research must clear

ethical hurdles that are not insignificant. The cumulative effect of

those constraints may help explain the infrequency of its use.

Part III included discussion of the institutional limits on longterm research imposed by the brief terms served by members and

chairs of advisory committees. In the research summarized in Part II,

the Committee seemed to have a clear vision of the types of changes

that would be desirable. When the chair and members of a committee have such clear vision of the need for change and the types of

changes that might work, current methods of “quick and clean” observational survey research seem appropriate and to have served the

Committee well. When, however, an innovative idea has unclear consequences, the Committee may wish to adopt an experimental

approach.395

The proposal considered recently by the Advisory Committee to

create a simplified set of procedures for specific sets of cases presents

an example of this type of innovation. The proposal would apply

truncated pretrial procedures and an early, firm trial date to cases

seeking less than a specified amount of damages.396 Some of the

many unknowns about such a proposal would be how many cases

might come within any mandatory categories, how many people

would use it within any voluntary categories, and what problems litigants, attorneys, and courts might encounter when using it. Research

designs traditionally used to support the rulemaking processes would

not adequately detect such problems. Survey research would of necessity present totally hypothetical questions and would most likely re395 The Rule 68 offer of judgment proposals, for example, may also have been

good candidates for experimental testing. The method of randomizing and controlling the application of the proposed rule would require some creative design work,

but the proposals could have undoubtedly benefited from field experience. Coming

up with an outcome measure that would distinguish between coerced and voluntary

settlements would also require creative design work. Forum-shopping issues, of

course, always plague such research, posing analytical problems about whether random selection of participants was successfully implemented. Had these methodological problems been overcome, experimental testing and reliable data might have

softened some of the opposition that Rule 68 proposals have generated. See supra

notes 194–99 and accompanying text (discussing Rule 68 proposals and outcome).

Proposed electronic discovery rules may also be candidates for experimental testing.

See supra note 256 and accompanying text.

396 See Civil Rules Advisory Comm., supra note 325, at 32–38.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 81 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1201

present an attempt to quantify variables that are primarily matters of

opinion at best, conjecture at worst. There appear to be no existing

systems in the federal courts that employ the proposed package of

rules.

In the Committee’s discussion about simplified procedures, this

author raised the question of implementing the proposed procedures

on an experimental basis.397 To date, the Committee has not pursued

that approach, perhaps for the same reasons it decided not to pursue

the simplified procedure mechanism at all. The item remains on the

Committee’s agenda as of this writing. Assuming for purposes of

discussion that the Committee might wish to test its proposed innovation, two fundamental reasons might inhibit pursuing an experimental approach. The first is that there does not appear to be any explicit

authority for the rulemakers to issue rules that are of less than national scope. The second is that there is no mechanism in place for

creating and administering national experiments.

As to the question of authority, the Rules Enabling Act vests in

the judicial branch “the power to prescribe general rules of practice

and procedure.”398 The core meaning of the term “general” would

seem to preclude having inconsistent rules apply within different

courts.399 Yet such inconsistent application of rules is central to experimental testing of proposed rules under controlled conditions.

An argument can be made that the inherent power of the courts

to govern their own affairs includes the power to create experimental

rules that are being tested in a limited number of districts.400 In effect, the 1993 “opt-out” provisions of Rule 26(a) created such a situa397 See id. at 36–37.

398 28 U.S.C. § 2072(a) (1994).

399 See Paul D. Carrington, Making Rules To Dispose of Manifestly Unfounded Assertions: An Exorcism of the Bogy of Non-Trans-Substantive Rules of Civil Procedure, 137 U. PA.

L. REV. 2067, 2079 (1989) (“‘[G]eneral’ should be presumed to mean that rules . . .

should not be limited in their application . . . to a particular geographic area.”).

400 For an argument that Article III of the Constitution vests general rulemaking

power in the federal courts, see Linda S. Mullenix, Unconstitutional Rulemaking: The

Civil Justice Reform Act and Separation of Powers, 77 MINN. L. REV. 1283 (1993). For a

contrary view, see Martin H. Redish, Federal Judicial Independence: Constitutional and

Political Perspectives, 46 MERCER L. REV. 697, 724–31 (1995), and for a rebuttal, see

Linda S. Mullenix, Judicial Power and The Rules Enabling Act, 46 MERCER L. REV. 733

(1995). This debate took place in the context of the Civil Justice Reform Act of 1990,

28 U.S.C. §§ 471–482 (1994), in which Congress explicitly authorized local district

courts to study various approaches to reducing costs and delays. Congress also explicitly created pilot and demonstration courts that were to study specific programs in a

systematic way. See §§ 104–105 of the Judicial Improvements Act of 1990, Pub. L. No.

101-650, 104 Stat. 5097.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 82 28-MAY-02 12:35

1202 notre dame law review [vol. 77:4

tion, albeit without any national control over the local decisionmaking

process. The inherent authority argument, however, calls for acting

in the face of the statutory limit that rules be of general applicability.

That, in turn, raises the question of whether Congress has the power

to limit the rulemaking authority,401 a point that is beyond the scope

of this Article.

An argument can also be made that the Judicial Conference has

the power to create experimental rules as part of its statutory duty to

“carry on a continuous study of the operation and effect of the general rules of practice and procedure now or hereafter in use . . . .”402

Indeed, the “now or hereafter in use” language suggests authority to

study proposed rules that the Judicial Conference has not put into

effect.

On the other hand, commentators have urged that creating an

explicit process for testing experimental rules should be done in conjunction with Congress, through legislative action.403 Given the overlap in power and jurisdiction over procedural issues between Congress

and the rulemakers, creating explicit statutory authority to conduct

experiments appears to be the prudent course. The examples of overlapping rule and statutory proposals encountered in Part II reveal that

Congress has repeatedly deferred to the Rules Enabling Act process it

created. Congress would presumably be more likely to defer to the

experimental process if it had played a role in its creation than if the

courts were to proceed unilaterally.

As to the question of the mechanism for adopting and monitoring experimental rules, no organized system exists. Commentators

have presented proposals to create such a system,404 but such proposals have posited local rules and individual local district courts as the

catalysts and proving grounds for experimentation. Indeed, many—

perhaps too many—innovations originate at the district and even the

chambers level. The question, however, is whether such uncontrolled

“experiments” should be adopted on a national level. Local districts

or chambers are not in a position to evaluate the workings of their

rules under controlled conditions. As Professor Leo Levin, former Director of the FJC, observed, district courts may be serving as “very busy

401 See generally Burbank, supra note 379 (examining congressional authority over

rulemaking).

402 28 U.S.C. § 331.

403 See, e.g., Levin, supra note 11, at 1585–86 (discussing a proposal for experimentation by local rule); Tobias, supra note 11, at 286 (discussing a proposal for Judicial

Conference approval of experimental local rules).

404 See, e.g., Levin, supra note 11, at 1590–93 (commentary on assuring productive

systems); Walker, supra note 8, at 76.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 83 28-MAY-02 12:35

2002] empirical research in civil rulemaking 1203

laboratories . . . but virtually no one is collecting data.”405 In addition,

data from local courts that have voluntarily adopted an innovation

may have acted out of an idiosyncratic local legal culture. Other

courts may not see themselves as similarly situated. Data from such

local experiments may simply not be applicable on a national level.

Professor Levin recommended creating a new committee of the

Judicial Conference to supervise and administer a program of experimentation that would involve both reviewing proposals brought to the

committee as well as pursuing “initiatives suggested by the committee.”406 Taking Professor Levin’s proposal to its logical conclusion

would entail creating or empowering a national body, a component of

the Judicial Conference, to formulate and adopt experimental rules to

be tested on a national level. Because the function of reviewing and

approving experimental rules rests so heavily on skills already used

effectively by the Standing Committee with the assistance of the advisory committees, the proper locus of the power would seem to reside

in the Standing Committee. Designing a new entity seems likely lead

to competing claims for authority to draft and promulgate national

rules.

Vesting the power to review proposed experiments in the Standing Committee, or any national body would be likely to exacerbate the

tension between the value of uniform national rulemaking and the

value of autonomy for local courts to administer and manage their

own caseloads. Randomly assigning a court to test an experimental

treatment can be expected to impose demands on that court’s resources. Those demands may lead those courts to seek to opt out of a

proposed experiment. Opting out, however, defeats the principle of

random assignment. Without attempting to resolve this dilemma, it is

important to note that the federal courts’ tradition of local autonomy

represents another hurdle to conducting experimental research on a

national level.

As shown earlier, expressions of interest in experimental testing

of proposed rules have been rare and episodic to date. Ethical constraints circumscribe the opportunities for experimentation even further. There is no reason to expect that a mechanism for creating

405 Levin, supra note 11, at 1581–82. Professor Tobias suggests congressional

adoption of a 1991 proposal to amend Rule 83 to permit experimental local rules if

approved by the Judicial Conference. See Tobias, supra note 11, at 286. For the proposed 1991 amendment to Rule 83, which was not adopted, see COMM. ON RULES OF

PRACTICE AND PROCEDURE OF THE JUDICIAL CONFERENCE OF THE U.S., PROPOSED RULES,

reprinted in 137 F.R.D. 53, 152–55 (1991). A proposed experimental local rule would

be limited to a period of five years or less. Id., reprinted in 137 F.R.D. 53, 153 (1991).

406 Levin, supra note 11, at 1589–90.
\\Server03\productn\N\NDL\77-4\NDL404.txt unknown Seq: 84 28-MAY-02 12:35

1204 notre dame law review [vol. 77:4

national rule-based experiments would be used so frequently that it

would tax the capacity of the Standing Committee and its advisory

committees. Accordingly, what is needed is a statute that would vest

the power to create experimental rules in the Standing Committee. If

the demand exceeds the capacity of the Standing Committee, that

would be the appropriate time to consider other alternatives.

In summary, moving from the rhetorical calls for experimental

research into the reality of implementing experimental research demands an operational mechanism for suggesting, reviewing, implementing, and monitoring rule-based experiments.407 Such a

mechanism needs to be created at a national level and to have expertise in rulemaking. With appropriate authority from Congress and assistance from the FJC and the Administrative Office of the United

States Courts, the Standing Committee could serve those functions.

In a common-law fashion, the Committee would develop expertise in

identifying appropriate subjects for experimentation, establishing

means of randomly selecting courts to participate, reviewing proposed

research designs, selecting control groups, setting time limits on the

experiments, and reviewing the data. Such a mechanism could draw

on infrequently-used experimental research tools that can enhance

our knowledge of the impact of proposed rules on litigants, attorneys,

and courts.

Creating such a mechanism will require that rulemakers shift

their expectations about the timing of major rules changes. Members

and chairs might serve as catalysts for long-term projects that their

successors will pick up and conclude. This change in roles would allow the committees, in appropriate situations, to contemplate calling

for experimental testing of controversial proposals before enacting a

new national rule. Having such a mechanism would present another

option for coping with proposed rules changes, proposals that offer

no empirical basis for conclusions about their expected effects.

Reserving experimental study for significant and debatable rule

changes coincides with ethical limitations on experimentation in the

law. Using experimentation sparingly also promotes conservation of

scarce rulemaking and research resources. On those occasions when

the Standing Committee finds experimental research on proposed

rules appropriate, the end result should be highly likely to improve

both the quality of rules adopted and the quality of our knowledge of

the justice system.

407 See supra notes 358–64 and accompanying text.