Federal Judgeships: The General Accuracy of District and Appellate Judgeship Case-Related Workload Measures
Released: 2009-09-30T00:00:00

Letter

Case Weights Are Intended to Measure Judicial Time Required to Handle Their Caseloads

The demands on judges’ time are largely a function of both the number and complexity of the cases on their dockets. To measure the case-related workload of district court judges, the Judicial Conference has adopted weighted case filings. The purpose of the district court case weights was to create a measure of the average judge time that a specific number and mix of case filed in a district court would require. Importantly, the weights were designed to be descriptive not prescriptive—that is, the weights were designed to develop a measure of the national average amount of time that judges actually spent on specific cases, not to develop a measure of how much time judges should spend on various types of cases. Moreover, the weights were designed to measure only case-related workload. Judges have noncase-related duties and responsibilities, such as administrative tasks, that are not reflected in the case weights.

With few exceptions, such as cases that are remanded to a district court from the court of appeals, each civil and criminal case filed in a district court is assigned a case weight. For example, in the 2004 case weights, drug possession cases are weighted at 0.86 while civil copyright and trademark cases are weighted at 2.12. The total annual weighted filings for a district are determined by summing the case weight associated with all the cases filed in the district during the year. A weighted case filings per authorized judgeship is the total annual weighted filings divided by the total number of authorized judgeships. For example, if a district had total weighted filings of 4,600 and 10 authorized judgeships, its weighted filings per authorized judgeships would be 460. The Judicial Conference uses weighted filings of 430 or more per authorized judgeship as an indication that a district may need additional judgeships. Thus, a district with 460 weighted filings per authorized judgeship could be considered for an additional judgeship. However, the Judicial Conference does not consider a district for additional judgeships, regardless of its weighted case filings, if the district does not request any additional judgeships.

1993 Case Weights Reasonably Accurate, But Accuracy of 2004 Case Weights Cannot Be Statistically Determined

In our 2003 report, we found the district court case weights approved in 1993 to be a reasonably accurate measure of the average time demands a specific number and mix of cases filed in a district court could be expected to place on the district judges in that court. The methodology used to develop the weights used a valid sampling procedure, developed weights based on actual case-related time recorded by judges from case filings to disposition, and included a measure (standard errors) of the statistical confidence in the final weight for each weighted case type. Without such a measure, it is not possible to objectively assess the accuracy of the final case weights.

At the time of our 2003 report, the Subcommittee on Judicial Statistics of the Judicial Conference’s Judicial Resources Committee had approved the research design for revising the 1993 case weights, with a goal of having new weights submitted to the Resources Committee for review in the summer of 2004. The design for the new case weights relied on three sources of data for specific types of cases: (1) data from automated databases identifying the docketed events associated with the cases; (2) data from automated sources on the time associated with courtroom events for cases, such as trials or hearings; and (3) consensus of estimated time data from structured, guided discussion among experienced judges on the time associated with noncourtroom events for cases, such as reading briefs or writing opinions.

According to the FJC, the Subcommittee wanted a study that could produce case weights in a relatively short period of time without imposing a substantial record-keeping burden on district judges. The FJC staff provided the Subcommittee with information about various approaches to case weighting, and the Subcommittee chose an event-based method—that is, a method that used data on the number of and types of events, such as trials and other evidentiary hearings, in a case. The design did not involve the type of time study that was used to develop the 1993 case weights. Although the proposed methodology appeared to offer the benefit of reduced judicial burden (no time study data collection), potential cost savings, and reduced calendar time to develop the new weights, we had two areas of concern—the challenge of obtaining reliable, comparable data from two different data systems for the analysis and the limited collection of actual data on the time judges spend on cases.

First, the design assumed that judicial time spent on a given case could be accurately estimated by viewing the case as a set of individual tasks or events in the case. Information about event frequencies and, where available, time spent on the events would be extracted from existing administrative data bases and report and used to develop estimates of the judge-time spent on different types of cases. For event data, the research design proposed using data from two data bases (one of which was new and had not been implemented in all district courts) that would have to be integrated to obtain and analyze the event data. The FJC proposed creating a technical advisory group to address this issue.

Second, the research design did not require judges to record time spent on individual cases. Actual time data would be limited to that available from existing data bases and reports on the time associated with courtroom events and proceedings for different types of cases. However, a majority of district judges’ time is spent on case-related work outside the courtroom. The time required for noncourtroom events would be derived from structured, guided discussion of groups of 8 to 13 experienced district court judges in each of the 12 geographic circuits (about 100 judges in all). The judges would develop estimates of the time required for different events in different types of cases within each circuit using FJC-developed “default values” as the reference point for developing their estimates. These default values would be based in part on the existing case weights and in part on other types of analyses. Following the meetings of the judges in each circuit, a national group of 24 judges (2 from each circuit) would consider the data form the 12 circuit groups and develop the new weights.

The accuracy of judges’ time estimates is dependent upon the experience and knowledge of the participating judges and the accuracy and reliability of the judges’ recall about the average time required for different events in different types of cases—about 150 if all the case types in the 1993 case weights were used. These consensus data could not be used to calculate statistical measures of the accuracy of the resulting case weights. Thus, the planned methodology did not make it possible to objectively, statistically assess how accurate the new case weights are—weights whose accuracy the Judicial Conference relies upon in assessing judgeship needs.

We noted that a time study conducted concurrently with the proposed research methodology would be advisable to identify potential shortcoming of the event-based methodology and to assess the relatively accuracy of the case weights produced using that methodology. In the absence of a concurrent time study, there would be no objective statistical way to determine the accuracy of the case weights produced by the proposed event-based methodology—a major difference with the methodology used to develop the 1993 case weights.

Accuracy of Courts of Appeals Case-Related Workload Measure Cannot Be Assessed

The principal quantitative measure the Judicial Conference uses to assess the need for additional courts of appeals judgeships is adjusted case filings. The measure is based on data available from standard statistical reports for the courts of appeals. The adjusted filings workload measure is not based on any empirical data regarding the time that different types of cases required of appellate judges.

The Judicial Conference’s policy is that courts of appeals with adjusted case filings of 500 or more per three-judge panel may be considered for one or more additional judgeships. Courts of appeals generally decide cases using constantly rotating three-judge panels. Thus, if a court had 12 authorized judgeships, those judges could be assigned to four panels of three judges each. In assessing judgeship needs for the courts of appeals, the Conference may also consider factors other than adjusted filings, such as the geography of the circuit or the median time from case filings to disposition.

Essentially, the adjusted case filings workload measure counts all case filings equally, with two exceptions. First, cases refilled and approved for reinstatement are excluded from total case filings. Second, pro se cases— defined by the Administrative Office of the U.S. Courts as cases in which one or both of the parties are not represented by an attorney—are weighted at 0.33, or one-third as much as other cases, which are weighted at 1.0. For example, a court with 600 total pro se case filings in a year would be credited with 198 adjusted pro se case filings (600 x 0.33). Thus, a court of appeals with 1,600 filings (excluding reinstatements)—600 pro se cases and 1,000 non-pro se cases—would be credited with 1,198 adjusted case filings (198 discounted pro se cases plus 1,000 non-pro se cases). If this court had 6 judges (allowing two panels of 3 judges each), it would have 599 adjusted case filings per 3-judge panel, and, thus, under Judicial Conference policy, could be considered for an additional judgeship.

The current court of appeals workload measure represents an effort to improve the previous measure. In our 1993 report on judgeship needs assessment, we noted that the restraint of individual courts of appeals, not the workload standards, seemed to have determined the actual number of appellate judgeships the Judicial Conference requested. At the time the current measure was developed and approved, using the new benchmark of 500 adjusted case filings resulted in judgeship numbers that closely approximated the judgeship needs of the majority of the courts of appeals, as the judges of each court perceived them. The current courts of appeals case-related workload measure principally reflects a policy decision using historical data on filings and terminations. It is not based on empirical data regarding the judge time that different types of cases may require. On the basis of the documentation we reviewed for our 2003 report, we determined that there is no empirical basis or assessing the potential accuracy of adjusted case filings as a measure of case-related judge workload.

Various Proposals Have Been Considered for Changing the Court of Appeals Workload Measure

In the past decade the Judicial Conference has considered a number of proposals for developing a revised case-related workload measure for the courts of appeals judges, but has been unable to reach a consensus on any approach. As part of its assistance to the Conference in this effort, the FJC in 2001 compiled a document that reviewed previous proposals to develop some type of case weighting measure for the courts of appeals. Table 1 outlines some of these proposals and their advantages and disadvantages, as identified by the FJC. Generally, methods that rely principally on empirical data on actual case characteristics and judge behavior (e.g., time spent on cases) are more appropriate than those that rely principally on qualitative data because statistical methods can be used to estimate the accuracy of the resulting workload measure.

We recognize that a methodology that provides greater empirical assurance of a workload measure’s accuracy will require judges to document how they spend their time on cases for at least a period of weeks. However, we believe that the importance and cost of creating new federal judgeships requires the best possible case-related workload data using sound research methods to support the assessment of the need for more judgeships.

Our 2003 Recommendations and the Judiciary’s Response

In our 2003 report we recommended that the Judicial Conference of the United States update the district court case weights using a methodology that supports an objective, statistically reliable means of calculating the accuracy of the resulting weights; and develop a methodology for measuring the case-related workload of courts of appeals judges that supports an objective, statistically reliable means of calculating the accuracy of the resulting workload measures and that addressed the special case characteristics of the Court of Appeals for the D.C. Circuit.

Neither of these recommendations has been implemented.

With regard to our 2003 recommendation for updating the district court case weights, the FJC agreed that the method used to develop the new case weights would not permit the calculation of standard errors, but that other methods could be used to assess the integrity of the resulting case weight system. In response, we noted that the Delphi technique to be used for developing out-of-court time estimates was most appropriate when more precise analytical techniques were not feasible and the issue could benefit from subjective judgments on a collective basis. More precise techniques were available for developing the new case weights and were to be used for developing new bankruptcy court case weights.

The methodology the Judicial Conference decided to begin in June 2002 for the revision of the bankruptcy case weights offered an approach that could be usefully adopted for the revision of the district court case weights. The bankruptcy court methodology used a two-phased approach. First, new case weights would be developed based on the time data recorded by bankruptcy judges for a period of weeks—a methodology very similar to that used to develop the bankruptcy case weights that existed in 2003 at the time of our report. The accuracy of the new case weights could be assessed using standard errors. The second part represents experimental research to determine if it is possible to make future revisions of the weights without conducting a time study. The data from the time study could be used to validate the feasibility of this approach. If the research determined that this were possible, the case weights could be updated more frequently with less cost than required by a time study. We believe this approach would provide (1) more accurate weighted case filings than the design developed and used for the development of the 2004 district court case weights, and (2) a sounder method of developing and testing the accuracy of case weights that were developed without a time study.

With regard to our recommendation improving the case-related workload measure for the courts of appeals, the Chair of the Committee on Judicial Resources commented that the workload of the courts of appeals entails important factors that have defied measurement, including significant differences in case processing techniques. We recognize that there are significant methodological challenges in developing a more precise workload measure for the courts of appeals. However, using the data available, neither we nor the Judicial Conference can assess the accuracy of adjusted case filings as a measure of the case-related workload of courts of appeals judges.

Contacts and Staff Acknowledgments

For further information about this statement, please contact William O. Jenkins Jr., Director, Homeland Security and Justice Issues, on (202) 512- 8777 or jenkinswo@gao.gov.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.