Federal Judgeships: The General Accuracy of District and Appellate Judgeship Case-Related Workload Measures
Released: 2013-09-10T00:00:00

Letter

Background

The demands on judges’ time are largely a function of both the number and complexity of the cases on their dockets. To measure the case- related workload of district court judges, the Judicial Conference has adopted weighted case filings. The purpose of the district court case weights was to create a measure of the average judge time that a specific number and mix of cases filed in a district court would require. Importantly, the weights were designed to be descriptive, not prescriptive—that is, the weights were designed to develop a measure of the national average amount of time that judges actually spent on specific cases, not to develop a measure of how much time judges should spend on various types of cases. Moreover, the weights were designed to measure only case-related workload. Judges have noncase-related duties and responsibilities, such as administrative tasks, that are not reflected in the case weights.

With few exceptions, such as cases that are remanded to a district court from the court of appeals, each civil or criminal case filed in a district court is assigned a case weight. For example, in the 2004 case weights—which are still in use—drug possession cases are weighted at 0.86, while civil copyright and trademark cases are weighted at 2.12. The total annual weighted filings for a district are determined by summing the case weight associated with all the cases filed in the district during the year. A weighted case filing per authorized judgeship is the total annual weighted filings divided by the total number of authorized judgeships. The Judicial Conference uses weighted filings of 430 or more per authorized judgeship as an indication that a district may need additional judgeships. Thus, for example, a district with 460 weighted filings per authorized judgeship, including newly requested judgeships, could be considered for an additional judgeship. However, the Judicial Conference does not consider a district for additional judgeships, regardless of its weighted case filings, if the district does not request any additional judgeships.

Accuracy of Judiciary’s Workload Measures Could Not Be Determined

1993 District Case Weights Reasonably Accurate, but Accuracy of 2004 Case Weights Could Not Be Statistically Determined

In our 2003 report, we found the district court case weights approved in 1993 to be a reasonably accurate measure of the average time demands a specific number and mix of cases filed in a district court could be expected to place on the district judges in that court. The methodology used to develop the weights used a valid sampling procedure, developed weights based on actual case-related time recorded by judges from case filings to disposition, and included a measure (standard errors) of the statistical confidence in the final weight for each weighted case type. Without such a measure, it is not possible to objectively assess the accuracy of the final case weights.

At the time of our 2003 report, the Subcommittee on Judicial Statistics of the Judicial Conference’s Judicial Resources Committee had approved the research design for revising the 1993 case weights, with a goal of having new weights submitted to the Judicial Resources Committee for review in the summer of 2004. The design for the new case weights relied on three sources of data for specific types of cases: (1) data from automated databases identifying the docketed events associated with the cases; (2) data from automated sources on the time associated with courtroom events for cases, such as trials or hearings; and (3) consensus of estimated time data from structured, guided discussion among experienced judges on the time associated with noncourtroom events for cases, such as reading briefs or writing opinions.

As we reported in 2009, according to FJC, the subcommittee wanted a study that could produce case weights in a relatively short period of time without imposing a substantial record-keeping burden on district judges. FJC staff provided the subcommittee with information about various approaches to case weighting, and the subcommittee chose an event- based method—that is, a method that used data on the number and types of events, such as trials and other evidentiary hearings, in a case. The design did not involve the type of time study that was used to develop the 1993 case weights. Although the proposed methodology appeared to offer the benefit of reduced judicial burden (no time study data collection), potential cost savings, and reduced calendar time to develop the new weights, we had two areas of concern—the challenge of obtaining reliable, comparable data from two different data systems for the analysis and the limited collection of actual data on the time judges spend on cases.

First, the design assumed that judicial time spent on a given case could be accurately estimated by viewing the case as a set of individual tasks or events in the case. Information about event frequencies and, where available, time spent on the events would be extracted from existing administrative databases and reports and used to develop estimates of the judge time spent on different types of cases. For event data, the research design proposed using data from two databases (one of which was new in 2003 and had not been implemented in all district courts) that would have to be integrated to obtain and analyze the event data. FJC proposed creating a technical advisory group to address this issue. In August 2013, FJC officials told us that the process of integrating the two data systems, though labor-intensive, was successful and resulted in accurate data. However, we have not reviewed the integration process for the two data systems, so we cannot determine the effectiveness of this process or whether accurate data resulted.

Second, we reported that the research design did not require judges to record time spent on individual cases, as was done for the 1993 case weights. Actual time data would be limited to that available from existing databases and reports on the time associated with certain courtroom events and proceedings for different types of cases. However, a majority of district judges’ time is spent on case-related work outside the courtroom. The time required for noncourtroom events—and some courtroom events that did not have actual time data available—would be derived from structured, guided discussion of groups of 8 to 13 experienced district court judges in each of the 12 regional circuits (about 100 judges in all). The judges would develop estimates of the time required for different events in different types of cases within each circuit using FJC-developed “default values” as the reference point for developing their estimates. These default values would be based in part on the existing case weights and, in part, on other types of analyses. Following the meetings of the judges in each circuit, a national group of 24 judges (2 from each circuit) would consider the data from the 12 circuit groups and develop the new weights.

The accuracy of judges’ time estimates is dependent upon the experience and knowledge of the participating judges and the accuracy and reliability of the judges’ recall about the average time required for different events in different types of cases—about 150 if all the case types in the 1993 case weights were used. In 2003, we found that these consensus data could not have been used to calculate statistical measures of the accuracy of the resulting case weights. Thus, we concluded that the planned methodology did not make it possible to objectively, statistically assess how accurate the new case weights are—weights whose accuracy the Judicial Conference relies upon in assessing judgeship needs.

In August 2013, AOUSC officials stated that, since 2005, for purposes of determining the need for an additional authorized judgeship, a district’s weighted case filings per authorized judgeship is calculated by including the potential additional judgeship. For example, if a district had total weighted filings of 4,600 and 9 authorized judgeships, and it planned to request 1 additional judgeship, its weighted filings per authorized judgeship, for purposes of the judgeship request process, would be 460. Without including the potential additional judgeship in the calculation, the weighted case filings would be about 511. AOUSC officials stated in August 2013 that the judiciary adopted the proposed methodology in 2004 and does not have plans to update the 2004 district court case weights.

Accuracy of Courts of Appeals Case-Related Workload Measure Could Not Be Assessed

In 2003, we found that the principal quantitative measure the Judicial Conference used to assess the need for additional courts of appeals judgeships was adjusted case filings. The measure is based on data available from standard statistical reports for the courts of appeals. The adjusted filings workload measure is not based on any empirical data regarding the time that different types of cases required of appellate judges.

The Judicial Conference’s policy is that courts of appeals with adjusted case filings of 500 or more per 3-judge panel may be considered for 1 or more additional judgeships. Courts of appeals generally decide cases using constantly rotating 3-judge panels. Thus, if a court had 12 authorized judgeships, those judges could be assigned to four panels of 3 judges each. In assessing judgeship needs for the courts of appeals, the conference may also consider factors other than adjusted filings, such as the geography of the circuit or the median time from case filings to disposition.

Essentially, the adjusted case filings workload measure counts all case filings equally, with two exceptions. First, cases refiled and approved for reinstatement are excluded from total case filings. Second, pro se cases are weighted at 0.33, or one-third as much as other cases, which are weighted at 1.0. For example, a court with 600 total pro se case filings in a year would be credited with 198 adjusted pro se case filings (600 x 0.33). Thus, a court of appeals with 1,600 filings (excluding reinstatements)—600 pro se cases and 1,000 non-pro se cases—would be credited with 1,198 adjusted case filings (198 discounted pro se cases plus 1,000 non-pro se cases). If this court had 6 judges (allowing two panels of 3 judges each), it would have 599 adjusted case filings per 3- judge panel, and, thus, under Judicial Conference policy, could be considered for an additional judgeship.

The current court of appeals workload measure, which, AOUSC officials stated, was adopted in 1996, represents an effort to improve the previous measure. In our 1993 report on judgeship needs assessment, we found that the restraint of individual courts of appeals, not the workload standards, seemed to have determined the actual number of appellate judgeships the Judicial Conference requested. At the time the current measure was developed and approved, using the new benchmark of 500 adjusted case filings resulted in judgeship numbers that closely approximated the judgeship needs of the majority of the courts of appeals, as the judges of each court perceived them. The current courts of appeals case-related workload measure principally reflects a policy decision using historical data on filings and terminations. It is not based on empirical data regarding the judge time that different types of cases may require. On the basis of the documentation we reviewed for our 2003 report, we determined that there was no empirical basis for assessing the potential accuracy of adjusted case filings as a measure of case-related judge workload.

Judiciary Has Taken No Action to Address Our 2003 Recommendations

In our 2003 report, we recommended that the Judicial Conference of the United States update the district court case weights using a methodology that supports an objective, statistically reliable means of calculating the accuracy of the resulting weights, and develop a methodology for measuring the case-related workload of courts of appeals judges that supports an objective, statistically reliable means of calculating the accuracy of the resulting workload measures and that addresses the special case characteristics of the Court of Appeals for the D.C. Circuit.

Neither of these recommendations has been implemented, and in August 2013, AOUSC officials stated that the judiciary does not have plans to update the 2004 district court case weights or the 1996 court of appeals adjusted filings weights. With regard to our 2003 recommendation for updating the district court case weights, we reported that FJC agreed that the method used to develop the new case weights would not permit the calculation of standard errors, but that other methods could be used to assess the integrity of the resulting case weight system. In response, we noted that the Delphi technique to be used for developing out-of-court time estimates was most appropriate when more precise analytical techniques were not feasible and the issue could benefit from subjective judgments on a collective basis. More precise techniques were available for developing the new case weights and were to be used for developing new bankruptcy court case weights.

In our 2003 report, we also concluded that the methodology the Judicial Conference decided to begin in June 2002 for the revision of the bankruptcy case weights offered an approach that could be usefully adopted for the revision of the district court case weights. The bankruptcy court methodology used a two-phased approach. First, new case weights were to be developed based on time data recorded by bankruptcy judges for a period of weeks—a methodology very similar to that used to develop the bankruptcy case weights that existed in 2003 at the time of our report. The accuracy of the new case weights could be assessed using standard errors. The second part represents experimental research to determine if it is possible to make future revisions of the weights without conducting a time study. The data from the time study could be used to validate the feasibility of this approach. If the research determined that this was possible, the case weights could be updated more frequently with less cost than required by a time study. We concluded in 2003 that that approach could provide (1) more accurate weighted case filings than the design developed and used for the development of the 2004 district court case weights, and (2) a sounder method of developing and testing the accuracy of case weights that were developed without a time study. However, we have not reviewed the effectiveness of this methodology or confirmed whether the judiciary implemented this approach.

With regard to our recommendation on improving the case-related workload measure for the courts of appeals, the Chair of the Committee on Judicial Resources commented that the workload of the courts of appeals entails important factors that have defied measurement, including significant differences in case-processing techniques. We recognized that there were significant methodological challenges in developing a more precise workload measure for the courts of appeals. However, we stated that using the data available, neither we nor the Judicial Conference could have assessed the accuracy of adjusted case filings as a measure of the case-related workload of courts of appeals judges.

The Ranking Member of the Subcommittee on Bankruptcy and the Courts has requested that we conduct a full review of the case-related workload measures for district court and courts of appeals judges, including a follow-up on our 2003 recommendations. Such a review will allow us to evaluate the judiciary’s methodology and efforts over the last 10 years.

Mr. Chairman, this concludes my statement for the record.

Contact and Staff Acknowledgments

For further information about this statement, please contact David C. Maurer, Director, Homeland Security and Justice Issues, on (202) 512- 9627 or maurerd@gao.gov. In addition to the contact named above, the following individuals also made major contributions to this testimony: Chris Currie, Acting Director; David P. Alexander, Assistant Director; Brendan Kretzschmar; Jean M. Orland; Rebecca Kuhlmann Taylor; and Janet G. Temko.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.