Legal Services Corporation: More Needs to Be Done to Correct Case Service Reporting Problems
Released: 1999-09-27T00:00:00

Letter

Background

LSC was established in 1974 as a private, nonprofit, federally funded corporation to provide legal assistance to low-income people in civil matters. LSC provides the assistance indirectly through grants to competitively selected local programs. LSC distributes funds to the grantees on the basis of the number of low-income persons living within a service area. Grantees may receive additional funding from non-LSC sources.

During 1998, LSC funded 262 local grantees that operated through approximately 900 neighborhood law offices employing about 3,600 attorneys and 1,400 paralegals. Each local program is governed by its own board of directors and is required to spend at least 12.5 percent of its LSC grant to encourage private attorney involvement (PAI) in delivering legal services to low-income clients. In fiscal years 1998 and 1999, LSC received appropriations of $283 million and $300 million, respectively.

LSC’s authorizing legislation restricts it from engaging in lobbying; political activities; class actions except under certain conditions; and cases involving abortion, school desegregation, and draft registration or desertion from the military. Annual appropriations laws have placed additional restrictions on the activities in which LSC grantees can engage, even with non-LSC funds. In 1996, for example, grantees were prohibited from engaging in challenges to welfare reform, litigation on behalf of prisoners, representation in drug-related public housing evictions, and representation of certain categories of aliens.

Grantees must serve only those clients who meet financial and citizenship/alien eligibility requirements. With respect to client financial eligibility, local programs are to establish their own criteria, which, in general, require that clients’ income not exceed 125 percent of the federal poverty guidelines. With appropriate documentation of the grantee’s decision, clients who are between 125 percent and 187.5 percent of the federal poverty level may be found eligible. LSC regulations require that grantees (1) adopt a form and procedure to obtain eligibility information and (2) preserve that information for audit by LSC. With regard to citizenship/alien eligibility, only citizens and certain categories of aliens are eligible for services. For clients who are provided services in person, a citizen attestation form or documentation of eligible alien status is required. For clients who are provided services on the telephone, grantees must document that they made inquiries regarding the individuals’ citizenship/alien eligibility.

LSC uses a Case Service Reporting (CSR) system to gather quantifiable information from grantees on the services they provide that meet LSC’s definition of a case. The CSR Handbook is LSC’s primary official guidance to grantees on how to record and report cases. According to the 1999 CSR Handbook, which revised and expanded the guidance in LSC’s 1993 Handbook, information about cases is an important indicator of the number of legal problems that programs address each year, and LSC relies on such case information in its annual request for federal funding for legal services.

Audit reports on five grantees issued by LSC’s OIG between October 1998 and July 1999 reported that all five grantees misreported the number of cases they had closed during calendar year 1997 and the number of cases that remained open at the end of that year. The OIG found that all five grantees overstated the number of closed cases, while four overstated and one understated open cases. The OIG attributed the overreporting to such factors as (1) counting as cases telephone calls in which individuals were not provided any legal assistance and only partial eligibility determinations were made; (2) counting wholly non-LSC funded cases as LSC cases; (3) double counting of the same cases; (4) reporting cases as closed during 1997, or as still open at the end of 1997, when service ceased in prior years; and (5) counting as cases the provision of services to over-income clients. In June 1999, in response to Congress’ request for information on whether the 1997 case data of other LSC programs had problems similar to those reported by LSC’s OIG, we issued a report on our audit of five of LSC’s largest grantees: Baltimore, Chicago, Los Angeles, New York City, and Puerto Rico. We found similar types of reporting errors at the five grantees and estimated that, overall, 75,000 of the 221,000 open and closed cases that the five grantees reported to LSC were questionable. Interviews that we conducted with LSC officials and executive directors of the 5 audited grantees indicated that they had taken or were planning to take steps to correct the causes of these case-reporting problems.

Objectives, Scope, and Methodology

Our objectives were to determine (1) what efforts LSC and its grantees have made to correct problems with case service reporting, and (2) whether these efforts are likely to resolve the case reporting problems that occurred in 1997.

To address the objectives, we reviewed documents that contained LSC case reporting guidance, interviewed cognizant officials at LSC headquarters, and conducted structured telephone interviews with a random sample of grantee executive directors. Specifically, we reviewed LSC regulations and LSC’s 1993 and 1999 CSR Handbook, as well as supplemental guidance that LSC distributed to grantees in the form of program letters and frequently asked questions and answers about case reporting. We also collected documentation and interviewed LSC officials and grantee executive directors to gather information about a self- inspection process that LSC required all grantees to undertake in order to determine the accuracy of their 1998 case data. In our telephone interviews with executive directors, we asked if the directors viewed LSC’s case reporting requirements as being clear, what changes they have made or planned to make as a result of the requirements, and the results of their self-inspection of 1998 case data. We also interviewed LSC officials and grantee executive directors about areas of case reporting that they felt needed further clarification.

To identify the universe of LSC grantees, an LSC official referred us to the list of LSC programs on the corporation’s Internet Web site. As of July 15, 1999, the Internet site listed 256 programs. From this list, we randomly selected 80 programs for our sample. We developed the interview instrument, pretested it with executive directors from two grantees, revised the instrument based on the pretest results, and completed approximately 1-hour-long structured interviews by telephone with the executive directors of 79 of the 80 programs, for a 99 percent response rate. (After numerous attempts over several days, we were unable to contact the executive director of one LSC grantee.) Our sample was designed so that we could generalize our findings with 95-percent confidence and a maximum 10-percent margin of error to the universe of LSC grantees.

We performed our work from July through September 1999 in accordance with generally accepted government auditing standards. We requested comments on a draft of this report from the President of LSC. LSC’s comments are discussed at the end of this letter and included as appendix I.

LSC Clarified CSR Requirements and Grantees Have Been Making Program Changes to Comply With the Requirements, but Some Grantees Remain Unclear About Certain Requirements

LSC issued a new CSR Handbook and distributed other written communications intended to clarify reporting requirements to its grantees. Most grantees indicated that the new guidance helped clarify LSC’s reporting requirements, and virtually all of them indicated that they had or planned to make program changes as a result of the requirements. Many grantees, however, identified areas of case reporting that remained unclear to them.

LSC Issued New CSR Guidance

To address specific problems identified by the OIG and LSC’s own internal program reviews, LSC has issued revised reporting guidance. In November 1998, LSC issued the 1999 CSR Handbook, which replaced the 1993 CSR Handbook. The 1999 handbook instituted changes to some of LSC’s reporting requirements and provided more detailed information on other requirements than previously existed. LSC also distributed program letters and a list of frequently asked CSR questions and answers that further elaborated on points made in the 1999 handbook.

The 1999 CSR Handbook instituted several notable changes to case reporting requirements. These included (1) procedures for timely closing of cases; (2) procedures for management review of case service reports; (3) procedures for ensuring single recording of cases; (4) requirements to report LSC-eligible cases, regardless of funding source; and (5) requirements for reporting PAI cases separately. In addition, grantees were required to use automated case management systems and procedures that would ensure that program managers had timely access to accurate information on cases and the capacity to meet their reporting requirements.

On November 24, 1998, LSC informed its grantees that two of the changes in the 1999 CSR Handbook were to be applied to the 1998 case data. The two changes pertained to timely closing of cases and management review of case service reports. The timely closing provision required grantees to ensure that cases in which legal assistance had ceased in 1998, and was not likely to resume, would be closed prior to grantees’ submission of case service reports to LSC in March 1999. To the extent practicable, cases in which the only assistance provided to the client was counsel and advice, brief service, or referral after legal assessment were to be closed in the year in which these types of service were provided. Cases involving other types of service were to be closed in the year in which program staff determined that further legal assistance was unnecessary, not possible, or inadvisable and a closing memorandum or other case-closing notation was prepared. The management review provision required the executive director, or a designee, to review the program’s case service reports prior to their submission to LSC in order to ensure their accuracy and completeness.

The remaining new provisions of the 1999 CSR Handbook were not applicable to 1998 cases. For example, for 1998, there was no requirement for grantees to ensure that cases were not double counted. For 1999, LSC is requiring the use of automated case management systems and procedures to ensure that cases involving the same client and specific legal problem are not reported to LSC more than once. For 1998, grantees could report only those cases that were at least partially supported by LSC funds. For 1999, LSC is requiring grantees to report all LSC-eligible cases, regardless of funding source. LSC intends to estimate the percentage of activity spent on LSC service by applying a formula that incorporates the amount of funds grantees receive from other funding sources compared with the amount they receive from LSC. For 1998, grantees were required to report their LSC-funded PAI cases together with non-PAI cases. For 1999, PAI cases are to be reported separately.

In addition to changing certain reporting requirements, the 1999 handbook also provides more detailed guidance to grantees than the 1993 handbook. For example, the 1999 handbook provides more specific definitions of what constitutes a “case” and a “client” for CSR purposes. The 1999 handbook also addresses documentation requirements that were not discussed in the 1993 handbook. For example, the 1999 handbook indicates that, except for telephone service cases, the client’s file must contain an attestation of citizenship or documentation of alien eligibility in order for the case to be reported to LSC. The 1999 handbook also imposes requirements for documentation of information on client income and assets. The handbook states that, for all cases reported to LSC, the eligibility documentation must include specific information about income and assets. The 1999 handbook also contains a requirement that legal assistance, to be counted as a case, must be provided by an attorney or paralegal.

Grantee Directors Reported That They Are Implementing Changes to Comply With Reporting Requirements

On the basis of our survey, we estimate that over 90 percent of grantee executive directors viewed the changes in the 1999 CSR Handbook as being clear overall, and virtually all of them planned to or had made at least one change to their program operations as a result of the revised case reporting requirements. Program changes that were cited included revising policies and procedures, providing staff training, modifying forms and/or procedures used during client intake, implementing computer hardware and software changes, and increasing review of cases.

Nearly half of the grantees indicated that they planned to or had revised their policies and/or procedures to comply with the 1999 handbook requirements, and slightly less than a third planned to or had conducted staff training on the requirements. Respondents told us that the focus of their training was on such issues as the current definition of a case, how to determine and document clients’ financial eligibility, how to determine and document citizenship/alien eligibility, timely closing of cases, and prevention of duplicate case reporting.

More than half of the grantees indicated that they planned to or had changed their intake forms and/or procedures. Of the 41 respondents who made this comment, 26 said they were making these changes in order to document client income and assets.

Slightly over 40 percent of the grantees reported that they planned to or had made computer changes. The computer changes included such actions as adding new fields to their automated case management (e.g., to ensure that client eligibility and acceptance information is recorded), making programming changes (e.g., to identify duplicate cases, ensure that cases are not assigned separate numbers for the same client with the same legal problem, and better document client assets), and installing new software (e.g., to generate reports so that they can track how long cases are open).

About 10 percent of grantee executive directors indicated that they could better comply with CSR requirements if they had uniform case management software. LSC, too, believes that grantees’ case management systems should provide for more consistency in the collection and processing of CSR information. According to an LSC official, LSC is developing a strategy for modifying grantees’ case management systems so that data errors can be detected and prevented. As part of the strategy, LSC has hired an expert in case management systems and is working to develop standard input requirements for these systems. LSC intends to work with case management system vendors to modify the systems so that they prevent cases from being accepted if grantees do not record the required compliance information. LSC expects modifications to the case management systems to be implemented in calendar year 2000 and available for application to grantees’ 2000 CSR data. In the nearer term, LSC plans to work with a contractor to develop customized case management queries that would enable grantees to detect and remedy errors in their 1999 case data. In 1999, LSC plans to pilot test the computer queries at five programs that have the most commonly used case management systems.

Nearly three-fourths of the grantees indicated that they planned to or had increased their review of cases to comply with LSC requirements. Respondents cited such review activities as more intense monitoring of open cases, to ensure that they are closed in a timely manner, and more thorough monitoring of PAI cases. Some grantees said they have directed increased attention to reporting requirements during routine reviews of cases, while others said they instituted more frequent reviews of cases. One executive director reported to LSC that managing attorneys would meet with all case-handling staff and reemphasize the importance of keeping accurate activity records. These managing attorneys are to monitor compliance with the requirement initially on a weekly basis and then on a random basis. If deficiencies are discovered, the managing attorney is to direct the case handler to correct them immediately, and the managing attorney is to recheck the file within 48 hours to ensure compliance with the directive. Another executive director told us that his program had not previously reviewed CSR data. He said that they now are reviewing reports, along with listings of open and closed cases, to ensure their compliance with LSC guidelines. Some executive directors said that they planned to become more involved in reviewing case files and case management reports.

Many grantees reported making other efforts to comply with reporting requirements, such as disseminating the new handbook to case handlers, developing compliance checklists, emphasizing the importance of compliance to staff, sending written instructions or memos to staff, holding meetings and discussions, providing more feedback to case handlers, and more stringently enforcing requirements.

Many Grantees Remain Unclear About Certain Reporting Requirements

Although most of the grantee executive directors reported that the new LSC guidance helped clarify requirements, many of them also indicated that they were still unclear about certain requirements and that additional clarification was needed. Among the areas of confusion or uncertainty that executive directors identified were requirements pertaining to asset and citizenship/alien eligibility documentation, single recording of cases, and who can provide legal services.

Asset documentation: The 1999 CSR Handbook states that, for all cases to be reported to LSC, the eligibility documentation should include specific information about income and assets. About 30 percent of the executive directors indicated that LSC’s requirements for documenting client assets were clear only to some, to little, or to no extent. Of the 24 respondents who gave this response, 23 made comments to the effect that LSC should clarify what it means by assets and asset limits and/or clarify its documentation requirements for client assets. In a July 14, 1999, program letter to grantees, LSC noted that although many grantees inquire about applicants’ assets, they do not consistently document either the inquiries or the applicants’ responses. In its program letter, LSC sought to clarify its requirements for documenting and preserving the asset information obtained from each applicant. Because we conducted our telephone survey in late July and early August, we do not know how many of the executive directors with whom we spoke had reviewed LSC’s July 14 guidance and felt that it sufficiently clarified their questions concerning asset documentation. However, two respondents told us that LSC’s July 14 guidance was still unclear. One respondent said, for example, that the program letter left unclear the meaning of the term “household goods,” whether exempted assets had to be documented, who was to value the goods, and who was to determine what to count and what not to count as part of assets. Our own analysis of LSC guidance on the asset issue indicated that LSC has not been consistent in its directives to grantees about the specificity of asset information they must have in order to comply with CSR reporting requirements. For example, in its March 24, 1999, communication on frequently asked CSR questions and answers, LSC instructed grantees to document, at minimum, the total amount of household assets accessible to the client. In its July 14, 1999, program letter, LSC instructed grantees to identify and document all of the liquid and nonliquid assets of all persons who are resident members of a family unit.

Citizenship/alien eligibility documentation requirements for telephone cases: The 1999 CSR Handbook does not explicitly address the citizenship or eligible alien status documentation requirements for situations where assistance is provided only over the telephone. Nearly one-fourth of the executive directors indicated that LSC’s documentation requirements in this area were clear only to some, to little, or to no extent. Of 19 respondents who gave this response in our survey, 15 said that more clarification was needed on the grantees’ documentation responsibilities. Respondents said, for example, that they were confused about exactly when they needed to obtain a written attestation, whether it was sufficient simply to record that questions about citizenship/alien eligibility had been asked, whether certain types of service required documentation while others did not, and whether the requirement changed if an individual receiving assistance over the telephone came into the office to drop off documents. In its July 14, 1999, program letter to grantees, LSC sought to clarify its requirements for documenting citizenship/alien eligibility information in telephone cases. LSC stated that it requires recipients to make appropriate inquiry of every telephone applicant and record the inquiry and response. All such documentation is to be maintained in the client file. We do not know how many of the respondents to our survey were familiar with LSC’s July 14 guidance when we interviewed them in late July and early August.

Single recording of cases: The 1999 CSR Handbook requires that programs ensure that cases involving the same client and specific legal problem are not recorded and reported to LSC more than once. Over one-fourth of the executive directors indicated that the requirement for preventing duplicate case reporting was clear only to some, to little, or to no extent. Of the 22 respondents who gave this response in our survey, 17 said that the distinction between specific and related legal problems is difficult to determine. Several respondents cited examples in family law cases where more than one problem can arise within one family case. Depending on the situation, it was not clear to them at what point legally related issues become separate enough to count as separate cases. One respondent indicated that attorneys in his program have a hard time interpreting this requirement, and that he got calls every week about this issue.

Provider of legal assistance: The CSR handbook requires that legal assistance be provided by an attorney or paralegal in order for a service to be called a case. Slightly over 40 percent of executive directors indicated that LSC was clear only to some, to little, or to no extent about who can provide such legal assistance. Respondents noted in our telephone survey, for example, that the terms “case handler,” and “paralegal” are not clearly defined, and that they did not know whether nonlawyers (e.g., intake workers) who are supervised by lawyers can provide legal assistance. Comments made to us by the executive directors revealed that grantees held varying views about who can provide legal assistance to clients. Several respondents specifically stated that this is an important issue that LSC should address in its handbook. Our own analysis of LSC guidance on this issue indicates that LSC has not been consistent in its advice to grantees. For example, in its communication with grantees on frequently asked CSR questions and answers, LSC stated that a telephone conversation between an intake specialist and a caller who was accepted for service can be counted as a case if the caller received some advice that addressed a specific legal problem. LSC officials told us that LSC’s current position on this issue is that the person giving legal advice must be someone who is authorized to practice law or is under the supervision of an attorney in accordance with local rules of practice. Therefore, legal advice can be given by a (1) lawyer, (2) paralegal, or (3) intake specialist or law student under the supervision of a lawyer, as long as the assistance does not violate local rules of practice.

Most Grantees Certified to the Accuracy of Their 1998 CSR Data, but Questions About Data Accuracy and Interpretation Remain

LSC sought to determine the accuracy of grantees’ case data by requiring that grantees do self-inspections of their open and closed caseload data for 1998. Grantees were required to determine whether the error rate in their data exceeded 5 percent. If the error rate was 5 percent or less, they could certify that their data were substantially correct. If the error rate was higher than 5 percent, they were to determine how the problems identified could be addressed. LSC found that about three-fourths of the grantees were able to certify to the substantial accuracy of their data. LSC used the results of the self-inspections to estimate the total number of case closings in 1998. Our review of LSC’s self-inspection process raised concerns about the accuracy and interpretation of the results, and what the correct number of certifying programs should be.

LSC Grantees Conducted Self-Inspections of 1998 CSR Data

On May 14, 1999, LSC issued a memo to all grantees instructing them to complete a self-inspection procedure by July 1, 1999. The purpose of the self-inspection was to ensure that (1) grantees were properly applying instructions in the 1999 edition of the CSR Handbook that were applicable to the 1998 data, and (2) LSC had accurate case statistical information to report to Congress for calendar year 1998.

LSC provided detailed guidance to grantees on the procedures for the self- inspection. Each grantee was to select and separately test random samples of open and closed cases to determine whether the number of cases it reported to LSC earlier in the year was correct. Grantees were to verify that the case file contained a notation of the type of assistance provided, the date on which the assistance was provided, and the name of the case handler providing the assistance. Grantees were also to determine whether assistance had ceased prior to January 1, 1998; was within certain service categories as defined by the 1999 handbook; was provided by an attorney or paralegal; and was not prohibited or restricted. Finally, grantees were to verify that each case had eligibility information on household income, household size, household assets, citizenship attestation for in-person cases, and indication of citizenship/alien status for telephone-only cases. According to LSC officials, the self-inspection was a single procedure that was undertaken within a limited time period, and LSC did not expect the self-inspection to resolve all case-reporting problems.

In requiring grantees to verify that their 1998 case files contained information on client assets and an indication of citizenship/alien status for telephone-only cases, LSC imposed stringent criteria on the self- inspections. The criteria were stringent in that LSC had not promulgated explicit documentation requirements related to these issues until it released the 1999 CSR Handbook and July 14 program letter. Grantees were not required to apply these new documentation requirements to their 1998 case data. Recognizing that in 1998 many grantees did not keep sufficient documentation on assets or citizenship/alien eligibility to comply with the stringent self-inspection requirements, LSC allowed grantees some latitude in determining whether they were in fundamental compliance with the requirement that legal assistance only be provided to eligible individuals. In conversations with grantee executive directors during the self-inspection, LSC officials advised them that if they had a level of certainty that their program staff had asked questions about assets, and if their certainty was sufficient for them to sign a form attesting to this, that was an acceptable basis for asserting compliance with the reporting requirement for assets. Similarly, if grantees were sufficiently certain that, for telephone cases, their program staff had asked questions about citizenship/alien status but had not documented the inquiry, that too was acceptable. Finally, LSC allowed grantees some latitude with respect to the requirement that an attorney or paralegal must be the provider of legal assistance. LSC advised grantees during the self-inspection that it was acceptable for them to count as valid cases instances in which legal advice was given by a (1) lawyer, (2) paralegal, or (3) intake specialist or law student under the supervision of a lawyer, as long as the assistance did not violate local rules of practice.

If any single aspect of a case failed to meet LSC’s requirements, the case was to be classified as an error for reporting purposes. If the grantees found that their CSR case sampling had an error rate of 5 percent or less, the program directors and policy board chairs were to sign a certification form and return it to LSC. Grantees who could not certify to the correctness of their 1998 CSR data were to submit a letter to LSC describing (1) the problems they had identified during the self-inspection process and (2) the corrective actions they had instituted to address the problems. Grantees could resubmit their 1998 CSR data to LSC if they identified one or more problems in the random sample and corrected their entire 1998 database so that the problems no longer appeared. If, by correcting the problems, the error rate in the data was reduced to 5 percent or less, the grantees could resubmit their 1998 data along with a signed certification attesting to the substantial accuracy of the resubmitted data. In this way, grantees who were unable to certify at one point in time could certify at a later point in time. As of July 29, 1999, 26 grantees had resubmitted their 1998 CSR data after having made corrections to the data, and 20 of them had certified their data.

Most Grantees Certified Their 1998 CSR Data

According to LSC officials, about three-fourths of the grantees certified the accuracy of their 1998 case data. As of August 26, 1999, LSC documents indicated that 199 of 261 grantees (76 percent) reported substantially correct CSR data to LSC. The remaining 62 grantees (24 percent) did not certify to LSC that their CSR data were substantially correct. On the basis of the self-inspection results, LSC estimated that grantees closed 1.1 million cases in 1998.

LSC officials told us that they were surprised that such a large number of grantees were able to certify their 1998 CSR data. They attributed the lower-than-expected error rates to the following factors:

The self-inspection did not attempt to identify duplicate cases. LSC officials explained that, during most of 1998, LSC did not have a standard regarding duplicate cases, and that they believed it would have been too burdensome on grantees for LSC to require them to apply the new standard retroactively. According to an LSC official, duplicate cases are best caught at the time of intake. The official also noted that duplicate cases had not in the past been found to be a major problem in comparison with other identified problems.

Grantees received the new 1999 CSR Handbook in November1998, and a number of them had implemented the requirements that applied to their 1998 data by the time they submitted their 1998 case statistics.

Grantees were aware of the problem that the OIG had identified with cases coded as referral after legal assessment. That is, some programs had inappropriately reported as CSR cases numerous instances of making a telephone referral without providing legal advice and/or without documenting an individual’s eligibility. Because grantees had been sensitized to this issue, LSC officials believed that they were less likely to count these referrals as cases in 1998.

According to LSC officials, CSR problems were more common at larger grantees that had heavier case loads and multiple branches. They noted that approximately 30 of the 50 grantees with the largest caseloads were unable to certify their 1998 data. They hypothesized that larger programs may have difficulty certifying if such programs have (1) numerous branch offices, one or more of which are out of compliance with regulations and causing reporting problems for the entire program, and/or (2) numerous sources of funding and more compliance requirements, which could add to the complexity of their reporting process. The officials acknowledged that these hypotheses needed further exploration.

LSC officials also reported that noncertifying grantees identified two principal problems. One pertained to the lack of citizenship attestations in case files. The second pertained to the lack of information on client assets. LSC officials also noted that some grantees had reported matters, such as referrals, as cases, and some had reported cases as open when they did not meet the timely closing requirement. In our telephone interviews with 79 executive directors, 24 reported that they did not certify their 1998 data to LSC. Factors that the executive directors of noncertifying programs believed greatly affected the errors in their 1998 data included lack of clear guidance from LSC (cited by 11 respondents), computer problems (cited by 7 respondents), and insufficient attention by their programs to administrative matters (cited by 5 respondents).

Computer problems that were noted included difficulties merging databases and using several different software packages, and problems with upgrading the computer system.

LSC Used Self-Inspection Results to Estimate Number of Cases Closed

Based on the self-inspection results, LSC estimated that its grantees closed about 1.1 million cases in 1998. LSC arrived at this figure by subtracting the total number of closed cases estimated to be in error (135,498) from the total number of cases that grantees reported to LSC (1,260,351). As of July 29, 1999, LSC estimated that its grantees’ total closed caseload for 1998 was 1,124,853.

LSC intends to report only the estimated number of cases closed in 1998, even though the self-inspections were to encompass both open and closed cases. According to LSC officials, LSC has less confidence in the open-case numbers than the closed-case numbers. One reason for this is that they believe open cases are more likely than closed cases to have timely closing problems. A second reason is that some grantees experienced problems when converting to a new computer system. System conversions sometimes caused dates to be lost or cases to be mistakenly coded as being open.

Self-Inspection Results Raised Concerns

Our review of LSC’s self-inspection results raised some concerns about LSC’s interpretation of the results and about the accuracy of the data provided to LSC by grantees. As a result, we could not assess whether the number of certified programs and case closures that LSC estimated for 1998 is correct, lower, or higher than it should be.

Although LSC provided instructions to grantees on how they should select test samples and what case information they should verify, LSC did not issue standardized procedures for grantees to use in reporting the results of their self-inspections. Grantees that could not certify their data wrote letters to LSC in which they described the errors they uncovered. The letters contained varying degrees of detail about the errors. Some programs provided an overall error rate and did not separately report how much error they found in open and closed cases. Others provided detailed information on both the number of errors found in open and closed cases, respectively, as well as the number of errors broken out by type. Since LSC did not have a standard protocol for collecting the results of the self- inspections, in some cases LSC officials had to rely on interpreting grantee letters that described the problems that were discovered. LSC officials believe that their numerous contacts with grantees who called them to ask questions about the self-inspections, combined with their analysis of the grantee letters, enabled them to correctly determine the number of certifying programs and estimate the number of closed cases.

We are uncertain how many programs should have been counted as certified because we are uncertain if LSC applied a consistent definition of “certification.” Most programs that were on LSC’s certification list determined that they had error rates of 5 percent or less for both open and closed cases. However, LSC placed some programs on the certified list if the program’s overall error rate for closed cases was 5 percent or less, even if the overall error rate actually was higher than 5 percent. We encountered this situation in two instances in which executive directors told us in telephone interviews that they did not certify their CSR data because their overall error rate exceeded 5 percent. However, these programs appeared on LSC’s list of certified programs. When we asked an LSC official about this, he told us that they advised grantees that if their closed case error rate did not exceed 5 percent, they should “partially certify” their data. In response to our inquiry, the official reviewed the certification letters submitted by nearly 200 grantees, and he identified 5 certified programs whose error rates for open cases exceeded 5 percent. Given that some grantees submitted only an overall estimate of data error, we do not know how many programs qualified to be certified overall, just for closed cases, or just for open cases.

In another instance, an executive director told us that she did not certify her program because 8 of the 14 case-handling offices had error rates exceeding 5 percent. Nonetheless, this program appeared on LSC’s certified list. An LSC official explained that, after reviewing the information provided by this program, the official agreed that the program should not have been classified as certified. In a fourth instance, in which an executive director reported to us that he did not certify his program, an LSC official said that, although they thought the grantee’s data had been corrected, the program had not yet certified its data and might need to do additional sampling.

We are also concerned that LSC’s instructions to grantees on how to conduct the self-inspections may have led some of the smaller grantees to select too few test cases to make a reliable assessment of the proportion of error in their case data. For example, LSC instructed grantees to select every tenth case for review if the program handled less than 1,000 cases in 1998. This was to be done separately for open and closed cases. Based on 1998 case statistics that grantees submitted to LSC between January and March 1999, several programs would have based their certification determinations on reviews of a relatively small number of cases. Seven grantees had fewer than 300 closed cases, and 43 grantees had fewer than 300 open cases. Three programs had fewer than 300 cases combined. The smallest program would have based its self-inspection on 3 closed cases and 1 open case. We believe that, in general, samples of 30 or fewer cases are too small to provide reliable estimates of the total number of case data errors. Because these were smaller grantees, this limitation would have had little effect on LSC’s estimate of total closed caseload. However, it could have affected—by either overstating or understating—LSC’s count of the number of certified programs.

LSC does not know how well grantees conducted the self-inspection process, nor how accurate the results are. We spoke with several executive directors who did not correctly follow LSC’s reporting requirements. In one case, the executive director sought clarification from LSC headquarters about the use of a CSR closure code and was given incorrect verbal guidance. The situation concerned whether applicant files could be closed as “client withdrew” cases when individuals who were accepted for service and assigned to staff attorneys did not subsequently appear for a meeting with the attorney. According to both the executive director and an LSC official, the executive director was told that if the program attempted to contact these individuals, by telephone or letter, to determine whether they were still interested in obtaining legal assistance, they could be counted as cases. The executive director told us that she would review her entire database for this type of error and make corrections. This program was included on LSC’s certified list, but we do not know whether the program would stay on that list if all the errors were identified.

Another executive director told us that he was concerned that LSC did not want grantees to count assistance over the telephone as a case. This is not an entirely correct interpretation of LSC guidance since, under certain conditions, LSC permits legal assistance over the telephone to be counted as a case. Although this program was included on LSC’s certified list, any valid telephone cases that were not counted would have erroneously increased its error rate.

A third executive director told us that the timely closing rule required him to close new cases in December. If the person sought assistance with a similar problem in January, it would be treated as a new case because a new reporting year had begun. In a March 1999 written communication intended to supplement the 1999 handbook, LSC advised grantees to exercise discretion about when to close cases that were opened near the end of the year. LSC did not require grantees to close all cases at year-end. These examples illustrate the possibility that incorrect interpretations of LSC guidance may have resulted in some programs certifying their 1998 data when they shouldn’t have, and other programs not certifying their 1998 data when they should have. An LSC official told us that, although they have conducted CSR training sessions for grantee executive directors, there are thousands of case handlers in grantee offices who have not received similar training. The official acknowledged that written guidance and telephone contacts with grantees may not be sufficient to ensure correct and consistent understanding of reporting requirements, and that LSC may consider alternative ways of providing training to staff, such as through videos.

Incorrect interpretations of LSC guidance could also have affected the accuracy of LSC’s estimate of closed cases. The Inspector General told us that his office has completed audits of six grantees’ 1998 case data. One additional audit of a grantee’s CSR data had not yet been completed. The six completed audits included both certifying and noncertifying grantees. According to the Inspector General, the results of the six completed audits would be provided to Congress by September 29, 1999.

LSC officials told us that the self-inspection was valuable and that LSC plans to have grantees complete self-inspections again early next year as part of the 1999 CSR reporting process. We agree that a self-inspection process can be valuable, provided that grantees have clear, consistent, and appropriate guidance on the procedures for reviewing their case data, determining whether to certify the data, correcting errors, and providing their results to LSC in a standardized way that facilitates validation of the results.

Conclusions

LSC’s 1999 CSR Handbook and other written communications have improved the clarity of reporting requirements for its grantees. However, many grantees remain unclear about and/or misunderstood certain aspects of the reporting requirements. LSC’s practice of disseminating guidance primarily by written or telephone communications may not be sufficient to ensure that grantees correctly and consistently interpret the requirements.

LSC sought to determine the accuracy of grantees’ 1998 case statistics by requiring grantees to conduct self-inspections. However, we do not know the extent to which the results of the self-inspection process are accurate. The validity of the results are difficult to determine because LSC did not standardize the way that grantees were to report their results, some of the grantees used samples that were too small to assess the proportion of error in their data, some grantees did not correctly follow LSC’s reporting guidance, and LSC had done no verification of the grantees’ self-inspection procedures.

We do not believe that LSC’s efforts, to date, have been sufficient to fully resolve the case reporting problems that occurred in 1997.

Recommendations

We recommend that the President of LSC clarify and disseminate information on the specific information on client assets that grantees must obtain, record, and maintain; clarify and disseminate information on the types of citizenship/alien eligibility information grantees must obtain, record, and maintain for clients who receive legal assistance only over the telephone; clarify and disseminate LSC’s criteria for single recording of cases; clarify and disseminate LSC’s policy concerning who can provide legal assistance to clients for the service to be counted as a case; explore options for facilitating correct and consistent understanding of reporting requirements, such as developing and disseminating a training video for grantee staff; develop a standard protocol for future self-inspections to ensure that grantees systematically and consistently report their results for open and closed cases; direct grantees to select samples for future self-inspections that are sufficient to draw reliable conclusions about magnitude of case data errors; and finally, ensure that procedures are in place to validate the results of LSC’s 1998 self-inspection, as well as of any future self-inspections.

Agency Comments

The President of LSC provided written comments on a draft of this report, which are printed in full in appendix I. LSC generally agreed with our findings and stated that it intends to implement our recommendations.

Concerning the issue of the clarity of case reporting guidance, LSC’s letter stated that LSC is troubled by our finding that some grantees are continuing to have difficulty in this area. LSC’s letter reiterated a point that we made in the report; namely, that the overlap in time between our data collection effort and LSC’s distribution to grantees of a July 1999 program letter intended to clarify client eligibility documentation requirements may have caused some of the executive directors not to factor in the new guidance when they responded to our telephone interview questions. Although, as we state on pages 9 and 10 of the report, some respondents may not have been familiar with LSC’s July 1999 guidance, two respondents told us that they were familiar with the new guidance and that they were still unclear about the requirements dealing with assets. We also state on page 10 of the report that our own analysis indicated that LSC was not consistent in its guidance to grantees on what asset information they needed to obtain from clients. Therefore, we do not believe that grantees’ lack of clarity concerning reporting requirements on client eligibility was due solely to the overlap in the time period of our data collection and LSC’s distribution of the July 1999 program letter.

With respect to grantees’ lack of clarity about how to determine duplicate case counting, LSC indicated that it will consider providing additional guidance to grantees through further revisions to the CSR handbook. LSC also indicated that it will revise the handbook to clarify the issue of when legal assistance by nonlawyers can be reported as a case.

LSC also reiterated that it is developing additional methods to supplement written guidance, including (1) conducting training and technical assistance, (2) developing case management standards that would detect and prevent cases from being accepted and reported if the required eligibility documentation was not obtained, and (3) developing database queries for grantees to apply to their case management systems to identify instances where cases lacked required compliance documentation.

Concerning the self-inspection results, LSC noted that its principal purpose in requiring grantees to self-inspect their 1998 case data was to ensure that case-reporting guidance was being properly applied. LSC stated that assessing the accuracy of the 1998 case statistics that grantees submitted to LSC in March 1999 was a secondary purpose of the self-inspection. LSC believes that the self-inspection increased grantees’ awareness of reporting requirements and prompted them to make changes in their case reporting practices. Indeed, in LSC’s view, the self-inspection requirement resulted in more program improvements than did the 1999 CSR handbook and other written guidance issued by LSC because grantees that identified significant problems were required to take corrective actions. LSC notes correctly that, in relation to the self-inspection, our report focuses on the accuracy of LSC’s determination of the number of grantees that certified their data, and the factors that affected the accuracy of LSC’s estimate of the total number of closed cases in 1998. Our review focused on the stated goals of the self-assessment. Although we did not assess whether, relative to LSC’s other efforts, the self-inspection had a greater effect on grantees’ awareness of and compliance with CSR reporting, we do report that 26 grantees made corrections to their 1998 data as a result of their self- inspections.

As arranged with your offices, unless you publicly announce the contents of this letter earlier, we plan no further distribution until 7 days after the date of this letter. At that time, we will send a copy of the report to the Chairmen and Ranking Minority Members of LSC’s appropriations and legislative committees and to Mr. John McKay, the President of LSC. The major contributors to this report are acknowledged in appendix II. If you or your staff have any questions concerning this report, please contact me or Evi L. Rezmovic on 202-512-8777.

Comments From the Legal Services Corporation

GAO Contacts and Staff Acknowledgments

GAO Contacts

Acknowledgments

In addition to those named above, Mark Tremba, Kristeen McLain, Jan Montgomery, David Alexander, Barry Seltser, Lemuel Jackson, Brian Lipman, and Walter Vance made key contributions to this report.

The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touch- tone phone. A recorded menu will provide information on how to obtain these lists.