Adult Education: Measuring Program Results Has Been Challenging
Released: 1995-09-08T00:00:00

Letter

Introduction

The National Adult Literacy Survey estimated that approximately 90 million American adults have deficient literacy skills. Of those, between 40 and 44 million adults—about 22 percent of the country’s adult population—have severe problems with literacy, defined as the ability to read, write, and speak English and compute and solve problems proficiently. An additional 50 million adults are likely to encounter some problems functioning in society and need improved literacy skills.

The Adult Education Act (AEA) is administered by the Department of Education. The act represents the primary federal effort to alleviate problems in adult literacy and provides the basic legislative authority and largest source of federal funds for programs that benefit educationally disadvantaged adults. The act’s largest program is the Adult Education State-Administered Basic Grant Program (State Grant Program). In fiscal year 1995, federal funding for this program was $252 million, while state and local sources provided $890 million, or 78 percent of the program’s total budget. For the first 15 years of the State Grant Program (1966 to 1980), federal expenditures exceeded total state and local contributions; however, total state and local contributions have since surpassed federal expenditures. (See table I.2 in app. I for total annual expenditures.) Figure 1.1 compares federal expenditures with state and local expenditures since the AEA’s passage in 1966.

Although total state and local contributions currently far exceed federal expenditures, federal dollars still total more than half the funds for adult education in almost half of the states. The contribution of each state relative to the federal contribution varies widely, depending on each state’s commitment to providing adult education services. For example, in fiscal year 1991, state and local contributions ranged from a low of 21 percent to a high of 96 percent; conversely, federal expenditures ranged from 4 to 79 percent. Since 1992, the AEA has restricted the federal share of each state’s expenditure to no more than 75 percent. (See table I.4 in app. I for further information on expenditures by state.)

The AEA makes grants to states and requires that they be used in accordance with federally approved state plans. In developing their plans, states must assess the needs of adults, including educationally disadvantaged adults, and the capability of programs and institutions to meet those needs. The Department of Education annually makes its grants to states on the basis of the number of individuals in each state who are at least 16 years old, not enrolled in school, and lack a high school degree or General Educational Development (GED) credential. Local adult education providers then apply to the states for funds.

Following are the three most common types of instruction offered under the State Grant Program:

Adult Basic Education (ABE), which is instruction designed for adults functioning below the eighth grade level;

Adult Secondary Education (ASE), which is instruction designed for adults functioning at the secondary level that may culminate in a high school diploma or may serve as preparation for the GED examination; and

English as a Second Language (ESL), which is instruction designed to teach English to non-English speakers. (See table I.3 in app. I for further information on enrollment by instructional area.)

Programs funded under the AEA are important in providing basic literacy skills needed by clients of federal employment training programs such as Perkins Vocational Education (VOC ED), administered by the Department of Education; Job Opportunities and Basic Skills (JOBS), administered by the Department of Health and Human Services; and the Job Training Partnership Act (JTPA), administered by the Department of Labor. Consequently, the AEA and employment training legislation require coordination among these programs to avoid duplication and enhance service delivery.

The National Literacy Act of 1991 amended the AEA and authorized several new programs. Major provisions included the creation of the National Institute for Literacy, the establishment of state and regional literacy resource centers, and a requirement for the Department of Education to develop model indicators of program quality to guide states in developing their own indicators for improved program evaluation.

The 104th Congress is considering legislation that would consolidate adult education and other programs and provide one or more block grants to states. The Senate bill would repeal most existing federal employment training programs, including the State Grant Program, and replace them with a single block grant. The House bill would also repeal most employment training programs but replace them with four block grants, including a separate grant for adult education and literacy programs.

Objectives, Scope, and Methodology

At the request of the Chairman and Ranking Minority Member of the former House Committee on Education and Labor and the Chairman and Ranking Minority Member of the former Subcommittee on Elementary, Secondary, and Vocational Education, we reviewed several issues related to the AEA. Specifically, we examined the goals of the AEA and its largest program (the State Grant Program), the population served by the program, program services, and its coordination with federal employment training programs and the extent to which the State Grant Program ensures accountability for program quality and results, including how states have implemented quality indicators.

We focused our review primarily on the State Grant Program because it is the largest of the AEA’s funded programs. In fiscal year 1995, 83 percent of AEA funds were allocated to this program.

To obtain nationwide information on the State Grant Program, we interviewed federal officials from the Department of Education. We also reviewed Department of Education data and recent national studies, including the National Evaluation of Adult Education Programs and the National Adult Literacy Survey.

We selected three states for closer review: California, Connecticut, and Iowa. We selected these states because they provided some geographic dispersion and represented a range of (1) state and local financial commitments (as demonstrated by the percentage of matching funds each state contributes), (2) program size (as demonstrated by dollars and enrollment), and (3) ESL enrollment levels. Within each state, we visited at least two communities that we selected with the help of state adult education officials. We selected communities that represented different types of locales (urban, suburban, rural) and were involved in a variety of local coordination activities.

To identify the goals of the AEA, including the State Grant Program, we reviewed federal legislation. To determine the populations served and services provided by the program, we reviewed Department of Education data and national studies. We also interviewed local adult education providers.

To provide information on the coordination of AEA programs with employment training programs, we interviewed federal officials at the Departments of Education, Labor, and Health and Human Services, and held discussions with national experts, including representatives of the National Institute for Literacy. We also reviewed studies on coordination. In the states we visited, we met with state officials from adult education and JOBS, JTPA, and VOC ED programs. In Iowa and parts of California, where the community college system is the major adult education provider, we also met with state community college representatives. At local levels in the three states, we met with adult education providers as well as representatives of local employment training programs.

To provide information on program accountability and quality, we interviewed Department of Education officials and held discussions with national adult education experts. We also reviewed the Department’s model indicators of program quality and studies on program accountability and quality issues. In addition, we interviewed state and local officials in California, Connecticut, and Iowa and reviewed program documents, including the quality indicators developed by these states.

We conducted our work between November 1994 and August 1995 in accordance with generally accepted government auditing standards.

Largest AEA Program Strives to Meet Many Goals and Needs

The AEA is a broad and flexible act and its largest program, the State Grant Program, reflects this. The State Grant Program, the federal government’s primary adult education program, has many goals and enables people with a wide range of needs to receive instruction from a variety of service providers. Many clients of employment training programs are among those needing the basic skills taught by adult education. Although the program has some restrictions, it allows states considerable flexibility in the types of instruction they fund with their federal grants as long as they fund programs in accordance with federally approved state plans. In keeping with their state plans, which call for coordinating adult education with employment training programs, a variety of coordination activities were taking place in the states and communities we visited.

Goals Are Broad

Recognizing the wide range of adult literacy needs in this country, the Congress passed the AEA with broadly stated goals. Although adult education programs are commonly viewed as the means to obtain a high school diploma or its equivalent, the AEA established goals that are far broader and include citizenship and employment as well as the overall improvement of the adult education system. Specifically, the purpose of the AEA is to improve educational opportunities for adults who lack literacy skills necessary for effective citizenship and productive employment; expand and improve the current adult education delivery system; and encourage the establishment of adult education programs for adults to (1) acquire basic skills needed for literate functioning, (2) acquire basic education needed to benefit from job training and obtain and keep productive employment, and (3) continue their education to at least the secondary school level.

Adult Education Students Are Diverse

Adult education students have diverse needs, circumstances, and personal characteristics. A student might be a high school dropout, a client in a job training program, an immigrant or refugee, a displaced worker or homemaker, an adult in the workplace, a welfare recipient, or a retiree. Students who enroll in adult education classes vary in age, race and ethnicity, and employment status.

In one program we visited in a rural California town, a 23-year-old refugee was enrolled in an Adult Secondary Education (ASE) class. He was a machine operator with an eleventh grade education who planned to earn his GED and become a bilingual teacher. A 45-year-old unemployed mother of four with a third grade education was enrolled in an English as a Second Language (ESL) class. Having done seasonal work in the past, her goal was to obtain a GED and find work in the nursing field.

In an urban Connecticut program, a 19-year-old mother on welfare wanted to complete high school and become a cosmetologist. Although she had a tenth grade education, she needed the basic skills taught in an Adult Basic Education (ABE) class. In the same program, a 62-year-old immigrant who had been an accountant in Russia was attending ESL classes. Her goal was to become a U.S. citizen. In a suburban program in Connecticut, a 28-year-old part-time stock clerk was enrolled in an ASE class. He had a ninth grade education and lived with his parents. He hoped to earn his GED, attend college, and become a police officer.

In a rural town in Iowa, a 48-year-old father of four from Laos spoke no English and was enrolled in a beginning ESL class. He worked part-time as an upholstery worker but hoped to learn English well enough to get a full-time job. In a city in Iowa, a married, 35-year-old mother of three was enrolled in an ABE class. A former welfare recipient, she had a job as a child care aide that was contingent upon her earning a GED. Her goal was to earn the GED and keep her job.

National statistics also suggest that adult education students are fairly diverse. Nationwide, 38 percent of students enrolled in adult education classes in 1993 were between the ages of 16 and 24, 46 percent were between the ages of 25 and 44, and the remaining students were 45 years old or older, according to the Department of Education. Also, 36 percent of the students were white, 31 percent Hispanic, 18 percent black, 14 percent Asian or Pacific Islander, and 1 percent American Indian or Alaskan Native.

The National Evaluation of Adult Education Programs conducted a survey of students who entered the adult education system between April 1991 and April 1992. It found that 42 percent of these students were employed, and 58 percent were either unemployed or not in the workforce when they enrolled. During the year before enrollment, 43 percent of ABE students, 31 percent of ASE students, and 14 percent of ESL students received public assistance or welfare payments.

Many Employment Training Clients Need Adult Education Services

Many clients of federal employment training programs rely on the State Grant Program for the basic skills they lack. According to the Department of Labor, unless an attempt is made to upgrade the literacy skills of clients in federal employment training programs, clients’ success may be limited and access to the job market may be denied. Nationwide, almost 30 percent of JTPA clients are school dropouts, and as many as half may lack basic skills. One-fourth of JOBS clients in fiscal year 1992 were enrolled in a high school completion program.

Adult education enrollment has risen almost every year since 1966, and the Congress is considering welfare reform proposals that may place even greater demands on adult education providers. These proposals may also make the coordination among adult education, welfare, and employment training programs even more critical. Some states are already implementing their own welfare reform efforts that require certain welfare clients to obtain adult education or employment training services to receive assistance. For example, California’s JOBS program requires that welfare recipients have opportunities to remedy basic skill deficiencies and earn a high school diploma or GED credential. The state is currently required to provide adult education to its JOBS clients with low assessment scores and to continue to provide education until clients attain a specified level of proficiency.

Connecticut has piloted a welfare reform program that targets certain welfare recipients. Individuals in the pilot can receive needed remedial services, such as adult education or vocational training, for 2 years before being required to find jobs. According to a state official, the pilot was limited to two communities because of concerns about the state’s ability to provide remedial services to all needy individuals, particularly adult education services. If a client has only 2 years to seek remedial services and faces a waiting list for adult education services, both the client and the entire program are at risk, explained the official.

Similarly, Iowa’s JOBS program has a goal of moving people off welfare within 2 years by providing remedial education and employment training. This welfare reform effort has increased the percentage of welfare recipients required to participate in the JOBS program from 24 to 88 percent. This increase is achieved, in part, by exempting fewer welfare recipients from participating in the JOBS program. For example, only parents with children under 6 months of age are exempt; previously, parents with children under the age of 3 were exempt.

Wide Range of Adult Education Providers Use a Variety of Approaches

Under the State Grant Program, states may fund local educational agencies and a variety of public or private nonprofit agencies,organizations, and institutions to provide adult education classes. Most programs are administered by local educational agencies. Figure 2.1 shows the extent to which different organizations provide adult education.

Many adult education providers use flexible and, in some cases, less traditional approaches to education that may better suit the responsibilities and needs of adult students. For example, to make classes more accessible to adults, providers may offer both day and night classes. Unemployed adults may prefer daytime classes; adults who work or have child care responsibilities may only be able to attend night classes. We also found that some programs offer on-site child care, which can make it easier for parents to attend adult education classes.

The flexible “open-entry/open-exit” feature of most adult education providers may also better suit their students’ lives than the traditional September-to-June school year. The National Evaluation of Adult Education Programs found that 66 percent of adult education programs allowed students to enroll and begin instruction at any time.

Service providers use a variety of instructional methods to meet students’ needs. For example, the principal of an adult school in a rural California town explained that her program provides “individualized instruction,” which means that teachers assess the individual goals and abilities of the students and take these into consideration in planning classroom instruction. Several methods or a combination of methods may then be employed: large group lectures or presentations; small-group instruction, including role play or practice in conversation or writing skills; or one-on-one tutoring if the ratio of aides to students permits it.

Some programs encourage adults who need both basic skills and employment training to enroll in both concurrently; others recommend basic skills training first so that students have the necessary foundation for employment training. Concurrent enrollment, some state and local officials argue, may enhance learning and move adults into the workforce faster.

Within Confines of AEA, States Have Flexibility in Funding Instruction

The AEA limits states’ flexibility in determining how to spend their State Grant Program funds by specifying how a significant portion of the funds are to be spent. However, with the remaining unrestricted funds, the combinations and types of instruction states fund vary greatly.

The AEA specifies that states must spend at least 15 percent of their grants on teacher training and program innovation and at least 10 percent on programs serving incarcerated or institutionalized adults. No more than 20 percent can be spent on programs for certificates of high school equivalency, and no more than 5 percent can be spent on state administration.

States can decide on the types and combinations of instruction they wish to fund as long as they meet the AEA’s set-asides and fund programs in accordance with their state plans. Most providers offer the three most common types of instruction—ABE, ASE, and ESL. Enrollment in each varies greatly by state and community. Table 2.1 shows how enrollment levels vary by instructional area nationally as well as in the three states we visited.

States and Communities Coordinate With Employment Training Programs in a Variety of Ways

In keeping with state plans, which call for coordinating adult education with employment training programs, a variety of coordination activities were taking place in the states and communities we visited. These activities, however, were not easy to establish. They took time to develop and often depended on the perseverance of agency staff and local service providers. State and local coordination efforts included pooling funds, establishing one-stop centers, and developing uniform assessment systems.

Pooling Funds

Connecticut has pooled funds from many sources for Coordinated Education and Training Opportunities grants. When a service provider receives a grant, it may contain funds from one or more funding sources. These coordinated grants are implemented through regional workforce development boards responsible for a range of tasks, including identifying local needs, evaluating grant proposals, and overseeing operations. These grants have the advantage of allowing service providers to deal with a single planning process and a single request for proposal. However, according to officials of one regional workforce development board, although these grants may make things “seamless” for the client, the service provider still must meet all the federal reporting requirements of their many funding sources.

In Iowa, the community colleges coordinate funds from the State Grant, JTPA, and VOC ED programs. Each of the state’s 15 community colleges administers the State Grant Program and offers adult education classes. In addition, half of the colleges administer JTPA programs. Services from these many programs are often administered by staff who are both located at the college and operate within the same department. This arrangement facilitates coordinated program planning, service delivery, and referral of clients to multiple programs. At one college, administrators from the adult education, JTPA, and VOC ED programs told us that their close proximity enabled them to review a client’s total needs and provide the maximum allowable services. For example, a welfare recipient might receive adult education instruction from the State Grant Program, a clothing allowance from the JTPA program, and a transportation subsidy from the JOBS program.

Some of the coordination between California’s JOBS program and State Grant Program takes the form of financial support from many agencies. Adult education programs that serve the state’s JOBS clients receive adult education and JOBS funds and may also draw funds from an 8-percent set-aside of JTPA funds for education programs that are matched by the state. In some counties, the state’s JOBS program pays for adult education programs to meet JOBS’ data collection and reporting requirements, which include administering the same competency-based assessment to all state JOBS clients. Adult education does not cover these costs. One adult education principal told us that, if not for the JOBS program’s covering these costs, program staff would not be able to do as much record keeping or assessment as they do.

Establishing One-Stop Centers

All three states we visited had begun efforts to establish one-stop centers. These centers are intended to help clients who need services from many programs find all of the services they need in one location or go to a single location to access information about the services they need. Officials in one community spoke of the administrative burden imposed by the multiple federal program requirements of establishing a one-stop center.

All three states recently received grants from the Department of Labor to pilot one-stop centers, which are being established around the country even in places that have not received Department of Labor grants. The one-stop center we visited in California was established without a grant from Labor. The Department of Labor’s one-stop grants support voluntary state coordination. All three states considered adult education important and, thus, included adult education officials in planning their efforts.

With the help of the Department of Labor’s one-stop grant, Iowa recently opened its first one-stop center to provide services to clients of employment training programs. Adult education staff were on site to perform client intake and assessment. Adult education instruction has been offered on site since February 1995.

Connecticut was using its Department of Labor grant to develop one-stop centers, where client intake and evaluation would take place and where clients could be referred to multiple agencies for services they need. Local officials in one community said their goal was to develop three centers and install computers in libraries, bus terminals, and shopping malls so the public could access information on local services, such as adult education classes.

One community we visited in California set up a one-stop center without a Department of Labor grant. To prevent unnecessary duplication of services and facilitate successful completion of training and the transition to employment, this center established linkages to more than 100 agencies and businesses. Features of the center included a central information line, career library, computerized career assessment, and on-site employment interviews.

Developing Uniform Assessment Systems

A single assessment system used across state agencies can facilitate coordination and make access to services easier for clients. Using one system allows clients to move easily among education and training programs, provides a common assessment vocabulary so that all agencies can determine initial client proficiency levels as well as ongoing progress, and minimizes duplicative or unnecessary testing of clients. However, not all adult education and employment training officials agree that a single assessment system can appropriately measure adults’ skills.

To varying degrees, Connecticut and California were using common assessments. Connecticut required that adult education, JTPA, and JOBS programs all use the same assessment system. California’s JOBS program uses the Comprehensive Adult Student Assessment System (CASAS) for assessing its clients, but the state’s adult education program uses CASAS only on a sample of programs. Iowa was piloting CASAS but only for use by adult education providers.

Measuring Program Results Has Proven Difficult

Measuring results in the State Grant Program has proven difficult because program objectives have not been clearly defined and questions exist about the validity and appropriateness of student assessments and the usefulness of nationally reported data on results. Although the Department of Education has focused on developing model program indicators that states could use to evaluate local programs, experts and program officials disagree about whether the indicators alone will enhance accountability. Efforts to enhance the evaluation capabilities of state agency staff and improve data collection continue, but it is too early to assess their impact.

Difficulties in Establishing Program Objectives

Evaluating program results depends on clear program objectives as well as criteria for measuring the achievement of those objectives. The broad objectives of the State Grant Program give the states the flexibility to set their own priorities but, some argue, they do not provide states with sufficient direction for measuring results. Moreover, reaching a consensus on measurable objectives for adult education is difficult.

Because the State Grant Program’s objectives are so broadly defined, state officials have developed a variety of views on measuring program results. For example, some officials told us that they might measure program success by whether adults gained the skill to read to their children and, thus, contribute to their children’s literacy. Others might focus on whether adults can read street signs or the newspaper. And, in one state we visited, an official contended that completing high school and finding productive work should be the objectives of the states’ adult education programs because completing a basic skills program and becoming a citizen are no longer sufficient to succeed in society.

Several experts and program officials told us that the State Grant Program lacks a coherent vision of the skills and knowledge adults need to be considered literate. Similarly, some state officials said that they would like the federal government to further specify the types of results expected from state adult education programs.

Reaching consensus on measurable objectives, however, may be difficult since research findings are often inconclusive about the long-term benefits to adults of achieving various program results. For example, many adult education programs focus on preparing adults to take the GED examination as a means of high school completion. Yet research findings are mixed about whether GED attainment reflects increased literacy skills and whether GED recipients are economically better off than high school dropouts.

Concerns Raised About Adult Student Assessments

Ensuring accountability has also been hampered by limitations in the assessment instruments used to measure student outcomes in adult education programs. The research literature raises questions about the validity of standardized tests used to measure adult literacy, and local program staff have questioned the appropriateness of using these assessments to measure program results.

The AEA requires states to gather and analyze standardized test data as one way of evaluating local programs. These assessments tend to focus on either academic skills or functional literacy. Academic tests, such as the Tests of Adult Basic Education (known as “TABE”), focus on measuring such basic skills as reading comprehension, vocabulary, language expression, and mathematical proficiency. Functional literacy or competency-based tests, such as the Comprehensive Adult Student Assessment System (CASAS), focus on the ability to perform literacy-related tasks in situations faced by adults in everyday life at home, at work, or in the community.

Experts have questioned the validity of both the academic and functional literacy tests used in adult education programs. For example, two recent reviews point to a lack of normative data for the age ranges of participants in most adult education programs. Functional literacy tests may lack validity because they are not derived from theoretical models of ability but from everyday literacy tasks. According to a recent review, without further analyses, the instructional implications of test performance are unclear.

Thus, these assessments may not provide useful information about the skills and needs of adult students.

A more serious problem affecting the validity of assessments is the lack of research examining the long-term retention of learning gains in adult education programs. According to one researcher, a comprehensive search did not uncover a single published study on the effectiveness of adult education programs in helping adults retain the skills they may have acquired during instruction. This being the case, improved test scores may not necessarily mean that adults will be better equipped for high- skilled jobs, function better as parents, or participate more fully as citizens.

However, officials in the three states we visited felt that competency-based assessment systems could be useful in measuring progress in local adult education programs and, thus, strongly advocated these systems. California had developed CASAS and required its use in a sample of one-third of its adult education programs. Connecticut had designed its own competency-based testing system (adapted from CASAS) and required its use in all adult education programs. Iowa had recently decided to move toward a competency-based system and was piloting CASAS in 9 of its 15 community college districts.

Local adult education and employment training staff had mixed views about their states’ competency-based assessments. Some local program staff saw the CASAS assessment system as a valuable and flexible tool. However, some English as a Second Language (ESL) teachers were dissatisfied with the CASAS test as a measure of how well adult education students learned to communicate in English. And some employment training staff said that the CASAS test did not give them sufficiently specific information about their clients or focused too much on life skills.

Finally, several local staff questioned the appropriateness of CASAS as the sole assessment tool and, therefore, used CASAS in conjunction with other tests. Administrators and experts also told us that they thought no single test could measure all relevant aspects of student performance.

Adult Student Data Missing or Inaccurate

The poor quality of the data on adult education students collected at state and local levels also hampers accountability. Federal and state officials as well as recent studies have cited problems with these data. The studies have attributed difficulties in obtaining accurate data to the sporadic attendance patterns of adult students and the limited time and expertise of local adult education program staff.

State officials are required to submit to the Department of Education annual statistical performance reports that include information on students served by local programs. State-submitted reports include (1) the number of students served and their demographic characteristics, (2) the skill levels of students when they start adult education programs, (3) student progress over the program year, (4) eight types of student achievements, and (5) the number of students who do not complete their objectives and their reasons for separation. The reports also include information on program staff and the types of instructional settings in which students are served.

Department of Education officials acknowledged serious problems with the quality of the statistical reports, some of which are based on double counting or undercounting of students in adult education programs. Another Department official charged that many of the data are questionable and that very few local programs have record systems that allow them to report the data the Department requires.

Comments of officials in one state confirmed these data problems. They said that they did not have all the information the Department requires for their statistical reports because too many resources are required to collect the data. As a result, they simply do not report some of the data elements and provide estimates of the other information. They noted that the data they submit need not be certified and that the Department has never audited their statistical reports. Furthermore, they asserted that these data have nothing to do with receiving federal funds. The only thing that really counts, they said, is the number of adults in the state who do not have diplomas because that is what drives the funding formula.

Also, some local staff failed to see the utility in collecting the data that states require for reporting to the federal government. Some said they thought that the information they are required to report does not accurately reflect the accomplishments of their adult education students.

Difficulties in obtaining accurate data can also be attributed to attendance patterns of adult students and the limited capacity and expertise of local program staff. The open-entry/open-exit feature of many programs adds to the difficulty of tracking adult students. Because students may not stay in the program long or may attend on a sporadic basis, program staff do not always have sufficient information to report on student progress or results. Because local programs have difficulty following up on students, program officials may rely on information reported by teachers or the students themselves.

In addition, many local programs lack sufficient staff to handle data collection and reporting responsibilities, according to a survey of adult education programs in nine states. Programs are typically staffed by part-time personnel, and these responsibilities become an extra burden. Also, local program staff may lack expertise in collecting assessment data that can help track program effectiveness. For example, when the National Evaluation of Adult Education Programs asked local adult education program staff to provide certain assessment data, it found that about one-third of the information was invalid because (1) the wrong test forms were used, (2) data were inaccurately recorded, or (3) tests were administered at the wrong times. Similarly, as Connecticut began to implement a new assessment system statewide, administrators discovered that they needed to clarify program guidance because some local programs were mistakenly measuring literacy gains using a test designed solely for student placement.

Efforts to Improve Program Quality and Accountability

Federal efforts to improve quality and accountability have focused on (1) developing model indicators; (2) providing technical assistance to states and local programs on data collection, assessment, and developing performance standards and measures; and (3) requiring states to set aside funds for training and demonstration projects.

Federal and State Program Quality Indicators

Provisions of the National Literacy Act focus on improving quality in adult education programs by requiring the Secretary of Education to develop indicators of program quality. The indicators were to be used as models for judging state and local programs receiving federal funding. States were also required to develop and implement their own indicators, which might or might not correspond to the federal model, and use them to evaluate state and local programs.

The Department of Education developed model indicators by (1) reviewing adult education indicators already being developed by various states and indicators used by other federal programs, (2) meeting with experts and adult educators, (3) commissioning background papers by experts in the field, and (4) conducting workshops for state directors who would be responsible for developing and implementing the state indicators. The resulting eight model indicators of program quality are listed in table 3.1. The indicators cover student outcomes, that is, learner progress toward attainment of basic skills and competencies and learner advancement in the program. They also focus on recruiting and retaining adult education students and other indicators of program quality—planning, curriculum and instruction, staff development, and provision of support services.

The Department did not attempt to set performance standards for adult education programs but limited its work to developing indicators and providing some sample measures for each indicator. The Department defined an indicator as a variable that reflects effective and efficient program performance. It is to be distinguished from a specific measure used to determine the quantitative level of performance for the indicator. For example, to measure learner progress, states could use standardized test score gains, teacher reports of gains in communication competencies, or alternative assessment methods (such as portfolio assessments, student reports of attainment, or improvements in specific employability or life skills). An indicator is also to be distinguished from a performance standard, which defines acceptable performance in terms of a specific numeric criterion.

The National Literacy Act also required states to adopt indicators by July 1993 and use them to evaluate local programs. States were required to adopt, at a minimum, indicators for recruitment, retention, and student learning outcomes. However, decisions about whether to adopt the Department’s model indicators, what measures to use, and whether to develop performance standards were left to the states. A review of amendments to state adult education plans submitted in July 1993 showed that for the most part states had adopted indicators similar to the Department’s model, especially in the areas of student outcomes, recruitment, and retention. However, states were less consistent in how they measured indicators. The review found that states were using different standardized tests to measure learner progress and had defined learner advancement in different ways. A 1995 survey of state adult education directors showed that 16 states had implemented standards and 8 states had developed but not yet implemented standards.

Each of the three states we visited had developed standards for student outcomes, but not all of these standards were readily quantifiable. California had developed standards for seven levels of language proficiency for ESL students (the majority of adult education students in the state) but had not yet quantified performance on specific assessment measures. Standards for other kinds of adult education students in California had not yet been completed. Connecticut had set standards for educational gains expected over a specific time period and measured their achievement using test scores and the number of course credits or competencies attained. Iowa had set standards for grade level increases on standardized tests and for the performance of GED graduates on the GED exam. Since Iowa had not yet determined a specific strategy for competency-based education, the state had not yet established standards for competency-based tests.

Experts as well as federal and state officials with whom we spoke disagreed about whether developing indicators would improve accountability and program quality. Some were concerned that the indicators do not move the field forward because they do not specify the types of results the federal government expects from state and local programs. One federal official doubted whether the indicators alone would help state and local programs collect higher quality data. However, other experts told us that they thought the indicators were a good first step. Still others said that the federal government should not be setting standards for states because states’ literacy problems and clientele differ.

Too Soon to Assess Efforts to Enhance Accountability and Program Quality

It is too soon to tell whether state-developed indicators, measures, and performance standards will result in the collection of more useful data or help states evaluate local programs since the 1993-94 program year was the first year in which indicators were to be used for evaluation. One state we visited planned to use information collected during the 1993-94 program year as baseline data and begin to hold local programs accountable for performance on the state’s indicators in subsequent years.

Other federal efforts have been initiated to help states develop better accountability systems. Two of these efforts are designed to help build the capacity and expertise of state adult education staff to evaluate local programs. In 1993, the Department hired a contractor for a 3-year technical assistance effort designed to assist state education agencies with assessment, evaluation, and the development of performance standards and measures. And, in 1993, the National Institute for Literacy awarded grants to five states to develop performance measurement systems for literacy, with a specific focus on integrating systems used by different agencies that provide literacy services.

Department officials also told us that they were acting to improve the quality of data collected on adult education programs. In concert with state adult education directors, the Department has been examining whether to modify the existing federal reporting requirements. They have held several meetings but have not yet issued any recommendations. In addition, the Department has developed and tested an automated management information system that would allow programs to collect data on individual students and a computer program that would help states more easily convert data they collected to the statistical reports required by the Department. A field test of the management information system in selected local programs in five states revealed that local staff appreciated the system’s report-writing capabilities but remained highly resistant to performing data collection and entry.

In addition to these efforts, the requirement that states set aside a portion of their federal funds for demonstration projects and training may also help states move toward better accountability systems. Although the Department has not completed an ongoing national evaluation of the use of these funds, state officials asserted that the set-aside was critical to their efforts to improve program quality. All three states had used these federal funds, in part, to develop competency-based instruction and assessment systems; they had also used the funds to address state-specific issues. California had developed a training institute for ESL teachers, Connecticut had used some of the funds to help implement a new statewide management information system, and Iowa had held a state literacy conference to examine how to better measure adult student progress through qualitative assessments.

Conclusions and Agency Comments

Conclusions

The broad goals and flexibility of the AEA and its State Grant Program have resulted in a federal program that is serving many different populations, yet has difficulty determining its target populations, objectives, or a means to measure program results. Although the broad goals and corresponding flexibility give state and local officials the latitude to design programs and quality indicators tailored to their particular needs and priorities, some state officials and experts have voiced concerns that the federal government has not provided sufficient vision and guidance. This poses a challenge for developing accountability measures.

The program has had difficulty ensuring accountability for results—that is, being able to clearly or accurately say what program funds have accomplished. Although the Department of Education relies on federal reporting requirements and program quality indicators to provide this information, the data the Department receives are of questionable value. Because state and local client data are missing or inaccurate, attempts to make the program accountable may be compromised. Until further guidance is developed on measurable objectives and ensuring the quality of client data, state-developed indicators and standards are unlikely to improve accountability.

Agency Comments

In its written comments on a draft of the report, the Department of Education recognized that we identified the three areas that are critically important to improving accountability in adult education: clear purpose and expectations, good assessment instruments, and high-quality data.

The Department also stated its commitment to improving program accountability through several current initiatives. These initiatives include developing an individualized student record keeping system; moving toward an outcomes-based national data collection system; conducting evaluations of delivery systems, effective practice, assessment, and performance measurement; providing technical assistance in designing and using performance measures and standards; and developing training programs for adult education staff in collecting, analyzing, and reporting student and program data. (The Department’s letter appears in app. II.)