Chapter 4

Conducting Reviews to Improve Draft Standards

Before finalizing the content standards, it is critical to conduct a
broad review of the draft standards with state and local program staff,
content experts, relevant professional organizations, psychometricians,
policy makers, community members, and other stakeholders. Through
multiple reviews, the state can confirm that the content standards are
appropriate, accurate, and complete and that they represent broad-based
input. By conducting field and expert reviews, the state can better
support the legitimacy of the standards for use in curriculum,
instruction, and assessment.

The review process helps engage the adult education field in the
development of content standards, creating field ownership of the
standards and providing a context for the implementation of
standards-based education. These reviews can also provide an opportunity
to test how readily programs can implement the standards and identify
the support necessary for effective implementation. Feedback from
reviews helps the writing team revise the draft standards and helps the
state establish the policies and procedures for statewide
implementation.

Chapter 4 discusses three types of review:

-   **Validity** reviews are conducted for three purposes: (1) a review
    of content to determine the extent to which content standards
    represent the knowledge and skills most valued for adult learners to
    succeed in their multiple roles; (2) a review for bias to eliminate
    aspects of the standards that might bias learning for particular
    groups; and (3) a review for measurability to verify that the
    standards can be used to assess learner progress and performance.

-   **Alignment** reviews help determine the consistency between the
    draft standards and external benchmarks, curricula, assessments, or
    other standards within the state.

-   **Implementation** reviews are conducted to refine draft content
    standards and to help identify the processes and procedures that
    need to be established for the successful introduction and
    implementation of the standards.

Validity and alignment reviews may have already been conducted by the
writing team as part of the process of drafting content standards (see
the Draft Standards and Align Assessments and Curricula with Standards
sections in chapter 3). This chapter focuses on the role of multiple
stakeholders and experts from the field in conducting the reviews. In
addition, this chapter discusses specific strategies that the state can
use in conducting reviews:

-   **Statewide surveys** offer cost-effective ways to elicit feedback
    from a broad range of stakeholders.

-   **Focus groups** allow participants to share and explore ideas
    orally in an open discussion format.

-   **Expert reviews** offer special content and research expertise for
    evaluating the
    standards.![](media/image1.wmf){width="1.8395833333333333in"
    height="0.6638888888888889in"}

-   **Field tests** assess how effectively the standards will meet the
    needs of learners and educators in practice.

No single strategy can provide all the information needed to establish
validity, alignment, and feasibility for implementation. Therefore,
multiple strategies can be used for each purpose. Exhibit 4.1 identifies
some of the strategies used to conduct specific types of review.

Exhibit 4.1. *Strategies for Conducting Reviews*

  ------------------- ------------- ---------------- ---------------------
  Strategy            Review Type                    

                      Validity      Alignment        Implementation

  Statewide survey    X                              X

  Focus group         X                              X

  Expert review       X             X                

  Field test          X             X                X
  ------------------- ------------- ---------------- ---------------------

-   A validity review for content may be conducted in a variety of
    ways---through use of a statewide survey, focus groups, experts, or
    a field test. A statewide survey combined with any of the other
    strategies will enable the state to gather more in-depth
    information. Validity reviews for measurement and bias are best
    conducted via experts who are knowledgeable in these areas. A field
    test also can be informative, as it shows how different populations
    respond to the standards as they begin to develop strategies to
    assess learner progress.

-   An alignment review is conducted by individuals who have experience
    using the specific documents against which the standards are being
    aligned. In addition, a field test may be useful in gathering
    information about the alignment of standards with curricula and
    assessments.

-   An implementation review is conducted through a field test.
    Participants may engage in focus group discussions or complete
    surveys to provide feedback during the field test.

The review process, like drafting standards, takes time. It also
requires thoughtful planning to enable the state to obtain the feedback
necessary to refine the standards.

# Plan and Manage the Review Process

The number and types of reviews that a state will conduct depends, in
part, on state policies, requirements, and resources. For example, if a
state requires that adult education content standards be aligned with
other state standards (e.g., K--12 or workforce standards), then the
state must build in time and resources to conduct an alignment review to
determine the extent to which the sets of standards are aligned.

In deciding which reviews to conduct, the state should carefully
consider its resources in combination with other factors that can affect
conducting reviews: size of the state, number and location of adult
education programs, timeline for completing reviews, expertise of staff,
and political constraints. The state also should consider the kinds of
experts needed for its reviews of the standards. Most importantly, it
should consider how best to reach the most stakeholders given state
resources.

The state should identify someone to coordinate the review process and
oversee the development of a plan for conducting the reviews and
analyzing the feedback. This coordinator could be a state manager or
member of the coordinating committee. If the state plans to conduct
several different reviews, it may want to invite different members of
the coordinating committee to be responsible for overseeing each type of
review. Alternatively, if committee members are not available, the state
may consider hiring a private contractor to oversee the review process.

The coordinator of the review process should develop an action plan for
conducting the reviews. The following questions may help in developing
the plan:

-   Are particular reviews mandated by the state?

-   What is the state's purpose in conducting each review? What type of
    feedback is sought?

-   Who will conduct reviews?

-   What resources are available/needed to conduct reviews?

-   What is the timeframe for completing reviews?

-   Which stakeholders or field experts will participate in reviews
    (e.g., teachers or content specialists)?

-   How will feedback from reviews be used? By whom?

If a state decides to hold numerous reviews of different types, the
reviews may result in a large amount of quantitative and qualitative
data. As mentioned above, the state may consider contracting with an
external agency to coordinate the reviews and manage and analyze the
feedback.

# Select Reviews for Specific Purposes

This section describes reviews for validity, alignment, and
implementation. Each review serves a specific purpose. This section also
identifies and discusses representative groups that may participate in
each review.

## Validity Reviews

The underlying purpose of any review for validity is to determine the
extent to which the proposed standards represent the knowledge and
skills most valued for adult learners to succeed in their roles as
workers, citizens, family members, and lifelong learners. To this end,
validity reviews target the criteria for quality standards (see exhibit
3.3 in chapter 3). This section describes three elements to consider in
constructing a validity review---content, bias, and measurability. Each
of these elements requires specific criteria to assess the validity of
the draft standards for a wide range of learners and program types.

A validity review should include individuals who represent different
stakeholder groups and levels of expertise. The selection of reviewers
is based on the purpose of the review and the type of feedback that is
needed.

*Participants from the field* may include adult learners, instructors,
program administrators, content and curriculum specialists, professional
development staff, representatives from local businesses and community-
or faith-based organizations, and members of relevant professional
organizations. To help ensure high-quality, meaningful feedback, the
coordinating committee may want to establish the desired make-up of the
review groups (e.g., by geographical area, occupation, or program type)
and eligibility requirements for participation in the review (e.g.,
minimum years of experience in adult education or content area
expertise). Consider selecting participants from stakeholder membership
lists or soliciting nominations from representative stakeholder groups.

+-----------------------------------------------------------------------+
| A Note on Validity:\                                                  |
| What Is It and Why Do We Need to Know About It?                       |
+-----------------------------------------------------------------------+
| Validity concerns *what* is addressed in the standards and *how well* |
| that content is represented. It is one of the most fundamental        |
| considerations in developing content standards. Evidence of validity  |
| answers the question:                                                 |
|                                                                       |
| *Do the standards target the appropriate content?*                    |
|                                                                       |
| Validity should be built into a standards-based education system from |
| the outset; that is, with the development of content standards.       |
| Establishing the validity of standards requires a systematic process  |
| to determine whether the standards are representative of appropriate  |
| and valued knowledge and skills. Documentation of this process and    |
| validity evidence become increasingly important if the standards are  |
| challenged, typically during the implementation of curricula and      |
| assessments or the release of impact data based on the new standards. |
| Document the processes to establish validity so that data will be     |
| available if they are needed to support the content standards.        |
+-----------------------------------------------------------------------+

*Participants with specific expertise* may be recruited to review a
specific feature of the standards, such as bias or measurability.
Individuals with special expertise can provide current, research-based
perspectives on content standards and concrete recommendations for
revising draft standards and establishing implementation procedures.
Expert reviewers can be recruited by open invitation or by
recommendation. Look to institutions of higher education and large
school districts for recommendations.

**A review for content** can be used to assess (1) the rigor,
specificity, comprehensiveness, and coherence of the content of the
standards; (2) the clarity with which the standards are written and
presented; and (3) the manageability of the standards for teaching and
learning. A key to establishing evidence of content validity is to
target a representative sample of stakeholders to react to and provide
feedback on the draft standards. Content reviews do *not* involve
consensus-building procedures. The approach for conducting this type of
review is simply to collect all field input regarding the draft
standards and provide this input to the writing team for the purpose of
finalizing the draft content standards.

Below are some guiding questions for respondents to use in the content
review.

-   How important is this standard/indicator for learning the content
    area?

-   Does the standard/indicator represent knowledge, skills, or
    abilities necessary for success in the content area?

-   Does the standard/indicator reflect an appropriate level of rigor?

-   Is the standard/indicator specific enough to assure a common
    understanding of learning expectations?

Questions for the content review should focus on the highest level of
specificity possible in the standards. In some sets of standards, that
level would be represented by the standard, and in other cases, it could
be the indicator.

Additionally, respondents should be asked to judge the comprehensiveness
of the complete set of standards and to identify content requirements
that are missing from the draft standards.

-   Is important content missing from the draft standards?

-   Is content that is unnecessary included in the draft standards?

-   Is the set of standards comprehensive in its coverage of the content
    area?

-   Is the set of standards manageable for teaching and learning?

The Practice and Application section at the end of this chapter includes
an exercise that will help the state prepare to conduct a review to
validate the content of standards.

**A review for bias**---also known as a sensitivity or fairness
review---can be used to identify and eliminate aspects of the draft
standards that might bias learning for particular groups. The state
should recruit reviewers who have experience and expertise in one or
more areas of bias and who are representative of the adult learner
population. Issues of potential bias relate to age, gender,
race/ethnicity, culture, disability, socioeconomic status, community
type, and language.

In reviewing the content of the standards for potential issues of bias,
reviewers should consider the following questions.

-   Is the set of standards inclusive of all groups served by the
    content standards?

-   Are any groups of adult learners excluded by the set of standards?

-   Does the set of standards focus on relevant knowledge and skills
    rather than beliefs or opinions that are unrelated to the content
    area?

Feedback from a bias review is then used to refine and finalize the
content standards.

**A review for measurability** is used to establish evidence that the
draft content standards can be used to assess learner progress and
performance. The data gathered from this type of review ensure that the
standards are written and presented in a way that allows them to be
appropriately and consistently measured. Measurability helps to
determine whether the adult learner meets the standards.

For a review for measurability, select individuals with experience and
expertise in educational measurement. These participants should also
have an understanding of the contexts and unique concerns of adult
education. Due to the specificity of this aspect of a validity review,
only a small group of three to five experts may be needed.

Reviewers need to have a clear understanding of the standards for
measuring learner progress. The review is guided by questions such as
the following:

-   Is the standard sufficiently detailed to provide clear expectations
    of what the learner should know and be able to do?

-   Does the standard specify a learning outcome that can be reasonably
    measured?

A measurability review takes into consideration the entire range of
assessments and tests used by programs, including nationally
standardized tests and locally developed informal assessment tools.

## Alignment Reviews

Chapter 3 discussed initial steps taken by the writing team to crosswalk
draft content standards with existing state-adopted standards or other
external benchmarks. However, a crosswalk is not all there is to
alignment. Once the standards have been drafted, alignment reviews are
useful in finalizing the standards and preparing for their
implementation. There are several purposes for conducting alignment
reviews: (1) to determine whether the proposed standards are consistent
with other standards within the state; (2) to ensure that the proposed
standards are, to the extent possible and appropriate, consistent with
other significant policy reforms affecting the state's education system;
(3) to determine whether proposed standards are consistent with external
benchmarks; (4) to align assessments with the proposed standards; and
(5) to align curriculum with the proposed standards.

Participants in alignment reviews should include those with expertise in
the content being aligned. For example, if the state mandates that adult
education content standards align with the K--12 content standards,
recruit reviewers who were involved in the development of the K--12
standards or who have expertise in their
implementation.![](media/image1.wmf){width="1.8395833333333333in"
height="0.6638888888888889in"}

An alignment review determines areas of congruence, alignment, and
contradiction between the draft content standards and the other
document(s). In orienting reviewers to their task, carefully review the
following concepts:

-   **congruence**---coinciding exactly, reflecting the same meaning;
    often using the same language

-   **alignment**---in agreement, complementary; reflecting similar
    meaning without gaps in content

-   **contradiction**---inconsistent with or in opposition; containing
    conflicting text

It is important that reviewers understand that the draft standards do
not need to mirror a document to be aligned with it. The words, phrases,
and structure of the draft standards do not need to be exactly the same
as those of the other document for alignment to be confirmed. This
distinction will be of particular importance if the standards must align
with K--12 content standards. K--12 content standards are designed for
children and not for adult learners; they are not appropriate for adult
education. For this reason, the draft content standards may have a
different sequence or extend beyond the scope of K--12 standards while
maintaining alignment with them.

When determining alignment, focus the reviewers' task with the following
questions.

-   Do the draft standards build on the document with which they must
    align?

-   Do the draft standards contradict the document with which they must
    align?

-   Are these documents in agreement, and, where they are not in
    agreement, is that a conscious and well-rationalized decision?

Areas of contradiction are of most concern and need to be addressed in
discussion by the reviewers. Use a consensus-building process to provide
one set of recommendations on the alignment of the draft standards.
Recommendations of the alignment reviewers should be clear and complete
with explicit suggestions on how to revise the standards to resolve any
areas of contradiction.

When conducting alignment reviews, identify all documents or systems
with which the content standards must align and allow sufficient time to
ensure that each is fully understood by the reviewers. Also, if the
draft content standards must align with multiple documents, it is best
to conduct and document each alignment analysis separately at different
meetings so as not to confuse alignment issues.

**Aligning curriculum** with the content standards is an initial step
toward implementing standards-based education. Alignment reviews can be
used to clearly and simply communicate connections between the
curriculum and the content standards, identify gaps in the content of
curriculum, and make suggestions for filling those gaps. It is important
for reviewers to recognize that a curriculum is likely to go beyond the
scope of the content standards, as the curriculum provides the
instructional scope and sequence and includes suggestions for teaching
strategies, learning activities, and texts and other resource
materials.![](media/image2.wmf){width="1.8395833333333333in"
height="0.6770833333333334in"}

**Aligning assessments** with content standards is a complex process and
one that is critical to the implementation of standards-based education.
Only when aligned with the content standards can assessment provide any
clear indications of the impact of standards-based education reform.

Webb (1999) suggests four criteria for judging the alignment between
assessment and content standards:

-   **Categorical concurrence**---the extent to which content standards
    and assessment address the same content. Do the same or consistent
    categories of content appear in both the standards and the
    assessment?

-   **Depth-of-knowledge consistency**---the extent to which content
    standards and assessment require the same complexity of knowledge.
    Is what is elicited from adult learners on the assessment as
    cognitively demanding as what is expected of learners as stated in
    the standards?

-   **Range-of-knowledge correspondence**---the extent to which the
    breadth of knowledge represented by the standards and that measured
    by assessment are comparable. Is the span of knowledge expected of
    learners by the standard comparable to the span of knowledge
    required of learners to correctly respond to corresponding test
    requirements?

-   **Balance of representation**---the extent to which the knowledge
    represented in both content standards and assessment is equally
    distributed in both items. Are test items equally mapped across
    standards and indicators?

## Implementation Reviews

An implementation review helps the state identify the processes and
procedures that need to be established for the successful implementation
of the standards statewide. It also provides concrete feedback from the
field that the coordinating committee and writing team can use to
finalize the content standards.

Participants in an implementation review should be instructors,
coordinators, and administrators who are broadly representative of state
programs in terms of size, geographic location, type, funding, and
student diversity. In selecting participants for implementation reviews,
it is appealing to recruit staff from stronger programs, and they are
often the first to volunteer. However, having only "star" programs
participate in these reviews can skew the results and complicate
implementation by not providing a realistic picture of how all other
programs will be able to apply a standards-based system.

Implementation reviews address the following types of questions:

-   How well do the standards translate into curriculum and instruction?

-   How well do the standards translate into assessment?

-   What aspects of the standards document are particularly helpful?

-   What would need to be changed or added to make the standards more
    useful?

-   What are the professional development needs of instructors and
    administrators in implementing standards-based education?

-   What policies and procedures need to be in place to support the
    successful implementation of standards-based education?

-   What materials and other resources are needed to support the
    successful implementation of standards-based education?

An implementation review also provides insight on the kinds of
professional development and technical assistance that programs will
require in implementing the standards effectively; the role of the
program administrator in supporting the standards; and the materials,
resources, and staff necessary for supporting standards-based education.
Finally, this review provides concrete feedback from the field that the
coordinating committee and writing team can use to finalize the content
standards.

# Identify Strategies for Conducting Reviews

A variety of strategies can be used to conduct validity, alignment, and
implementation reviews. This section discusses four strategies:
statewide surveys, focus groups, expert reviews, and field tests. Often,
the best way to conduct a specific review is through a combination of
methods. Refer to exhibit 4.1 (earlier in the chapter) for the
strategies applicable to particular
reviews.![](media/image3.wmf){width="1.8451388888888889in"
height="0.5722222222222222in"}

There are some common steps in conducting surveys, focus groups, expert
reviews, or field tests. Knowing these commonalities up front can help
guide the planning process and combine efforts to save time and money.
Below are steps to consider.

-   **Recruit participants.** Seek broad representation by soliciting
    names from different stakeholder groups (e.g., learner and teacher
    associations, business community members, and education leaders).

-   **Recruit materials developers and group** **facilitators.** Provide
    the developers and facilitators with background information on
    standards-based education and the state planning decisions that
    influenced the development of the draft standards. Members of the
    coordinating committee can serve as facilitators. They can describe
    the intent of the standards movement and provide the reasoning
    behind key decisions made during earlier stages of the development
    process. Members of the coordinating committee may also be able to
    help identify professionals with expertise in questionnaire design
    or survey development and suggest key personnel to manage or
    coordinate the field test.

-   **Provide sufficient background information as part of all
    reviews.** Survey instruments should include a short informational
    summary of the standards initiative before the specific items listed
    for response. Members of focus groups, expert reviewers, and field
    test participants will also need a summary and the draft standards
    documents to review prior to a focus group, expert panel review, or
    field test. Having independently reviewed and reflected on the
    documents before meetings will allow participants to make
    contributions that are more
    significant.![](media/image1.wmf){width="1.8395833333333333in"
    height="0.6638888888888889in"}

-   **Orient reviewers to the purpose and format of the review.** Begin
    a review session with a brief orientation for participants to
    provide the context for their work. Describe state requirements and
    expectations regarding the standards, summarize the development of
    the draft standards, and provide the questions or feedback forms
    that will be used in the review. It is important that all
    respondents and reviewers understand the value of their feedback and
    that it will have an effect on the implementation of the standards.

-   **Document feedback.** Accurate and complete documentation of all
    questions, concerns, and recommendations made by participants during
    a review is critical to support the decisions for revising and
    adopting the standards. Record the composition of each review group,
    the process used during the review, the name of the person who
    developed or facilitated each review, and the detailed feedback
    provided. ![](media/image1.wmf){width="1.8395833333333333in"
    height="0.6638888888888889in"}

-   **Consider developing and using standardized feedback forms to
    gather information.** Although forms should be specific to the type
    and purpose of each review, there may be some common features (e.g.,
    demographic questions, format) that can be used on all forms across
    reviews.

-   **Compile and analyze feedback.** Analyze feedback across reviews by
    looking for trends and recurrent themes and at ratings for different
    items. Summarize the findings in a way that can be used easily by
    the coordinating committee and writing team to make revisions to the
    draft standards and implementation procedures. For example, it is
    helpful to capture the number and types of participants (e.g.,
    experienced administrators or English as a Second Language
    instructors) that correspond to different ratings and themes that
    emerge. Compile and organize actual comments made during the review
    to help guide the revision.

The manager of the review process should keep these common steps in mind
as decisions are made on the types and numbers of reviews to conduct.
The following sections provide an overview of four
strategies---statewide surveys, focus groups, expert reviews, and field
tests---that the state may use to finalize the content standards.

## Statewide Surveys

A statewide survey is a cost-effective approach to soliciting input from
as many stakeholders as possible in a short amount of time. Consider
recruiting respondents through an open invitation and administering the
survey via the Internet to reach a broad audience. Surveys are commonly
used in reviews for content validity and for implementation.

The review coordinator should work with measurement and evaluation
experts to design a survey that solicits the information necessary to
validate the standards. Additionally, these measurement experts will
analyze and help interpret the findings. If technical knowledge is not
available among state staff, consider contracting with an external
agency for this work.

Conducting a statewide survey is a specialized process based on
overlapping steps. The basic steps are as follows:

1\. Design the survey and data collection system.

2\. Produce survey materials.

3\. Field test the survey with members of the writing team.

4\. Advertise and distribute the survey.

5\. Maintain the survey (if Web-based).

6\. Collect, clean, and analyze data.

7\. Compile and report results.

To simplify their task, provide survey respondents with a scale that
they can use to make judgments about the draft content standards. For
example, in the case of a validity review, respondents might rate the
standards for content using the following scale:

-   **Essential---**Without competence in this area, learning of the
    content would be hindered.

-   **Very important---**Competence in this area contributes
    substantially to the overall learning of the content.

-   **Somewhat important---**Competence in this area contributes
    slightly to the overall learning of the content.

-   **Not needed---**Competence in this area has no effect on the
    overall learning of the content or represents knowledge, skills, or
    abilities that can be learned outside of the classroom.

As described previously, a validity review for content should focus on
the level of most specificity in the standards.

## Focus Groups![](media/image3.wmf){width="1.8451388888888889in" height="0.5722222222222222in"}

Convening a focus group, or discussion group, will provide richer
feedback on the draft standards than can be gained through a statewide
survey. In addition to answering posed questions, participants are
likely to raise questions and issues that were not considered
previously. Given that this strategy will produce more detailed feedback
than a statewide survey, it can be used effectively as a follow-up to
survey responses. This combination of statewide survey and focus group
is particularly useful in capturing the full range of criteria for
quality standards that is the target of a validity review for content.
Additionally, focus groups can provide feedback from representatives of
most, if not all, stakeholder groups.

In preparing to conduct focus group sessions, carefully consider the
number, size, length, and locations (state regions and specific sites)
of the sessions in terms of state budget and resources. It is important
to hold enough focus group sessions at convenient locations to ensure
statewide representation of stakeholders. Consider recruiting
participants through an application or nomination process. Depending on
the nature of the review, a focus group session may include eight to
twelve participants and take from three hours to a full day. Plan
appropriately for on-site costs (e.g., refreshments) and travel costs.

Determine how many focus groups will be held with a mix of stakeholders
(e.g., adult education practitioners, professional development
providers, and administrators) and how many with a single stakeholder
group (e.g., learners or instructors). A good mix of stakeholders at a
focus group session will produce richer feedback, as participants
discuss the issues and provide suggestions that take into account their
diverse perspectives. Alternatively, there may be times when limiting
stakeholder representation will actually provide information that is
more truthful. ![](media/image1.wmf){width="1.8395833333333333in"
height="0.6638888888888889in"}

Consider using members of the coordinating committee and writing team as
facilitators of focus groups. Be sure to train facilitators to their
task. The purpose of a focus group is to gather both positive and
negative feedback. Facilitators should not attempt to redirect or argue
with the participants of a focus group. The data collected at focus
group sessions are used to inform standards development and
implementation procedures.
![](media/image1.wmf){width="1.8395833333333333in"
height="0.6638888888888889in"}

Staff designated to coordinate the focus groups will want to have
protocols developed in advance that address the purposes for the review.
Appendix C, Tools and Templates, provides a sample format for developing
a focus group protocol. A protocol also will make it easier to organize
and analyze information gathered during the focus groups.

If the purpose of the focus group is to review documents, walk the
participants through all relevant documents, confirming their
understanding of these materials and addressing any questions or
concerns that may arise. If specific forms are used to gather
participant feedback on the standards, discuss one or two standards and
how to use the forms with the group so that participants understand the
type and level of feedback being sought.

To ensure that the discussions are accurately captured, it is useful to
have a scribe and audiotape recorder. Solicit from the group explicit
suggestions for changes to the draft standards whenever possible, and be
sure to carefully record these suggestions.

Following each focus group session, compile and analyze the feedback.
Once all sessions are complete, analyze the feedback across sessions.
Again, look for trends and recurrent themes, noting the number and types
of respondents. For example, the feedback may reveal issues that occur
only among experienced instructors or only in rural settings. This type
of information will be helpful to the writing team in the revision of
the draft content standards and establishment of implementation
procedures.

## Expert Reviews

An expert review is useful in alignment reviews and validity reviews for
content, bias, and measurability, as well as for reviewing draft content
standards for legislative or policy issues (e.g., across state
departments). Use experts to review a specific aspect of the draft
content standards. Expert reviews are valuable because experts are
current on the research and are generally familiar with expectations for
standards across the nation, so they can provide a broader perspective.
In addition, they are independent of the process and thus less likely to
be biased.![](media/image1.wmf){width="1.8395833333333333in"
height="0.6638888888888889in"}

Experts can be identified by state staff, the coordinating committee, or
the advisory committee. As reviewers, experts may not be representative
of all stakeholders, and the data collected from expert reviews may be
limited to a specific aspect of the standards. However, these reviews
provide useful information when paired with other approaches for
establishing validity. Before experts begin their reviews, share with
them the state's policies and procedures for developing standards and
guidelines for reviewing the standards. Guidelines will vary depending
on the nature of the review.

If the experts are asked to review materials, make certain these are
provided well in advance to allow them to prepare for any discussions or
raise any questions. Also consider asking the experts in advance of the
review whether there are any resources or supplementary materials that
they will need to conduct this work and make those materials available
to them.

Expert reviews may involve costs associated with consultant fees and
travel. If the state is considering convening a panel of experts, plan
appropriately for participant travel, refreshments, and a suitable site.
Consider having state staff who serve on the coordinating committee
facilitate an expert panel. Ask the panel to examine each element of the
standard (e.g., standard, indicator, objectives, supplementary
materials) and then the full set of standards. This two-step strategy is
important because some issues may not be apparent when examining
individual standards or elements of standards. Only when the draft is
considered as a whole do some issues become apparent.

After reviewing the standards, discuss the concerns raised by the
experts. Carefully document all questions, concerns, citations, and
recommendations experts make as they conduct their reviews. Consider
using a form to record recommendations, making sure that the experts are
explicit about any suggested changes and options for revision. Exhibits
4.2 and 4.3 provide sample feedback forms that experts can use, one form
for a bias review and another form for an alignment review.

The documentation from an expert review will be in the form of very
specific questions, concerns, and recommendations. Compile all feedback
so that the coordinating committee and writing team can easily
understand the feedback when they refine the draft content standards. In
summarizing the results of an expert review, be sure to note any areas
of dissention and by whom among the expert reviewers.

Exhibit 4.2. *Validity Review Feedback Form Focusing on Bias*

![](media/image4.wmf){width="7.083333333333333in"
height="2.3506944444444446in"}**This sample feedback form can be used
for conducting a validity review that focuses on bias.**

  ------ ---------- --------------- ------------------ ----------------- -----------------
  Item   Standard   Issue           Description        Committee         
                                                       Recommendations   

                                                       Suggested         Comments
                                                       Revision          

  1      4.7.a      Language use    "...man-made..."   Delete            If necessary,
                    (gender)                           "man-made."       change to
                                                                         "human-made."

  2      8.12.d     Accessibility   Small print on a   Provide large     Develop
                    (disability)    computer screen    print.            alternative
                                                                         formats.

  3                                                                      
  ------ ---------- --------------- ------------------ ----------------- -----------------

![](media/image4.wmf){width="7.083333333333333in"
height="3.3819444444444446in"}Exhibit 4.3. *Alignment Review Feedback
Form*

**This sample feedback form can be used for an alignment review.**

  ----------------- --------------- ---- ----------- ----------------- ----------------------------------------------------
  Standard          Alignment                        Committee         
                                                     Recommendations   

                                                     Suggested         Comments
                                                     Revision          

  2.11: Learners    Congruent            K--12                         Learning sequence differs
  will check own                         Standards                     
  reading                                3.8, 3.9,                     
  comprehension                          4.6, 5.3,                     
                                         6.7                           

                    Aligned                                           

                    Contradictory                                      

  Recommendation:   Delete                           Retain as is      ![](media/image5.wmf){width="1.0138888888888888in"
                                                                       height="0.2638888888888889in"}Revise

  2.12: Learners    Congruent                                          Not measurable
  will become                                                          
  lifelong readers                                                     

                    Aligned                                            

                    Contradictory                                     

  Recommendation:   Delete                           Retain as is      Revise
  ----------------- --------------- ---- ----------- ----------------- ----------------------------------------------------

## Field Tests

The name---field test---says it all: to test under natural operating
conditions. A field test is a small-scale trial, assessing how well the
standards can be implemented in educational settings. It provides
information necessary for the development and implementation of
statewide standards-based education. Through classroom trials and
discussions about standards-based curriculum and instruction, state
teams learn![](media/image1.wmf){width="1.8395833333333333in"
height="0.6638888888888889in"}

-   how well the standards meet the needs of teachers and learners,

-   what procedures and processes need to be in place to implement the
    standards effectively, and

-   how to revise and finalize the draft standards.

Revisions to the draft standards may include making adjustments related
to levels and indicators. Additionally, a field test can be used to help
the state determine the kinds of professional development and technical
assistance that are needed for implementation.

The coordinating committee can designate a field test manager to oversee
all stages of a field test. Ideally, the manager is familiar with
multiple methods for collecting data at the program level and with
strategies to support field test participants in their tasks. The
manager oversees the process and provides the support necessary for an
effective field test.

Field testing is a multifaceted process. Therefore, it is important that
the field test manager, with input from members of the coordinating
committee, develop a plan that addresses all stages of the field test
and is feasible given state and local resources. The plan, submitted to
the state director for approval, might address the following components:

-   **recruitment of participant programs,** including criteria for
    selecting sites

-   **orientation and professional development,** including strategies
    for supporting staff participation and topics for professional
    development

-   **data collection,** including questions to address and activities
    and timeframes for collecting data

-   **documentation and analysis of feedback,** including strategies for
    organizing and using feedback

Each component of the plan should identify individuals who will be
responsible for carrying out the activities. Therefore, it is important
to identify the roles and responsibilities of field test participants,
the field test manager, and other state staff from the outset.

The plan should also provide an estimated timeline for conducting the
field test. Allocate time for recruiting participant programs, orienting
participants to the standards and their roles in the field test,
training participants on using data collection tools, collecting data,
and analyzing and reporting field test findings. Depending on state and
local resources, three to six months is sufficient time to collect
preliminary data; conducting a complete field test may require a year or
more.

The Practice and Application section at the end of this chapter includes
an exercise to help the state plan for a field test.

### Recruit Participant Programs![](media/image6.wmf){width="1.8451388888888889in" height="0.6673611111111111in"}

The first step in a field test is to establish criteria for selecting
programs to participate. The coordinating committee can play a key role
in identifying the selection criteria. The committee may seek programs
based on (1) experiences with new initiatives, (2) size, (3) populations
served, (4) geographical location, or (5) delivery provider (e.g.,
community college, adult school, community-based organization). After
the criteria have been developed, the state has several options for
recruitment. It can recruit programs by recommendation and/or by
invitation based on the recruitment criteria. Whichever option the state
chooses, it is important to let participating sites know the time
investment that will be required and the resources they will need to
commit for the field test.

### Orient and Provide Ongoing Support and Professional Development

Once programs agree to participate in the field test, it is important to
orient all participants to their tasks. Orientation may include

-   presenting the state's goals for standards-based
    education,![](media/image1.wmf){width="1.8395833333333333in"
    height="0.6638888888888889in"}

-   reviewing the draft content standards and discussing how they are
    similar to and different from current practice,

-   reviewing the purposes of the field test and the types of data that
    will be collected,

-   defining participant roles in the field test, and

-   explaining how the findings from the field test will be used to
    inform the draft standards and the implementation of a
    standards-based system.

Field test participants may need professional development and ongoing
support to carry out their roles in the review process effectively.
Training should be tailored to each type of participant (e.g.,
instructor, administrator, curriculum specialist) and address
task-specific issues and processes. For example, during a field test,
instructors may need help in learning how to develop local curricula,
instructional units, and strategies to monitor learner progress that are
based on the standards.

Program administrators may also need ongoing support to identify and put
into practice local policies and procedures that will support
implementation of standards-based education. For example, program
administrators may want to set up a mentoring or coaching system to
assist instructors in implementing the standards, and they may need
support and assistance in setting up these systems.

To support participants, the field test manager should consider
convening participant program staff, locally or by region, to help build
a learning community and to encourage participants to collaborate,
problem solve, and share experiences. The dialogue established during
these meetings can be continued via e-mail, an electronic mailing list,
telephone, or other distance communication. The manager may also want to
conduct debriefing meetings at the program site on a regular basis to
identify challenges and successes in implementing the standards. These
meetings are another way to problem solve and support participants
during the field test.

### Collect Field Test Data

One of the purposes of a field test is to help the state determine how
well local programs can implement the standards and to identify the
kinds of policies, procedures, resources, professional development, and
technical assistance necessary for local programs to implement
standards-based education effectively. To target these issues, collect
data purposefully. Begin by identifying field test questions that
address implementation issues. Consider framing the feedback with the
following questions:

-   How do instructors translate standards into curriculum and
    instruction?

-   How do instructors monitor student progress in meeting the
    standards?

-   What resources and additional professional development would be
    helpful to instructors and administrators in implementing
    standards-based education?

For the purposes of validating and revising the draft content standards,
develop a set of field test questions for the instructors to consider as
they use the standards in the classroom. Consider the following
questions:

-   What aspects of the standards are easy or difficult to use?

-   What language, if any, in the standards is not clear?

-   What in the standards should be changed? Deleted? Added? Moved to
    another level?

Using simple feedback forms and other tools (e.g., observation and
implementation checklists, lesson templates, and reflection logs) will
make data collection easy for participants. Forms that are unstructured,
too complicated, or too long will not be effective, if they are used at
all.

Provide clear instructions and train participants on how to use the data
collection tools, including who should complete them and when. For
example, some forms may be useful when completed after each lesson, unit
of instruction, or administration of an assessment. Additionally, the
field test manager can provide programs with a timeline for their data
collection efforts that indicates when programs should submit feedback
to the manager. A timeline will encourage ongoing data collection and
help participants plan for the integration of data collection in their
practice.

Exhibits 4.4 and 4.5 are sample forms that instructors can complete to
provide feedback on their classroom implementation of the standards.
Exhibit 4.4---Sample Feedback Form on the Applicability of Standards to
Instruction---will help determine the applicability of the standards and
indicators to the specific levels of instruction, for example, beginning
ABE or low intermediate ABE. Exhibit 4.5---Sample Instructor Feedback
Form on Applicable Classroom Activities---can help demonstrate how the
standards are addressed by classroom activities, such as resource
materials, assignments, and classroom assessments. Appendix C, Tools and
Templates, has two sample templates that can be adapted for gathering
instructor feedback---the Kentucky Content Standards Review Survey and
the Standards Usability Checklist Instructor Feedback Form. Such
documentation will be helpful not only for the field test but also for
helping instructors to see changes in their classroom practices.

Exhibit 4.4. *Sample Feedback Form on the Applicability of Standards to
Instruction*

This type of feedback form can help determine the applicability of the
standards or indicators to specific levels of instruction (e.g.,
beginning ABE, low intermediate ABE).

> Review the indicators that are appropriate to the instructional level.
> For each indicator, rate its applicability to the level being taught
> by using the following scale: applicable, predominantly applicable,
> slightly applicable, or not applicable. If unsure of a rating, mark
> "Do not know." Use the comments box to provide additional information
> or suggestions on the application of the indicator to classroom
> instruction (e.g., move to different level of instruction or define
> terms).

+----------------------+-----------------------------------------------+
| Indicator R.1.a: Use |                                               |
| sound symbol         |                                               |
| correspondences to   |                                               |
| decode new words.    |                                               |
+----------------------+-----------------------------------------------+
| Rating               | Comments                                      |
+----------------------+-----------------------------------------------+
| □ Applicable         |                                               |
|                      |                                               |
| □ Predominantly      |                                               |
| applicable           |                                               |
|                      |                                               |
| □ Slightly           |                                               |
| applicable           |                                               |
|                      |                                               |
| □ Not applicable     |                                               |
|                      |                                               |
| □ Do not know        |                                               |
+----------------------+-----------------------------------------------+
| Indicator R.1.b:     |                                               |
| Locate words or      |                                               |
| items in             |                                               |
| alphabetical or      |                                               |
| numerical lists\     |                                               |
| (e.g., books,        |                                               |
| directories,         |                                               |
| indices).            |                                               |
+----------------------+-----------------------------------------------+
| Rating               | Comments                                      |
+----------------------+-----------------------------------------------+
| □ Applicable         |                                               |
|                      |                                               |
| □ Predominantly      |                                               |
| applicable           |                                               |
|                      |                                               |
| □ Slightly           |                                               |
| applicable           |                                               |
|                      |                                               |
| □ Not applicable     |                                               |
|                      |                                               |
| □ Do not know        |                                               |
+----------------------+-----------------------------------------------+

> ![](media/image7.wmf){width="6.829861111111111in"
> height="7.355555555555555in"}*Note:* Adapted from R. J. Marzano and J.
> S. Kendall. *Awash in a sea of standards* (Aurora, CO: Mid-continent
> Research for Education and Learning, 1998).
> <http://www.mcrel.org/topics/productDetail.asp?productID=120>
> (accessed December 2007).

Exhibit 4.5. *Sample Instructor Feedback Form on Applicable Classroom
Activities*

![](media/image4.wmf){width="6.385416666666667in" height="1.8125in"}Use
this form to record instruction and assessment activities for standards
and indicators.

+-------------------+-------------------+------------------------------+
| Standard          | Classroom         |                              |
|                   | A                 |                              |
|                   | ctivity/Resources |                              |
|                   | Addressing        |                              |
|                   | Standard\         |                              |
|                   | (e.g., materials, |                              |
|                   | assignment,       |                              |
|                   | assessment)       |                              |
+-------------------+-------------------+------------------------------+
| **2.7. Learners   |  Curriculum      | -   Reading workbook (pp.    |
| use decoding      |                   |     16--20)                  |
| skills to read.** |  Assessment      |                              |
|                   |                   | -   Learning activities to   |
|                   |                   |     find explicitly taught   |
|                   |                   |     sounds used in language  |
|                   |                   |     experience story         |
|                   |                   |                              |
|                   |                   | -   Instructor observation   |
+-------------------+-------------------+------------------------------+

Another effective strategy for data collection is to conduct a focus
group with field test participants. Moreover, interviews or focus groups
with a sampling of participants can provide more in-depth feedback and
expand on the findings derived from the forms. To conduct interviews or
focus groups, identify the major themes in the data from feedback forms
and structure the sessions around those themes. Interview and focus
group sessions lend themselves to raising new issues that may not have
been identified through the feedback forms. Be sure to capture and
explore any new issues raised during an interview or focus group. This
information may also be useful in adapting and revising the standards
and for planning professional development.

### Document and Analyze Feedback

Documenting findings and processes during the field test is critical, as
it supports the decisions on how to revise the draft content standards
for final adoption by the state. Documentation also helps to clarify
procedures and orientation information and to identify professional
development needs for future implementation of the standards.

Just as data collection and documentation are ongoing processes,
analyzing the field test feedback also should be an ongoing process.
Continually analyzing feedback allows the field test manager to make
midcourse corrections or changes to the implementation procedures, as
necessary or possible.

The field test manager, or designee, is responsible for analyzing
participant feedback as it is collected, noting trends and recurrent
issues across participant programs. The size of the trial may determine
how many people are needed to do the analysis. It may be too large a
task for one person, in which case a group approach might be used.
Analyses should be guided by the inquiries made during the field test
(e.g., questions asked during interviews or ratings of standards on
feedback forms).

Organize feedback by the central purposes of the field test:
(1) validating draft content standards and (2) identifying policies,
procedures, and professional development for implementing standards. Use
the feedback to answer the following overarching questions:

-   What do the data reveal about how well the content standards
    translate into curriculum? Into instruction? Into assessment?

-   What training and resources will be helpful for statewide
    implementation of the content standards?

-   What local administrative policies and procedures support
    implementation of standards-based education?

To understand all the issues, combine different types of data (e.g.,
open-ended responses, ratings based on a scale, yes/no responses) and
data from a variety of sources (e.g., feedback forms, interviews,
observations). Some data may produce more questions than answers. Follow
up with participating programs, as needed, to help clarify unexplained
results.![](media/image1.wmf){width="1.8395833333333333in"
height="0.6638888888888889in"}

Review the data collection forms to find suggested revisions for the
standards (e.g., content, format, presentation). For example, it may
become apparent during data analysis that participants are responding to
some aspect of the content standards that had been overlooked or
dismissed as unimportant by the writing team. Through classroom
observations or responses on feedback forms, it may become evident that
programs are consistently misinterpreting or inappropriately applying
particular standards or indicators. Such cases may require the writing
team to clarify the standards. To facilitate this work, it is helpful
for the field test manager to make recommendations to the writing team
on how to revise the standards to avoid similar problems during
statewide implementation.

Analysis can also indicate ways to improve the implementation of
standards-based education, especially if participant programs try
different approaches to implementing the standards. For example, some
programs may decide to use program administrators or instructional
specialists to mentor staff in implementing the standards. Other
programs may decide to institute biweekly meetings to discuss
implementation with staff members. And still other programs may develop
an online resource bank of standards-related materials. By comparing
data from programs using different approaches, the analysis may show
that particular implementation strategies are more effective than others
for certain types of programs.

# Use Feedback from Reviews to Finalize Content Standards

The coordinator(s) of external reviews will need to compile the results
of each type of review and then share the information with the writing
team and coordinating committee. The writing team takes the lead on
revising the standards based on the survey results, the recommendations
of the review committees, and the field test feedback. This is the final
stage in which to infuse the standards with the consensus of the field.
Team members may not agree with all feedback, but should be encouraged
to appropriately act on behalf of the field in finalizing the content
standards and document their decisions for these actions.

It is helpful to provide the writing team with guidelines on how to
finalize the draft standards. The coordinating committee is responsible
for providing these guidelines, which give the writing team three
options:

-   delete the standard,

-   revise the standard, or

-   retain the standard as is.

For example, given a survey, the coordinating committee may choose to
set a minimum rating of importance that must be met by each standard.
Using the sample survey scale in this chapter (see Statewide Surveys
section), the minimum rating of importance might be set at "very
important." Any standard receiving an average rating of "somewhat
important" or "not needed" would be flagged for action by the writing
team. The team's course of action (delete, revise, or retain the
standard) is determined by the guidelines. For example, if alignment to
the Adult Diploma Program (ADP) standards has been mandated by the
state, a guideline would be to retain standards that are directly
aligned with the ADP standards. If the flagged standard is aligned with
the ADP standards and its deletion would bring the set of standards out
of alignment, then it should be revised or retained as is. If the
flagged standard is not directly aligned with the ADP standards, the
team can delete it.

Clear, specific guidelines will ensure that all team members follow
common procedures and that state policies are preserved. They will also
keep the team from revisiting decisions and issues from previous
meetings.![](media/image1.wmf){width="1.8395833333333333in"
height="0.6638888888888889in"}

The facilitators of each review should be present to discuss the review
processes used and the relevant feedback with the writing team. It is
important that the writing team members understand that feedback from
the review phase offers suggestions only, and their role is to make
final recommendations to the coordinating committee. Suggested revisions
from a bias review, however, can have legal consequences and should be
seriously considered.

There are two basic steps the writing team needs to take: review the
feedback from each review and document all actions taken in response to
that feedback.

**Step 1: Review the feedback.** Present feedback from each review
separately, to focus the writing team on one issue at a time. Each
review is likely to produce a considerable amount of feedback. Provide
the team with all feedback (demographic, quantitative, and qualitative
data), but focus activities on feedback that requires action.

**Step 2: Document the decisions.** It is useful to have a state staff
member from the coordinating committee, as well as a member of the
writing team, record all team decisions. Documentation should include
the team's actions and support (e.g., citation from document with which
standards must align) for each action.

## State Review and Final Revisions

Upon completion of the draft standards, the coordinating committee
presents the standards to the appropriate state agency for review, final
revisions, and adoption. It may be necessary for members of the various
review committees involved in the development process to serve as expert
witnesses, testifying to the state on the work that they did and their
support of the content standards.

If the coordinating committee has provided the state with regular
progress reports on the development of content standards, there should
not be any surprises in this last stage of review and adoption. Be sure
to address all state questions and concerns with the documentation from
the planning, development, and review phases. Additionally, the state
may require impact analyses in deciding to adopt the content standards.

Submit the draft content standards to the state with an implementation
plan. The coordinating committee should develop this plan by addressing
four key elements for statewide implementation of a standards-based
education system: (1) communication of the goals and policies for
standards-based education, (2) professional development and technical
assistance, (3) monitoring of local implementation of standards, and
(4) evaluation of the initiative. Chapter 5 provides more information on
state implementation.

## Periodic Review and Revision of Content Standards

One challenge confronting standards-based education reform efforts is
establishing a schedule for revision. It may seem premature to plan for
changing newly minted content standards, but periodic reviews and
revisions will be needed to be responsive to policy and knowledge
advances in adult education. The difficulty in revisiting standards
development lies not only with the state and local resources involved
(i.e., human and financial resources), but also with the time and
planning required. Revision timelines must anticipate the significant
time it takes for reform to trickle through the system.

# Practice and Application

1\. A validity review ensures that the standards represent the
knowledge, skills, and abilities most valued for adult learners to
succeed as workers, citizens, family members, and lifelong learners. One
way to conduct a validity review for content is to gather feedback
through a focus group of stakeholders. Develop a plan for this type of
review, proposing which type(s) of stakeholders to include and a set of
questions that can be used to solicit feedback.

2\. Plan for field testing of the standards. Identify potential programs
in the state that may participate. Consider types of programs (e.g.,
family or workplace literacy), types of agencies (e.g., community-based
organization, adult school), the characteristics of the program (e.g.,
satellite site or full-service site with support systems), and the types
of adult learners served (e.g., students who have disabilities,
immigrants, incarcerated students).

> a\. What other considerations for program selection will the state
> take into account?
>
> b\. What incentives will be provided for participation?
>
> c\. Who will manage the field tests and coordinate data collection,
> analyses, and reporting?

# References

Marzano, R. J., and J. S. Kendall. 1998. *Awash in a sea of standards*.
Aurora, CO: Mid-continent Research for Education and Learning.
[http://www.mcrel.org/topics/\
products/120/](http://www.mcrel.org/topics/products/120/) (accessed
December 2007).

McNamara, C. n.d. *Basic guide to program evaluation*. St. Paul, MN:
Management Assistance Program for Nonprofits.
<http://www.mapnp.org/library/evaluatn/fnl_eval.htm> (accessed December
2007).

Morgan, D. L., and R. A. Krueger. 1997. *The Focus Group Kit.* 6 vols.
Thousand Oaks, CA: Sage Publications, Inc.

Webb, N. L. 1999. *Alignment of science and mathematics standards and
assessments in four states* (NISE Research Monograph No. 18).
Washington, DC: Council of Chief State School Officers; Madison, WI:
National Institute for Science Education, University of
Wisconsin--Madison.
<http://www.ccsso.org/content/pdfs/AlignmentPaper.pdf> (accessed
December 2007).