![](media/image1.png){width="1.2395833333333333in"
height="0.3506944444444444in"}

**TABLE OF CONTENTS**

> **Page**

#### I. Introduction and Methodology 3

A. Overview of ACSI Modeling 3

B. Segment Choice 3

C. Customer Sample and Data Collection 4

D. Questionnaire and Reporting 4

**II. Survey Results 5**

A. Customer Satisfaction (ACSI) 6

B. Customer Satisfaction Model 8

C. Drivers of Customer Satisfaction 10

*Technology 10*

*Documents 13*

*ED Staff/Coordination 16*

*Online Resources 19*

*ED-funded Technical Assistance 22*

D. Satisfaction Benchmarks 25

E. Complaints 26

#### III. Summary and Recommendations 27

[]{.mark}

#### Appendices

A. Questionnaire 31

B. Non-Scored Responses 47

####  C. Attribute Tables 51

D. Verbatim Responses by Program 75

E. Explanation of Significant Difference Scores 111

**Chapter I**

**Introduction & Methodology**

The American Customer Satisfaction Index (ACSI) is the national
indicator of customer evaluations of the quality of goods and services
available to U.S. residents. It is the only uniform,
cross-industry/government measure of customer satisfaction. Since 1994,
the ACSI has measured satisfaction, its causes and effects, for seven
economic sectors, 41 industries, more than 200 private sector companies,
two types of local government services, the U.S. Postal Service, and the
Internal Revenue Service. ACSI has measured more than 100 programs of
federal government agencies since 1999. This allows benchmarking between
the public and private sectors and provides information unique to each
agency on how activities that interface with the public affect the
satisfaction of customers. The effects of satisfaction are estimated, in
turn, on specific objectives, such as public trust. 

The ACSI is produced through a partnership of the University of Michigan
Business School, CFI Group, and the American Society for Quality.

######## Segment Choice 

A total of 10 groups, composed of eight program offices, ED*Facts*
Coordinators, and Chief State School Officers, participated in the 2008
U.S. Department of Education Grantee Satisfaction Survey. All 10 groups
had also participated in the 2006 and 2007 studies. The chart below
indicates the composition of survey respondents by program groups as a
percentage of all respondents.

#### ![](media/image4.wmf){width="5.825in" height="3.747916666666667in"}  {#section .unnumbered}

#### C. Customer Sample and Data Collection

The same programs that participated in 2006 and 2007 were included in
the 2008 Grantee Survey. Each program provided a list of Directors from
their program. Chief State School Officers were also included. ED
provided a total of 570 e-mail contacts. Data were collected from April
15, 2008 through June 26, 2008. Data collection was conducted primarily
by e-mail. In order to increase response reminder e-mails were sent to
non-responders and phone calls were also placed to non-responders where
respondents were given the option to complete the survey via phone. A
total of 362 responded to the invitation for a 63.5% response rate.
Thirty-five respondents indicated that they were not affiliated with one
of the participating program offices within the last 12 months, and were
therefore disqualified. Of those who responded and were qualified, 322
respondents provided valid responses. These are responses where at least
two-thirds of the questions were answered.

Response rates for each participating program for 2007 and 2008 are
provided below. For most of the programs response rates dipped slightly
from last year. However, as was the case last year all but two programs
had response rates above 50%.

#### 

#### 

#### 

#### ![](media/image5.wmf){width="5.75in" height="2.1277777777777778in"}

#### 

#### 

#### 

#### 

#### 

#### 

#### 

#### 

#### 

#### 

#### 

#### 

#### D. Questionnaire and Reporting

The questionnaire used is shown in Appendix A. The core set of questions
was developed in 2005 and has remained unchanged in each subsequent
administration of the survey. Each program had the opportunity to
include a set of questions specific to their program. Some programs
chose to add or modify their custom questions in 2008. Changes to the
questionnaire are noted and can be found with the questionnaire in
Appendix A.

[]{.mark}

Most of the questions in the survey asked the respondent to rate items
on a 1 to 10 scale. However, open-ended questions were also included
within the core set of questions, as well as open-ended questions
designed to be program-specific. Appendix C contains tables that show
scores for each question reported on a 0 to 100 scale. Results are shown
in aggregate and by program. All verbatim responses are included in the
back of the report in Appendix D, Verbatim Comments. Comments are
separated by program.

**Chapter II**

**Survey Results**

A. Customer Satisfaction (ACSI)

The **Customer Satisfaction Index** **(CSI)** is a weighted average of
three questions: Q30, Q31, and Q32, in the questionnaire in Appendix A.
The questions are answered on a 1 to 10 scale and are converted to a 0
to 100 scale for reporting purposes. The three questions measure:
Overall satisfaction (Q30); Satisfaction compared to expectations (Q31);
and Satisfaction compared to an 'ideal' organization (Q32).

**The 2008 Customer Satisfaction Index (CSI) for the Department of
Education grantees is 65.** Satisfaction with ED is up two points from
last year to reach its highest level since the measure with ED began in
2005. When considering the three questions separately, overall
satisfaction with ED's products and services reached a score of 70. ED
meeting expectations scored 63 and compared to the ideal was rated 59.

![](media/image6.wmf){width="7.429861111111111in" height="4.35625in"}

The chart below compares the satisfaction score of the U.S. Department
of Education with satisfaction scores from other federal grant awarding
agencies taken over the past three years and the most recent (December
2007) annual overall federal government average for benchmarking
purposes. The U.S. Department of Education's score is on the lower end
of federal grantee satisfaction scores. ED is now only three points
below the current federal government average.

![](media/image7.wmf){width="6.45in" height="4.419444444444444in"}

Satisfaction was up two points at the aggregate level. With respect to
program-level scores there has been some changes from last year as well.
The chart below reflects the grantees' 2008 Customer Satisfaction Index
with the Department by program and compares current scores with those
from 2007. As was the case in 2007, in 2008 three programs had
statistically significant changes in their satisfaction with the
Department among their Directors. The three programs are noted below
with asterisks, EDEN/ED*Facts* Coordinators, Chief State School
Officers, and Title III State Directors. None of the other changes shown
below, either gains or drops, were statistically significant at a 90%
level of confidence.

#### ![Bar chart: Customer Satisfaction Index Scores by Program 2007-2008](media/image8.wmf){width="7.75984251968504in" height="4.880708661417323in"}  {#bar-chart-customer-satisfaction-index-scores-by-program-2007-2008 .unnumbered}

#### B. Customer Satisfaction Model

The government agency ACSI model is a variation of the model used to
measure private sector companies. Both were developed at the National
Quality Research Center of the University of Michigan Business School.
Whereas the model for private sector, profit-making companies measures
Customer Loyalty as the principal outcome of satisfaction (measured by
questions on repurchase intention and price tolerance), each government
agency defines the outcomes most important to it for the customer
segment measured. Each agency also identifies the principal activities
that interface with its customers. The model provides predictions of the
impact of these activities on customer satisfaction.

[]{.mark}

The U.S. Department of Education Grantee Customer Satisfaction model --
illustrated below, should be viewed as a cause-and-effect model that
moves from left to right, with satisfaction (ACSI) in the middle. The
rectangles are multi-variable components that are measured by survey
questions. The numbers in the upper right corners of the rectangles
represent performance or attribute scores on a 0 to 100 scale. The
numbers in the lower right corners represent the strength of the effect
of the component on the left on the one to which the arrow points on the
right. These values represent \"impacts". The larger the impact value,
the more effect the component on the left has on the one on the right.
The meanings of the numbers shown in the model are the topic of the rest
of this chapter.

![](media/image9.wmf){width="7.75in" height="4.4743055555555555in"}

Attribute scores are the mean (average) respondent scores to each
individual question in the survey. Respondents are asked to rate each
item on a 1 to 10 scale, with "1" being "poor" and "10" being
"excellent." For reporting purposes, CFI Group converts the mean
responses to these items to a 0 to 100 scale. It is important to note
that these scores are averages and not percentages. The score should be
thought of as an index in which "0" represents "poor" and "100"
represents "excellent."

A component score is the weighted average of the individual attribute
ratings given by each respondent to the questions presented in the
survey. A score is a relative measure of performance for a component, as
given for a particular set of respondents. In the model illustrated on
the previous page Clarity, Organization, Sufficiency of detail,
Relevance, and Comprehensiveness are combined to create the component
score for "Documents."

Impacts should be read as the effect on the subsequent component if the
initial driver (component) were to be improved or decreased by five
points. For example, if the score for "Documents" increased by 5 points
(73 to 78), the Customer Satisfaction Index would increase by the amount
of its impact, 1.4 points, (from 65 to 66.4). *(Note: Scores shown are
reported to nearest whole number).* Similarly, if the Customer
Satisfaction Index were to increase by 5 points, "Complaints" would
decrease by 0.7%. If the driver increases by less than or more than five
points, the resulting change in the subsequent component would be the
corresponding fraction of the original impact. Impacts are additive.
Thus, if multiple areas were each to improve by 5 points, the related
improvement in satisfaction will be the sum of the impacts.

**C. Drivers of Customer Satisfaction**

**Technology**

######### Impact 1.2

Technology continues to have a high impact on grantee satisfaction with
an impact of 1.2 on satisfaction. The area of technology is up two
points from 2007 with a score of 67. This represents a statistically
significant increase from last year. The U.S. Department of Education's
effectiveness in using technology to deliver its services remains the
highest rated item in the area of technology at a rating of 72. The
Department's automated process to share accountability information,
effectiveness in improving state's reporting, and expected reduction in
federal paperwork all had significant gains from last year.

![1. Bar chart: US Department of Education Technology Aggregate Scores
2007-2008](media/image10.wmf){width="7.109842519685039in"
height="3.7456692913385825in"}

Respondents who rated "ED's effectiveness in using technology to deliver
services" low (below "6") were asked how the U.S. Department of
Education could better use technology to deliver its services. As was
the case in previous years, many respondents mentioned increasing the
use of conference calls and WebEx in order to promote better
communication without the need for travel. Podcasting was also mentioned
as a possible way to provide information to grantees. All verbatim
responses can be found in Appendix D.

While at an aggregate level, grantees' evaluation of Technology
increased two points, when considering Technology scores for the U.S.
Department of Education grouped by program, there are a few programs
with more sizable and significant changes from last year. Three programs
had a significant increase in their rating of Technology this year,
Adult Education and Literacy, Chief State School Officers, and Title
III. None of the programs rated Technology significantly lower in 2008.

![2. Bar chart: Technology Scores by Program
2007-2008](media/image11.wmf){width="7.99015748031496in"
height="4.880708661417323in"}

While ED's effectiveness in using technology to deliver its services was
the highest rated item overall in the area of Technology, the quality of
assistance from the automated process to share accountability
information was the highest rated item by four programs, State
Educational Technology Directors, State Title V, Part A Directors,
EDEN/ED*Facts* Coordinators, and Directors of Adult Literacy. So for
many programs this too is perceived as a strength for the Department.
Expected reduction in federal paperwork was the item that had the
greatest range of ratings. Chief State School Officers felt most
positively about the paperwork reduction (75). However, Lead Agency
Early Intervention Coordinators only rated this item 39 and four other
programs rated expected reduction in federal paperwork in the 50s.

![](media/image12.wmf){width="7.309722222222222in"
height="2.5506944444444444in"}

**Documents**

***Impact 1.4***

Documents continues to be a key satisfaction driver with an impact of
1.4. Performance in the area of Documents saw a sizeable and significant
four-point increase. All of the items in this area had statistically
significant increases over their 2007 ratings. Comprehensiveness in
addressing the scope of issues that you face and sufficiency of detail
to meet your program needs each improved by six points from last year.

![1. Bar chart: US Department of Education Documents Aggregate Scores
2007-2008](media/image13.wmf){width="7.109842519685039in"
height="3.7133858267716535in"}

The aggregate increase in the rating of Documents is due to a broad
increase in this area across most programs as a majority of the programs
(six) had significant increases over last year's ratings. No program has
a statistically significant decrease, and only two of the programs rated
Documents less than 70.

![2. Bar chart: Documents Scores by Program
2007-2008](media/image14.wmf){width="8.009842519685039in"
height="4.518110236220473in"}

Across most of the programs Documents received strong ratings for their
relevance to the grantees needs and for their organization of
information. Clarity and detail of the documents received solid ratings
from most programs, although Chief State School Officers and Title III
State Directors rated both areas in the 60s. Programs were less uniform
in their ratings of the Documents' comprehensiveness in addressing the
scope of issues that they face. Programs that rated Documents the
highest, such as State Title V, Part A Directors and Directors of Adult
Education and Literacy, gave strong ratings to this item. Conversely,
those programs giving lower scores to Documents tended to rate this item
lower. State Title III Directors rated comprehensiveness 59. State
Directors of Special Education and Lead Agency Early Intervention
Coordinators rated it 64 and 65, respectively.

![](media/image15.wmf){width="7.320138888888889in" height="2.69375in"}

**ED Staff/Coordination**

##### Impact 0.9

ED Staff/Coordination remains one of the higher-performing areas for the
U.S. Department of Education and has improved by three points since last
year. Its impact of 0.9 means that further improvements in this area
will yield a modest increase in grantee satisfaction with the
Department. All items in the area of Staff/Coordination had a
statistically significant improvement over last year. Knowledge of
relevant legislation, regulations, policies, and procedures improved by
four points to a rating of 85. This is the highest rated item in the
entire survey. Other ED Staff/Coordination items realized improvements
of two to four points at the aggregate level.

![1. Bar chart: US Department of Education ED Staff/Coordination
Aggregate Scores
2007-2008](media/image16.wmf){width="7.75984251968504in"
height="4.539763779527559in"}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

While overall ED Staff/Coordination had a three-point improvement from
last year, three of the programs, Title I, Title III, and Chief State
School Officers rated this area significantly higher in 2008. Only one
program, Special Education rated ED Staff/Coordination significantly
lower in 2008. None of the other programs saw significant changes from
last year in the area of ED Staff/Coordination.

![2. Bar chart: ED Staff/Coordination Scores by Program
2007-2008](media/image17.wmf){width="8.009842519685039in"
height="4.518110236220473in"}

Across all programs, grantees are finding the Department's Staff to be
knowledgeable of relevant legislation, regulations, policies, and
procedures. Ratings for this item range from a low of 80 to a high of
90. Even State Directors of Special Education, who only rated ED
Staff/Coordination 65 overall, rated knowledge 80. Accuracy of responses
also yields high scores across nearly all programs. The areas of
consistency of responses with ED Staff from different program offices
and collaboration with other ED offices in providing relevant services
were rated high by several programs including State Title V, Part A
Directors, Lead Agency Early Intervention Coordinators, Directors of
Adult Education and Literacy, and State Educational Technology
Directors. Chief State School Officers, Title III State Directors, and
State Directors of Special Education provided much lower ratings to
consistency and collaboration.

![](media/image18.wmf){width="7.309722222222222in"
height="2.828472222222222in"}

**Online Resources**

######### Impact 0.9

Online Resources had a three-point improvement in its score compared to
last year. Ease of submitting information to ED via the Web received
higher ratings in 2008 with a rating of 74. In this year's customer
satisfaction model it was found that the impact that Online Resources
has on satisfaction is substantially greater than it was last year with
an impact of 0.9. This means that future improvements in the area of
Online Resources will yield a larger increase in customer satisfaction
that what would have been previously expected.

![1. Bar chart: US Department of Education Online Resources Aggregate
Scores 2007-2008](media/image19.wmf){width="6.45in"
height="4.419291338582677in"}

Four programs rated Online Resources significantly higher in 2008 than
they did in 2007. This includes Chief State School Officers,
EDEN/ED*Facts,* Adult Education and Literacy, and Title III. Conversely,
Career and Technical Education rated the Department significantly lower
on Online Resource in 2008.

![2. Bar chart: Online Resources Scores by Program
2007-2008](media/image20.wmf){width="8.009842519685039in"
height="4.518110236220473in"}

For most programs ease of submitting information to the Department via
the Web received positive ratings. Only two programs, State Educational
Technology Directors and State Title I Directors rated ease of
submitting information below 70. Ease of finding materials online was a
different matter. Only EDEN/ED*Facts* Coordinators rated ease of finding
materials as high as 70. State Title I Directors (47), Career and
Technical Education State Directors (54), and State Title V, Part A
Directors (57) found ease of finding materials online most problematic.

![](media/image21.wmf){width="7.05in" height="2.2104166666666667in"}

**ED-funded Technical Assistance**

######### Impact 0.3

ED-funded Technical Assistance remains the highest scoring area for the
U.S. Department of Education. This year's score reached 80 with a
three-point improvement over last year's score. Its relatively low
impact of 0.3 means that a further improvement in ED-funded Technical
Assistance will only yield a very modest increase in satisfaction. Five
of the seven items in the area of ED-funded Technical Assistance had
statistically significant gains from last year. ED-funded Technical
Assistance was found to be responsive to questions, providing accurate
responses, and knowledgeable of relevant legislation, regulations,
policies, and procedures. Collaboration with ED staff in providing
relevant services also received a strong rating (81).

![1. Bar chart: US Department of Education ED-funded Providers of
Technical Assistance Aggregate Scores
2007-2008](media/image22.wmf){width="7.75984251968504in"
height="4.1645669291338585in"}

Three programs rated ED-funded Providers of Technical Assistance
significantly higher than they did in 2007. This includes
EDEN/ED*Facts,* Lead Agency Early Intervention Coordinators, and Chief
State School Officers. No other program had a statistically significant
change in their rating of ED-funded Providers of Technical Assistance.

![2. Bar chart: ED-funded Providers of Technical Assistance Scores by
Program 2007-2008](media/image23.wmf){width="8.009842519685039in"
height="4.518110236220473in"}

As was the case in 2007, ED-funded Providers of Technical Assistance
received the strongest ratings for knowledge, responsiveness, and
accuracy of responses across most programs with responsiveness receiving
a rating of 92 from ED/ED*Facts* Coordinators and Lead Agency Early
Intervention Coordinators. Collaboration with ED staff in providing
relevant services and collaboration with other ED-funded providers of
technical assistance was solid across nearly all programs as well, with
EDEN/ED*Facts* Coordinators and Directors of Adult Education and
Literacy rating Collaboration with ED staff 90. Sufficiency of legal
guidance in the response continues to be the biggest issue for most
programs with six programs rating sufficiency of legal guidance in the
60s. However, for State Educational Technology Directors, State Title V,
Part A Directors, EDEN/ED*Facts* Coordinators and Directors of Adult
Education and Literacy guidance appears to be sufficient with ratings
above 80 from these programs.

![](media/image24.wmf){width="7.320138888888889in"
height="2.9715277777777778in"}

**D. Satisfaction Benchmarks**

The satisfaction benchmark question "Overall, when I think of all of
ED's products and services, I am satisfied with their quality" was
included in the 2008 survey. Respondents rated their satisfaction with
all of the U.S. Department of Education's products and services on a
4-point scale. In 2008, 83% responded 'Agree' or 'Strongly Agree.' The
percentage who answered 'Agree' or 'Strongly Agree' have remained
relatively consistent over the course of the past four years only
ranging from a low of 79% in 2006 to a high of 83% in both 2005 and
2008.

#### 

![](media/image25.wmf){width="6.840277777777778in"
height="4.748611111111111in"}

#### E. Complaints

Only 2% of all respondents reported that they had formally complained to
the U.S. Department of Education within the past six months. This is
down from 3% reported last year. Only four programs even had any
respondents who had formally complained in 2008. This is also a decrease
in number from last year as in 2007 six programs had respondents who had
complained. As was the case in 2007, Chief State School Officers were
most likely to complain. However, complaints for this group were down
from 21% to 11%.

![Bar chart: Complaints Percentage by Program
2007-2008](media/image26.wmf){width="7.75in"
height="4.880708661417323in"}

**Chapter III**

#### Summary and Recommendations

Grantee satisfaction with the Department improved by two points in 2008
to 65. This is only three points below the latest federal government
average (68). Three programs, Directors of Adult Education and Literacy,
State Title V, Part A Directors, and EDEN/ED*Facts* Coordinators, have
satisfaction with the Department above the federal government average.
While a couple of programs, State Title I Directors and State Directors
of Special Education rated satisfaction with the Department lower in
2008 than they did in 2007, these drops were not statistically
significant. Three programs did have significant increases in
satisfaction with the Department -- Chief State School Officers,
EDEN/ED*Facts*, and Title III. Across all programs, complaints fell from
3% to 2%.

[]{.mark}

All satisfaction driver areas saw significant increases from 2007.
Documents had the largest increase (four points). Grantees continue to
find the areas of ED Staff/Coordination and ED-funded Technical
Assistance as the biggest strengths of the Department. Both ED Staff and
ED-funded Technical Assistance were thought to be knowledgeable,
provided grantees accurate responses, and were responsive to their
questions. For ED Staff, the areas of consistency of responses with ED
staff from different program offices, and collaboration with other ED
offices in providing services, while still among the lower rated items
were rated significantly higher in 2008. With ED-funded Technical
Assistance the only area that is somewhat of an issue with grantees is
the sufficiency of legal guidance received in responses.

As was the case last year, Technology and Documents have the most impact
on grantee satisfaction. This does not necessarily mean that these areas
are the most import to grantees, but rather given the current level of
performance across all areas, improvements in Technology and Documents
will yield the largest increases in customer satisfaction.

Technology and Online Resources remain the lower rated driver areas.
With respect to the Department's Online Resources, grantees were rather
positive about the ease of submitting information to the Department via
the Web. What was more problematic was finding materials online.
Grantees continue to give solid ratings to the Department's use of
technology to deliver its services. Grantees' ratings also showed
improvements in the areas of working with states to develop an automated
process to share accountability information and effectiveness of the
automated process in improving state's reporting. While the expected
reduction in paperwork received higher ratings in 2008, it still remains
the lowest rated item overall for the Department.

[]{.mark}

Documents, the area with the highest impact on satisfaction, improved
along all its attributes in 2008. Grantees felt the Documents were
relevant to their areas of need, had information well organized and for
the most part were clear. Comprehensiveness and detail, while improving
six points in 2008, still remain as the lowest rated items for
Documents. Further improvements in the detail that allows grantees to
meet their programs' needs and comprehensiveness in addressing issues
they face will drive overall grantee satisfaction with the Department.

##### Results by Program

-   Chief State School Officers had a significant increase in their
    satisfaction with the Department in 2008 -- up 12 points. Ratings
    across all of the Department performance areas were up for this
    group and complaints were down. Most performance areas including: ED
    Staff/Coordination, Online Resources, Technology, and Documents were
    rated in the 50s last year. This year all of those areas received
    ratings in the 70s or very near 70. (Documents was rated 69). One
    item of note, this group also continues to have the lowest response
    rate of any program with only 33% responding in 2008.

-   OELA Title III State Directors were the least satisfied with the
    Department in 2008 with satisfaction at 59. However, they did rate
    satisfaction significantly higher in 2008 than they did in 2007.
    Significantly higher ratings in the areas of Documents, Technology,
    ED Staff/Coordination, and Online Resources drove this increase in
    satisfaction. Title III Directors gave positive ratings to OELA for
    providing them an interpretation of Title III and helping them
    implement it in their state. However, the effectiveness of the
    relationship between Title I and Title III still receives low marks
    with respect to encouraging collaboration.

-   OESE State Education Technology Directors rated satisfaction higher
    in 2008, but not significantly so, and they continue to be among the
    least satisfied groups. Their evaluation of most of the driver areas
    did not change significantly from last year. However, they did rate
    Documents significantly higher with the biggest gains in the
    relevance and comprehensiveness of the Documents. State Education
    Technology Directors thought the relationship with the EETT program
    office was very effective. They found the guidance document provided
    by EETT office to be useful and that EETT was helpful in providing
    an interpretation and implementation of Title II, Part D.

-   OESE State Title I Directors rated their satisfaction with the
    Department lower in 2008 than they did in 2007. However, it was not
    a statistically significant drop. State Title I Directors rated most
    of the driver areas the same in 2008 as they did in 2007. The only
    exception was ED Staff/Coordination, which was rated higher in 2008
    with the biggest gains in ED Staff knowledge and accuracy of
    responses. In 2008, State Title I Directors found SASA to be more
    effective in using technology to enhance communication between the
    Department and the state. Information for monitoring for Title I was
    found to be available and useful. Technical Assistance on Even Start
    and Homeless Education also received strong ratings.

-   State Title V, Part A Directors gave the Department slightly lower
    satisfaction scores in 2008. However, the drop was not significant
    and they still remain among the most satisfied groups. There were no
    significant changes in their ratings of the driver areas. Title V,
    Part A Directors continued to give strong ratings to the Innovative
    Programs office for providing an interpretation and implementation
    of Title V, Part A. The guidance document and information at
    national meetings were found to be useful. The working relationship
    with the Title V, Part A program office was rated as being quite
    effective.

-   Lead Agency Early Intervention Coordinators continue to rate their
    satisfaction with the Department among the middle of the program
    groups. There was no change in their rating of satisfaction from
    last year. Their evaluation of driver areas did not show significant
    changes from last year with the exception of ED-funded Technical
    Assistance. Lead Agency Early Intervention Coordinators found
    ED-funded Technical Assistance to be more responsive, more accurate
    and gave higher ratings to their collaborations with ED staff and
    other ED-funded providers. OSEP's Technical Assistance and
    Dissemination Centers were found to be even more responsive to
    answering questions in 2008 compared to 2007. Monitoring and State
    Improvement Planning Division Contacts also received high marks for
    responsiveness, supportiveness in helping complete performance
    plans, and dissemination of accurate information.

-   State Directors of Special Education rated their satisfaction with
    the Department among the lowest of the groups. In particular, State
    Directors of Special Education gave significantly lower ratings to
    the ED Staff/Coordination. This was driven by significant and
    sizable drops in the sufficiency of legal guidance, collaboration
    with other ED offices and responsiveness to their questions.
    Technical Assistance and Dissemination Centers received
    significantly higher marks for the timeliness of disseminating
    information in 2008. Centers also received strong ratings for their
    responsiveness to answering questions and information requests.

-   EDEN/ED*Facts* Coordinators' satisfaction with the Department
    increased significantly in 2008. EDEN/ED*Facts* Coordinators are now
    among those groups most satisfied with the Department. Their
    increase in satisfaction was driven by higher ratings of Documents
    and Online Resources. ED-funded Technical Assistance, while not a
    key driver of satisfaction was also rated significantly higher by
    EDEN/ED*Facts* Coordinators in 2008. Documents were found to have
    more relevance and be more comprehensive for the scope of issues
    they faced. EDEN/ED*Facts* continues to give very high ratings to
    the Partner Support Center for the support they provide and the
    Department's EDEN/ED*Facts* team for their support being timely,
    useful, relevant, and accurate. Some of the lower rated areas saw
    significant gains in 2008, including the data submission process
    helping them meet federal mandates for data collection and
    submission and helping to improve the state data collection and
    submission processes.

-   OVAE Career and Technical Education State Directors satisfaction
    with the Department remains among the lowest of the programs. Online
    Resources was rated significantly lower in 2008 driven by a lack of
    ease of finding materials online. Conversely, Career and Technical
    Education State Directors rated Documents significantly higher in
    2008 with the biggest gain in the sufficiency of detail in the
    Documents to meet their needs. Those who were monitored by OVAE
    within the past year, found the federal monitoring process effective
    in identifying and correcting compliance issues in their state and
    improving program quality. One area where there may be an
    opportunity to improve involves the compatibility with state
    reporting systems for both the Consolidated Annual Report and the
    state plan submission database.

-   Directors of Adult Education and Literacy rating of satisfaction
    with the Department was the highest of all programs. Three of the
    driver areas received significantly higher scores in 2008. Online
    Resources was up due to a higher rating of ease of submitting
    information via the Web. Technology was up mostly due to an
    improvement in the quality of assistance regarding the automated
    process in sharing accountability information. Documents were found
    to have more detail, be more comprehensive and have more clarity.
    Directors of Adult Education found the National Reporting System to
    be easy to use and the training to support it to be useful. National
    meetings and conferences provided by OVAE receive high marks for the
    relevance and usefulness of the information.

[]{.mark}

[]{.mark}

***Recommendations***

While there have been sizeable improvements in 2008, the Department of
Education should continue to focus on improving the high-impact,
low-performing areas as first priorities. The grid below shows the
performance and impact of each driver area. Technology and Online
Resources are among the lower performing areas and both have
considerable impacts on satisfaction.

###### []{.mark}

###### As was the case in 2006, addressing Technology and Documents should be the highest priorities. The 

###### ![](media/image27.wmf){width="6.220138888888889in" height="4.184722222222222in"}

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### Within the area of Online Resources, improving the ease of finding materials online should be a high priority. While it appears to be an issue across most programs, State Title I Directors, Career and Technical Education State Directors, and State Title V, Part A Directors found it to be most problematic. Respondents thought the Department was doing better in using Technology in 2008. However, expected reduction in paperwork remains the lowest rated area and the relatively modest ratings for the automated process with respect to sharing accountability information and effectiveness in improving state's reporting suggest that further improvements to the automated process may be opportunities to improve grantee satisfaction.

###### []{.mark}

###### Respondents' scores indicate that they feel the Documents are clearer, more organized and more relevant. Comprehensive of the information in addressing the scope of issues grantees face may be a secondary area to target for improvement. 

###### 

###### The areas of ED Staff/Coordination and ED-funded Technical Assistance remain strengths, which saw further improvements in 2008 and the current levels of performance should be maintained in these areas. Continue to build on the gains in consistency of responses and collaboration with the other Department offices.

######  

###### []{.mark}

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### APPENDIX A : SURVEY QUESTIONNAIRE

###### 

###### 

###### 

U.S. Department of Education

## Grantee Satisfaction Survey 2008

### Introduction 

The Department of Education (ED) is committed to serving and satisfying
its customers. To this end, we have commissioned the CFI Group, an
independent third-party research group, to conduct a survey that asks
about your satisfaction with ED's products and services and about ways
that we can improve our service to you.

The CFI Group and the Department of Education will treat all information
in a secure fashion and will only provide aggregate results to
Department personnel. All information you provide will be combined with
information from other respondents for research and reporting purposes.
Your individual responses will not be released. This brief survey will
take about 15 minutes of your time.

If you have any questions about this survey, please contact Jeanne
Nathanson, 202-401-0618. Jeanne.Nathanson@ed.gov.

This interview is authorized by the U.S. Office of Management and Budget
Control No. 1800-0011.

*Please note that ALL questions on this survey (unless noted otherwise)
refer to your experiences over the PAST 12 MONTHS.*

Please click on the \"Next\" button below to begin the survey.

### Program Office 

Q1. Please indicate your current program office.

1\. Chief State School Officers (ASK CSSO1.)

2\. OELA -- Title III State Directors (SKIP TO STAFF1)

3\. OESE -- State Educational Technology Directors (SKIP TO STAFF1)

4\. OESE -- State Title I Directors (SKIP TO STAFF1)

5\. OESE -- State Title V, Part A Directors (SKIP TO STAFF1)

6\. OSERS/OSEP  -- Lead Agency Early Intervention Coordinators (SKIP TO
STAFF1)

7\. OSERS/OSEP -- State Directors of Special Education (SKIP TO STAFF1)

8\. OPEPD -- EDEN/ED*Facts* Coordinators (SKIP TO STAFF1)

9\. OVAE -- Career and Technical Education State Directors (SKIP TO
STAFF1)

10\. OVAE -- Directors of Adult Education and Literacy (SKIP TO STAFF1)

11\. None of the above currently applies (SKIP TO END)

### ED Staff/Coordination

***(ask Q2 only if Q1= 1.Chief State School Officers)***

Q2. Do you have regular contact with a senior ED officer who can respond
to your policy and programmatic questions?

1.  Yes

2.  No

    Please think about the interactions you have had with senior ED
    officers and/or other ED staff.

    PLEASE NOTE: This does not include ED-funded technical assistance
    providers, such as regional labs, national associations,
    contractors, etc.

    ***(ALL PROGRAMS OTHER THAN CHIEF STATE SCHOOL OFFICES START WITH
    Q3)***

    On a scale from 1 to 10, where "1" is "Poor" and "10" is
    "Excellent," please rate the senior ED officers' and/or other ED
    staff's:

    Q3. Knowledge of relevant legislation, regulations, policies, and
    procedures

    Q4. Responsiveness to your questions

    Q5. Accuracy of responses

    Q6. Sufficiency of legal guidance in responses

    Q7. Consistency of responses with ED staff from different program
    offices

    Q8. Collaboration with other ED program offices in providing
    relevant services

    ***(Ask Q9 only if Q8 is rated \<6)***

    Q9. Please identify your state's best example of collaboration
    across offices that you would offer as a model for ED.

### ED-funded Technical Assistance

Q10. Do you have interaction with ED-funded providers of technical
assistance (e.g., regional labs, national associations, contractors,
etc.) separate from ED staff?

1.  Yes

2.  No (SKIP TO WEB 1.)

3.  Don't know (SKIP TO WEB 1.)

    Please think about your interactions with ED-funded providers of
    technical assistance. On a 10-point scale, where "1" is "Poor" and
    "10" is "Excellent," please rate their:

    Q11. Knowledge of relevant legislation, regulations, policies, and
    procedures

    Q12. Responsiveness to your questions

    Q13. Accuracy of responses

    Q14. Sufficiency of legal guidance in responses

    Q15. Consistency of responses with ED staff

    Q16. Collaboration with ED staff in providing relevant services

    Q17. Collaboration with other ED-funded providers of technical
    assistance in providing relevant services.

### Online Resources

Please think about your experience using ED's online resources. On a
10-point scale, where "1" is "Poor" and "10" is "Excellent," please rate
the:

Q18. Ease of finding materials online

Q19. Ease of submitting information to ED via the Web (e.g., grant
applications, annual reports, accountability data)

### Technology

Q20. Now think about how ED uses technology (e.g., conference calls,
video-conferencing, Web conferencing, listservs) to deliver its services
to you. On a 10-point scale, where "1" is "Not very effective" and "10"
is "Very effective," please rate ED's effectiveness in using technology
to deliver its services.

*(Ask Q21 only if Q20 is rated\<6)*

Q21. Please describe how ED could better use technology to deliver its
services.

Q22. Think about how ED is working with the states to develop an
automated process to share accountability information. Please rate the
quality of this assistance from ED. Use a 10-point scale where "1" is
"Poor" and "10" is "Excellent."

Q23. How effective has this automated process been in improving your
state's reporting? Please use a 10-point scale where "1" is "Not very
effective" and "10" is "Very effective."

Q24. How much of a reduction in federal paperwork do you expect over the
next few years because of ED's initiative to promote the use of
technology in reporting accountability data (e.g. EDEN/ED*Facts*)?
Please use a 10-point scale where "1" is "Not very significant" and "10"
is "Very significant."

### Documents

Think about the documents (e.g., publications, guidance, memoranda) you
receive from ED.

On a 10-point scale, where "1" is "Poor" and "10" is "Excellent, please
rate the documents':

Q25. Clarity

Q26. Organization of information

Q27. Sufficiency of detail to meet your program needs

Q28. Relevance to your areas of need

Q29. Comprehensiveness in addressing the scope of issues that you face

### ACSI Benchmark Questions 

Now we are going to ask you to please consider ALL of ED's products and
services and not only those we just asked about.

Q30. Using a 10-point scale on which "1" means "Very Dissatisfied" and
"10" means "Very Satisfied," how satisfied are you with ED's products
and services?

Q31. Now please rate the extent to which the products and services
offered by ED have fallen short of or exceeded your expectations. Please
use a 10-point scale on which \"1\" now means \"Falls Short of Your
Expectations\" and \"10\" means \"Exceeds Your Expectations.\"

Q32. Now forget for a moment about the products and services offered by
ED, and imagine the ideal products and services. How well do you think
ED compares with that ideal? Please use a 10-point scale on which \"1\"
means \"Not Very Close to the Ideal\" and \"10\" means \"Very Close to
the Ideal.\"

Now please indicate the degree to which you agree or disagree with the
following statement.

Q33. Overall, when I think of all of ED's products and services, I am
satisfied with their quality.

1.  Strongly Agree

2.  Agree

3.  Disagree

4.  Strongly Disagree

5.  Does Not Apply

### Closing 

Q34. In the past 6 months, have you issued a formal complaint to ED to
express your dissatisfaction with the assistance you've received from an
ED staff member?

1.  Yes

2.  No

Q35. Finally, please describe how ED can improve its service to you.

Thank you again for your time. To complete the survey and submit the
results, please hit the "Finish" button below. Have a good day!

###### 

###### 

# 2008 -- OELA -- Title III State Directors Questions

### Custom Questions -- OELA -- Title III State Directors 

Think about the particular ways in which you have received technical
assistance from the Office of English Language Acquisition (OELA).

Think about the [one-on-one consultations]{.underline} you have had with
program officers. On a 10-point scale, where "1" is "Not very effective"
and "10" is "Very effective," please rate the effectiveness of the
one-on-one consultations in:

Q1. Providing you an interpretation of Title III

Q2. Helping you with your implementation of Title III in your state

Q3. What can OELA do over the next year to meet your state's technical
assistance and program improvement needs?

Think about your experiences seeking information at OELA's Clearinghouse
Web site (www.ncela.gwu.edu). On a 10-point scale, where "1" is "Not
very effective" and "10" is "Very effective," please rate the
effectiveness of the Web site in:

Q4. Providing you with the information you needed

Q5. Helping you inform programs serving ELLs in your state

Q6. Think about the [working relationship between Title III and Title
I]{.underline}.

> On a 10-point scale, where "1" is "Not very effective" and "10" is
> "Very effective," please rate how effective the Department has been in
> encouraging collaboration between Title I and Title III.

(Ask only if question is scored \<6)

Q7. Please describe how the working relationship between Title III and
Title I could be improved.

Think about the Title III Biennial Report that is being used to collect
data this cycle and the role of the regional meetings. On a 10-point
scale, where "1" is "Poor" and "10" is "Excellent," please rate the
regional meetings for:

Q8. Helping familiarize you with the Biennial Report form

Q9. Allowing for your input and comments for refining the Biennial
Report form

*Changes from 2007: None*

2008 -- OESE -- STATE EDUCATIONAL TECHNOLOGY DIRECTORS

### Custom Questions -- OESE -- State Educational Technology Directors 

Think about the particular ways in which you have received technical
assistance from the Enhancing Education Through Technology Program
(EETT).

First, consider the [one-on-one consultations]{.underline} with EETT
program officers. On a 10-point scale, where "1" is "Not very effective"
and "10" is "Very effective," please rate the effectiveness of the
one-on-one consultations in:

Q1. Providing you an interpretation of Title II, Part D (Enhancing
Education Through Technology)

Q2. Helping you with your implementation of Title II, Part D (Enhancing
Education Through Technology)

Think about the [guidance document]{.underline} provided by the EETT
program office.

Q3. On a 10-point scale, where "1" is "Poor" and "10" is "Excellent,"
please rate its usefulness.

Think about the Educational Technology State Directors\' [national
meetings]{.underline} (i.e., national technology conferences, SETDA
meetings) where the EETT program office made a presentation

Q4. On a 10-point scale, where "1" is "Poor" and "10" is "Excellent,"
please rate the usefulness of the information presented at these
meetings.

Think about the [federal monitoring process]{.underline} as it relates
to the Enhancing Education Through Technology program office. On a
10-point scale, where "1" is "Not very effective" and "10" is "Very
effective," please rate the effectiveness of the federal monitoring
process in:

Q5. Helping you with your compliance efforts

Q6. Helping you to improve performance results

Think about your [working relationship]{.underline} with the Enhancing
Education Through Technology program office.

Q7. On a 10-point scale, where "1" is "Not very effective" and "10" is
"Very effective," please rate the effectiveness of this relationship.

(Ask only if question is scored \<6)

Q8. Please describe how your working relationship with EETT could be
improved.

Q9. What can EETT do over the next year to meet your state's technical
assistance and program improvement needs?

*Changes from 2007: None*

2008 -- OESE -- STATE TITLE I DIRECTORS

### Custom Questions -- OESE -- State Title I Directors 

Think about the [technical assistance]{.underline} you have received
from the Title I office, Student Achievement and School Accountability
(SASA). On a 10-point scale, where "1" is "Poor" and "10" is
"Excellent," please rate the:

Q1. Usefulness of technical assistance on Standards and Assessments,
Instructional Support, and Fiduciary of Title I, Part A of NCLB

Q2. Usefulness of technical assistance on Neglected and Delinquent

Q3. Usefulness of technical assistance on Even Start

Q4. Usefulness of technical assistance on Homeless Education

Think about the [information on monitoring for Title I]{.underline} you
have received. On a 10-point scale, where "1" is "Poor" and "10" is
"Excellent," please rate the:

Q5. Availability of information on monitoring for Title I

Q6. Usefulness of information on monitoring for Title I

Q7. Think about how SASA uses electronic communications approaches such
as email, Web casts, and WebEx to provide you information. On a 10-point
scale, where "1" is "Not very effective" and "10" is "Very effective,"
please rate SASA's effectiveness in using technology to provide
information.

(Ask only if question is scored \<6)

Q8. Please describe how SASA could better use technology to provide
information.

Q9. Again, thinking about SASA's use of electronic communications
approaches to provide information: on a 10-point scale, where "1" is
"Not very effective" and "10" is "Very effective," please rate SASA's
effectiveness in using technology to enhance communication between ED
and the state.

Q10. What can SASA do over the next year to meet your state's technical
assistance and program improvement needs?

*Changes from 2007: Q7 and Q9 wording revised*

2008 **--** OESE -- STATE TITLE V, PART A DIRECTORS

### Custom Questions -- OESE -- State Title V, Part A Directors 

### (Innovative Programs) 

Think about the particular ways in which you have received technical
assistance from the Title V, Part A (Innovative Programs) office. First,
consider the [one-on-one consultations]{.underline} with Title V, Part A
program officers.

On a 10-point scale, where "1" is "Not very effective" and "10" is "Very
effective," please rate the effectiveness of the one-on-one
consultations in:

Q1. Providing you an interpretation of Title V, Part A

Q2. Helping you with your implementation of Title V, Part A

Q3. Think about the [guidance document]{.underline} provided by the
Title V, Part A program office. On a 10-point scale, where "1" is "Poor"
and "10" is "Excellent," please rate its usefulness.

Q4. Think about Title V, Part A [national meetings and conference
calls]{.underline} (including the Steering Committee's national meetings
and the program office's conference calls for orientation and follow-up
to the Steering Committee's national meetings) where the Title V, Part A
program office made presentations. On a 10-point scale, where "1" is
"Poor" and "10" is "Excellent," please rate the usefulness of the
information presented by the program office.

Think about the [federal monitoring process]{.underline} as it relates
to the Title V, Part A program office. On a 10-point scale, where "1" is
"Not very effective" and "10" is "Very effective," please rate the
effectiveness of the federal monitoring process in:

Q5. Helping you with your compliance efforts

Q6. Helping you to improve performance results

Q7. Think about your [working relationship]{.underline} with the Title
V, Part A program office. On a 10-point scale, where "1" is "Not very
effective" and "10" is "Very effective," please rate the effectiveness
of this relationship.

(Ask only if question is scored \<6)

Q8. Please describe how your working relationship with the Title V, Part
A program office could be improved.

Q9. What can the Title V, Part A program office do over the next year to
meet your state's technical assistance and program improvement needs?

#### *Changes from 2007: None*

2008 -- OSERS/OSEP -- LEAD AGENCY EARLY INTERVENTION COORDINATORS

### Custom Questions -- OSERS/OSEP -- Lead Agency Early Intervention Coordinators

Think about the technical support State Contacts from the Monitoring and
State Improvement Planning Division of the Office of Special Education
Programs provided. On a 10-point scale, where "1" is "Poor" and "10" is
"Excellent," please rate the staff's:

Q1. Responsiveness to answering questions

Q2. Supportiveness in helping you complete your state's federally
required performance plans/reports/applications

Q3. Dissemination of accurate information

Q4. Dissemination of information in a timely manner

Think about the [Technical Assistance and Dissemination
Centers]{.underline} from OSEP. On a 10-point scale, where "1" is "Poor"
and "10" is "Excellent," please rate the centers':

Q5. Responsiveness to answering questions

Q6. Responsiveness to information requests

Q7. Support to positively impact on your State's SPP improvement
targets.

Q8. Think about the [Communities of Practice]{.underline} from OSERS. On
a 10-point scale, where "1" is "Not very effective" and "10" is "Very
effective," please rate its effectiveness in addressing systems
improvement issues of the state.

Q9. What can OSEP do over the next year to meet your state's technical
assistance and program improvement needs?

##### Changes from 2007: added Q7

2008 -- OSERS/OSEP -- State Directors of Special Education

### Custom Questions -- OSERS/OSEP --State Directors of Special Education 

Think about the technical support State Contacts from the Monitoring and
State Improvement Planning Division of the Office of Special Education
Programs provided. On a 10-point scale, where "1" is "Poor" and "10" is
"Excellent," please rate the staff's:

Q1. Responsiveness to answering questions

Q2. Supportiveness in helping you complete your state's federally
required performance plans/reports/applications

Q3. Dissemination of accurate information

Q4. Dissemination of information in a timely manner

Think about the [Technical Assistance and Dissemination
Centers]{.underline} from OSEP. On a 10-point scale, where "1" is "Poor"
and "10" is "Excellent," please rate the centers':

Q5. Responsiveness to answering questions

Q6. Responsiveness to information requests

Q7 Support to positively impact on your State's SPP improvement targets

Q8. Think about the [Communities of Practice]{.underline} from OSEP. On
a 10-point scale, where "1" is "Not very effective" and "10" is "Very
effective," please rate its effectiveness in addressing systems
improvement issues of the state.

Q9. What can OSEP do over the next year to meet your state's technical
assistance and program improvement needs?

#### *Changes from 2007: Intro to Q1-4 changed, Q7 added, Q8 OSERS changed to OSEP*

2008 **--** OPEPD **--** EDEN/EDFACTS COORDINATORS

### Custom Questions -- EDEN/ED*Facts* Coordinators 

Think about the support provided by the U.S. Department of Education
EDEN/EDFacts team. On a 10-point scale, where "1" is "Poor" and "10" is
"Excellent," please rate the:

Q1. Timeliness of the support

Q2. Usefulness of the support

Q3. Accuracy of information

Q4. Relevance of information

Think about the EDEN/EDFacts data submission process. On a 10-point
scale, where "1" is "Not very effective" and "10" is "Very effective,"
please rate the effectiveness of the data submission process in:

Q5. Helping you to meet federal mandates for data collection and
submission

Q6. Helping you to streamline your federal data collection and
submission processes

Q7. Helping you to improve state data collection and submission
processes

Q8. How much of a reduction in federal paperwork do you expect over the
next few years because of the EDEN data submission process? Please use a
10-point scale where "1" is "Not very significant" and "10" is "Very
significant."

Q9. How much do you expect the data you provide to contribute to
improving education performance measurement? Please use a 10-point scale
where "1" is "Not very significant" and "10" is "Very significant."

Q10. Think about the training provided by the EDEN/EDFacts team on data
submission. On a 10-point scale, where "1" is "Poor" and "10" is
"Excellent," please rate the training's usefulness.

Q11. On a 10-point scale where "1" is", Poor" and "10" is "Excellent,"
please rate the support provided by the Department's Partner Support
Center.

Q12. What has been the most significant change to your state data
collection and submission process as a result of the EDFacts work?

Q13. How can the Department's ED*Facts* team be most helpful to you in
meeting federal mandates for data collection, submission, analysis, and
reporting in the coming year?

*Changes From 2007: None*

2008-- OVAE -- CAREER AND TECHNICAL STATE DIRECTORS

### Custom Questions -- OVAE -- Career and Technical State Directors

Think about the [Consolidated Annual Report (CAR)]{.underline} as a way
to report your state's performance data to OVAE. On a 10-point scale,
where "1" is "Poor" and "10" is "Excellent," please rate the CAR's:

Q1. User-friendliness

Q2. Compatibility with state reporting systems

If you were monitored by OVAE within the last year, think about the
[federal monitoring process]{.underline} as it relates to your Perkins
grant. On a 10-point scale, where "1" is "Not very effective" and "10"
is "Very effective," please rate the effectiveness of the federal
monitoring process in:

Q3. Identifying and correcting compliance issues in your state

Q4. Helping you to improve program quality

Q5. Think about the [national leadership conferences and
institutes]{.underline} offered by OVAE last year (i.e., Data Quality
Institute in Savannah, GA, and Programs of Study Institutes in Chicago,
IL, and Washington, DC). On a 10-point scale, where "1" is "Poor" and
"10" is "Excellent," please rate the effectiveness of these sessions on
helping you to improve the quality of your programs and accountability
systems.

Q6. Think about the [monthly Up-to-Date with DATE e-mails]{.underline}
that are sent to you from OVAE. On a 10-point scale, where "1" is "Not
very effective" and "10" is "Very effective," please rate the
effectiveness of these e-mails in keeping you informed about key issues
pertaining to all aspects of your Perkins grant (i.e., CAR reporting,
State Plan submissions).

Q7. Think about the [Peer Collaborative Resource Network
(PCRN)]{.underline} as it concerns OVAE. On a 10-point scale, where "1"
is "Poor" and "10" is "Excellent," please rate PCRN's usefulness to your
program.

If you used the [state plan submission database]{.underline} last year,
think about this process as a way of submitting your five-year state
plan to OVAE. (If you did not use the state plan submission database
please select "N/A.") On a 10 point scale, where "1" is Poor" and "10"
is Excellent," please rate the database on its:

Q8. User-friendliness

Q9. Compatibility with state reporting systems

Q10. What can OVAE do over the next year to meet your state's technical
assistance and program improvement needs?

*Changes From 2008: Q5 and Q6 reworded, Q7 from 2007 dropped*

2008 OVAE -- DIRECTORS OF ADULT ED AND LITERACY

### Custom Questions -- OVAE -- Directors of Adult Ed and Literacy

Think about the National Reporting System as a way to report your
state's performance data to OVAE. On a 10-point scale, where "1" is
"Poor" and "10" is "Excellent," please rate the NRS's:

Q1. Ease of reporting using the NRS Web-based system.

Q2. Think about the training offered by OVAE through its contract to
support the National Reporting System (NRS). On a 10-point scale, where
"1" is "Poor" and "10" is "Excellent," please rate the usefulness of the
training.

If you have been monitored, think about the federal monitoring process
as it relates to your AEFLA grant. On a 10-point scale, where "1" is,"
Not Very Effective" and "10" is "Very effective," please rate the
effectiveness of the federal monitoring process on the following:

Q3. Being well-organized

Q4. Providing pre-planning adequate guidance

Q5. Setting expectations for the visit.

Q6. Using state peer reviewers in the federal monitoring process.

Think about the national meetings and conference offered by OVAE. On a
10-point scale, where "1" is "Poor" and "10" is "Excellent", please rate
the information provided at these conference and institutes on the
following:

Q7. Being up-to-date

Q8. Relevance of information

Q9. Usefulness to your program

Think about the national activities offered by DAEL. On a 10-point
scale, where "1" is," Poor" and "10" is "Excellent," please rate the
activities on the following:

Q10. Usefulness of the products in helping your state meet AEFLA program
priorities.

Q11. How well the technical assistance provided through the national
activities address your program priorities and needs? Please use a
10-point scale where "1" means "does not address needs very well" and
"10" means "addresses needs very well."

Q12. What can DAEL do over the next year to meet your state's technical
assistance/program improvement needs?

Changes From 2007: None

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### APPENDIX B: NON-SCORED RESPONSES

**  **

**      **

######  

###### 

![](media/image28.wmf){width="7.6402777777777775in" height="6.13125in"}

###### 

###### APPENDIX C: ATTRIBUTE TABLES 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

![](media/image29.wmf){width="7.590277777777778in"
height="8.038194444444445in"}

![](media/image30.wmf){width="6.8in" height="7.372222222222222in"}

![](media/image31.wmf){width="7.1402777777777775in"
height="7.560416666666667in"}

![](media/image32.wmf){width="7.1402777777777775in"
height="7.699305555555555in"}

![](media/image33.wmf){width="7.440277777777778in"
height="2.290277777777778in"}

![](media/image34.wmf){width="7.1402777777777775in"
height="7.561111111111111in"}

![](media/image35.wmf){width="7.15in" height="2.2909722222222224in"}

![](media/image36.wmf){width="7.1402777777777775in"
height="7.561111111111111in"}

![](media/image37.wmf){width="7.8in" height="2.540277777777778in"}

![](media/image38.wmf){width="7.1402777777777775in"
height="7.561111111111111in"}

![](media/image39.wmf){width="7.15in" height="2.2909722222222224in"}

![](media/image40.wmf){width="7.1402777777777775in"
height="7.561111111111111in"}

![](media/image41.wmf){width="7.820138888888889in"
height="2.423611111111111in"}

![](media/image42.wmf){width="7.1402777777777775in"
height="7.561111111111111in"}

![](media/image43.wmf){width="7.8597222222222225in"
height="2.392361111111111in"}

![](media/image44.wmf){width="7.1402777777777775in"
height="7.561111111111111in"}

![](media/image45.wmf){width="7.820138888888889in"
height="2.845833333333333in"}

![](media/image46.wmf){width="7.1402777777777775in"
height="7.561111111111111in"}

![](media/image47.wmf){width="7.070138888888889in"
height="2.540277777777778in"}

![](media/image48.wmf){width="7.1402777777777775in"
height="7.561111111111111in"}

![](media/image49.wmf){width="7.040277777777778in"
height="2.8027777777777776in"}

###### APPENDIX D: VERBATIM RESPONSES BY PROGRAM

###### 

*The comments reported in this section appear in the original verbatim
wording and format as provided by the respondent.*

**Chief State School Officers**

***Q9. Please identify your state's best example of collaboration across
offices that you would offer as a model for ED.***

*Our agency is developing Response to Intervention models collaborative
between the Office of School Improvement and the Office of Special
Education; the curriculum unit in the Office of School Improvement
worked with the Office of Career and Technical Education to crosswalk
academic and CTE standards; the Educational Technology office is
coordinating data collection across all office and with the Center for
Educational Performance and Information to ensure consistent data
reporting and usage.*

*Regular, senior level staff meetings to help ensure collaboration.*

*The Division of Educator Quality (School Leadership), the Division of
Innovation and Support, and the Palmetto Priority Schools program (for
failing schools) are working on collaborative, innovative strategies to
turn around chronically low-performing schools.*

**Q21. Please describe how ED could better use technology to deliver its
services.**

First, ED needs to create services that would be of use to SEAs.

Too many times, technical assistance sessions are held within 10 days of
a grant announcement. This means that the cost of getting to DC (or even
regional sites) skyrockets for potential applicants. Rather than
site-based technical assistance sessions, webinars or webcasts would
provide a cost-effective, equally information means to provide technical
assistance to all potential applicants. Conference calls without
web-based support and documents are not that helpful.

**Q35. Finally, please describe how ED can improve its service to you.**

**Consistency**

BIE has a great working relationship with the current administrative
staff and leadership at ED. Naturally, this changes as personalities
change in various positions. Consistency in leadership and the
management team at ED would ensure that the positive relationship and
partnership we currently enjoy would continue on. Current leadership in
BIE realizes the tremendous responsibility that ED has in providing
quality education to all socio-economic and ethnic population in the
United States. BIE appreciates that ED is our partner in addressing this
quality delivery of services to Indian Children.

**Communication**

Learn what SEAs and LEAs really do.

Less legalese and more every day language/discussion regarding products
and services.

Written communication could be clearer. ED\'s application of
accountability approvals to states must be more transparent and
even-handed. ED\'s understanding of the administration of federal
programs, particularly in urban school districts, should derive more
from experience, insight, and research and less from politically-based
need for immediate change.

**Collaboration**

More collaboration between IDEA and NCLB on any level.

**Clarity of Requirements/Transparency**

More technical assistance and more transparency in how decisions are
made.

Most resources tend to be highly legal, complex, and a reiteration of
the laws and/or regulations. Materials are of a high level, but not as
practical and useful to the field. States must and do rewrite
information in simplified terms as well as how to turn \'law\' into
practice.

**Timelines**

The timeline for EDFacts may not be realistic due to data issues in the
various states. You may need to adjust these deadlines.

**Posting Proposals**

Post two funded grants from each funding cycle on program Web sites.
This would avoid the costs associated with FOIA requests\--personnel
time, copying time, mailing time. In this day and age, the way that FOIA
requests are handled are archaic and obsolete. Since December 2007, we
have been trying to get copies of funded proposals from four different
US ED program offices. To date, only one (VPSC) has provided these
proposals, and they arrived 5 months after the initial request. The
others: ITEST, Early Reading First, Educ Research on School Leadership
(through IES) have not only ignored repeated requests, they have also
not responded to inquiry from our U.S. Senators. We understand that
personnel time is valuable\--exactly the reason why the offices should
just go ahead and post funded proposals.

**Other**

Frankly, I am not expecting the ED to meet all of a state\'s needs.
Education is primarily the responsibility of a state and we should not
expect the USDE to provide the same level of service that should be
expected of State Departments of Education.

I don\'t really know of anything that would be relevant.

[]{.mark}

[]{.mark}

[]{.mark}

[]{.mark}

**OELA - TITLE III State Directors**

***Q9. Please identify your state's best example of collaboration across
offices that you would offer as a model for ED.***

*At the state level, Title I and Title III*

*Collaboration with other state content specialists to provide better
instructional support to ELLs.*

*Communication between Title I and Title III*

*NCLB programs are housed together and collaboration across programs
occurs on a daily basis and ongoing. Initiatives that involve more than
one program area are done together or tools are developed together.*

*No best example, but an example of where collaboration may be needed:
offices should be able to offer similar examples of issues such as what
constitutes supplant vs. supplement.*

*Our State does an excellent job of collaborating. Recently, we have
been collaborating on ELD Math and Science standards. Not only is the
Title III-LEP Coordinator involved, the math and science coordinators,
Title I Director, Content Area Director, and Migrant Coordinator have
been involved.*

*The Oregon Department of Education includes in its T III on-site
monitoring the ODE\'s Civil Rights Specialist. His role is to review
services for ELLs from the students\' rights perspective.*

*We work collaboratively among the Title I, IIA, IID, III, IV, and V
offices\--meeting twice each month. Our applications are consolidated
and our reviews of school districts are also consolidated.*

**Q21. Please describe how ED could better use technology to deliver its
services.**

*It would be nice to have some Director\'s meetings via webinars, etc.*

*More offerings via technology, such as podcasts, conference meetings.*

*Use more online PD resources to provide more training and guidance to
SEAs. Facilitate collaboration among states using online resources.*

####### Use Webinars

*Web-based training would be beneficial\--especially in light of the
expenses for traveling*

**Q3 (OELA). What can OELA do over the next year to meet your state\'s
technical assistance and program improvement needs?**

*Be clear; be consistent.*

*Be consistent in the guidance it provides; be more responsive to
written requests for clarification and issue written guidance that is
based on statute.*

*Continue to provide outstanding program managers and reduce paper work
and monitoring*

*Continue to send out interpretations and guidance documents.
Accompanying these with comprehensible, additional presentations is most
helpful.*

*Have written guidance as to how to implement Title III, especially for
novice state staff. It is very difficult to implement a program based
only on the legislation itself.*

*Host a 2008 OELA Summit Meeting.*

*I can\'t answer this question at this time.*

*I need to know about my state\'s submission to OELA for a change of
Annual Measurable Achievement Objectives (AMAOs). I am also looking
forward to the guidance concerning supplanting.*

*I think the one thing to give us more time on the notice of
interpretation regarding title 11 AMAO\'s. Once that document is
finalizes state will additional time and money to implement them.*

*Inform all states of each state\'s decisions, allow for blanket
application of decisions. Be uniform in treating states and let each
state know - in writing - what has been determined allowable/acceptable.
BE CONSISTENT. Have conviction in decisions and inform all states*

*Inform us when TIII Native American Program grants are awarded to
LEAs.*

*More clarity in guidance/e.g., Title III consortia and private
schools/home schools*

*More guidance re: ELLs with special needs.*

*No. I am very pleased with all the assistance I get.*

*Nothing*

*OELA\'s staff prompt and accurate responses are invaluable as we send
questions to them when a local LEA needs an answer right away.
Continuing to provide this service will continue to be much appreciated.
Also, Title III State Directors meetings in which we can share and learn
from each other are also invaluable.*

*Please provide an official document, which states can use as a guidance
document for Title III.*

*POINT THE APPROPRIATE LEADERSHIP. DEFINE THE ROLL OF OELA IN THE LEP
PARTNERSHIP. I AM WAITING FOR THE REGULATORY TO BE PUBLISHED.*

*Provide a simple clarification of students to count as LEPs.*

*Provide guidance and monitoring reports in a timely manner.*

*Provide guidance on meeting AMAO, which has been promised for years and
has not been given.*

*Provide more guidance on AMAOs and assessments for ELLs.*

*Provide proactive training and guidance instead of back tracking and
retribution. Have better communication with SEAs.*

*Provide specific models of recommended monitoring surveys.*

*Publish relevant guidance.*

*Responding in a timely manner to questions would be extremely helpful.
Having efficient State Director\'s Annual Meetings with a specific and
relevant agenda would also be most helpful.*

*Showcase states that have highly developed Title III programs and use
the funds effectively.*

*We are waiting for the AMAO, RTI, and Special Education guidelines
documents.*

*We need guidance on ELP assessment and accountability.*

*Work with us on the quality of the data requested and the venues
through which it is provided.*

***Q7 (OELA). Please describe how the working relationship between Title
III and Title I could be improved.***

*Build some requirements to collaborate into the legislation and/or
guidance.*

*I think we cannot have joint meetings and discussions. They don\'t
communicate very often between the two offices of Title I and Title
III.*

*Improve collaboration at federal level that trickles down to state*

*Invite Title I and Title III to the same meetings.*

*It appears like these two offices work in complete isolation. Perhaps
having Title I representation at the annual Title III Director\'s
meeting would be a good start.*

*Make Title III a part of Title I.*

*More communication between the two offices and agreement on guidance -
sometimes guidance is different between the two offices.*

*More state director collaboration. Shared or overlapping meetings.*

*Required regional meetings.*

*Since many of the Title III functions are moving to SASA, the
relationship should improve.*

*The level of funding for Title III should be greatly increased. Title I
seems to be able to fund most of its mandates, whereas Title III hardly
provides funding to implement all of its mandates or requirements.*

*The teeth of accountability rests with Title I because of the amount of
funding and the number of school administrative units affected. If Title
I was as knowledgeable about Title III as Title III is about Title I
perhaps English language learners would not be marginalized but would be
included in ALL STUDENTS. Title III targets a small fraction of school
administrative units and allows for too many accountability features for
non Title III receiving schools to ignore accountability requirements
for English language learners.*

*There IS no working relationship with Title I.*

*Title I may say one thing and then Title III says it is wrong and sites
us for something Title I said was okay.*

*Title I needs to take a more active role in this relationship and show
responsibility towards the needs of the ELL population.*

*Title III staff generally have the information on how to educate
language minority students which generally T I staffs do not. Setting
expectations and linking the work of T III and I can be a first step.*

*Title III staffers need to understand Title I.*

***Q35. Finally, please describe how ED can improve its service to
you.***

*Responsiveness/Timeliness*

*Assistance from Title III needs to be clear, consistent, and timely.
Guidance is never provided in writing, always verbal. States cannot act
on verbal interpretations of statute and it changes from meeting to
meeting. Program officer keeps changing and that results in lack of
consistency in assistance.*

*Be expedient with answers*

*Meet projected timelines for reports and responses.*

*Provide timely guidance on issues we face\...*

*Respond to the formal complaint!*

*Communication*

*Ensure staff working at the federal level is well trained and well
informed regarding the policies they are being required to implement and
oversee. Improve communication with SEAs to ensure that the information
provided is relevant and decisions made are not out of touch with the
reality of implementation at the state and school district level.*

*Spend more time in the field understanding how states differ.*

*Data Collection*

*Cut-down on the EXCESSIVE amount of data collection and reporting
required.*

*The more data that can be collected through the EDEN system rather that
written reports from states.*

*Web site*

*Make the site easier to navigate, with relevant information. Sometimes
things can be hard to find and it takes a lot of time. I just end up
calling.*

*Mostly my experiences on the grant side has been a little bit
challenging. It has been challenging to get a copy of the application.
The electronic part is user friendly it\'s just hard to get to download
an application. We had a lot of support in getting the assistance we
needed.*

*Meetings*

*Do not cancel the OELA Summit*

*Have regional meetings for technical assistance but provide for all
expenses to be covered through ED. Travel for state employees, even that
funded by federal funds, has been greatly curtailed in many states.*

*Coordination/Collaboration*

*More coordination between the different tile corporations. I am
involved in two and the differences in the support given.*

*Guidance Document*

*Have written guidance.*

*List the ELP tests each state uses, each state\'s rules for LEP
identification and definition of AMAO1 (progress) and AMAO2
(proficiency).*

*Please provide to states an official guidance document or regulations
for Title III. NCLB is getting close to reauthorization and states still
do not have a Title III guidance document. This is problematic in light
of how complicated AMAOs are.*

*We need guidance on implementing requirements. Guidance that has been
developed through consultation with experts and researchers is most
helpful. ED can improve services by continuing to rely on experts in the
field to develop guidance and technical assistance.*

*When guidance to one state is developed, send it to ALL states. Do not
put the burden of asking on the states. Do not treat each state
differently, do not continue to allow one state to do X and another not.
Consistency, fairness, and more than anything - put it in writing.
Develop conviction*

*When new Federal requirements are issues, like designing a RTI program
for ELLs and others, we need immediate guidelines from Title III office.
If not we are left to create this documents with what we understand
should be done. Then months or years after Title III comes out with
guidelines that sometimes are different then ours - creating confusion
and distrust on State agencies from the field. This can be avoided if
from the beginning we are all on the same page.*

*Mailing List*

*Keeping a current State Director\'s mailing list would be great. I\'ve
tried to get on their mailing list several times but I still don\'t get
their emails and updates.*

##### Other 

*All things considered, ED is doing well!*

*Continue to provide high quality technical assistance to states. \[Name
Deleted\] has been most helpful as our Title III Officer.*

*No, I am very pleased with the working relation that we have with
them.*

#### OESE -- State Educational Technology Directors

***Q9. Please identify your state's best example of collaboration across
offices that you would offer as a model for ED.***

*Have a conference call that involves programs that cross over. Ex.
Title IA and Title IID*

*Our state is not good at collaborating either.*

####### Program Requirements for Title IID Competitive\--with relation to private (non-public) school participation

*The auditing of federal programs and the Title IID helps me to get to
other offices to get the help I need.*

*Title IID and Title IA, IC, IIA, III, VA, and VIA/REAP - We all work on
one team in order to collaborate on professional development, technical
assistance, monitoring, and resources provided for LEAs.*

*We are working with the Center for Education Performance and
Information (CEPI) on data collection and longitudinal data systems. It
would be great for the USDOE Office to work more closely with EDEN. EDEN
is asking for ED tech data, but the USDOE is not providing the
requirement to make it part of our state\'s mandatory collection. This
is causing difficulties in determining how the information is collected.
My office is also responsible for a large portion of Title IIA statewide
activity dollars that is funding a statewide teacher portal, but Title
IIA and Title IID are separate initiatives. It would be better if the
priorities of those federal programs were more aligned.*

*We have worked very hard to link the school improvement process with
our NCLB programs unit.*

***Q21. Please describe how ED could better use technology to deliver
its services.***

*Easier access to on line resources, Webinars, on line trainings, etc.
Since funds are limited any type of TA that can be provided via a
distance with out travel would be beneficial.*

*ED could use the technology such as interactive video conferencing to
provide more services, collaboration with other Title programs
(connecting state directors of the different programs to talk about
issues including parental involvement, professional development in
educational technology tied to core content, sharing resources across
states including application forms, data collection tools/strategies,
curriculum, and digital content, etc.) Ed could help states network
across programs to leverage resources, learn from others\' successes
stories, and sharing resources developed in one state with other states
that may not have the staffing to create great resources.*

*ED needs to explore more advanced technologies such pod casting video
conferencing.*

*ED should be encouraged to use presentation tools such as WebEx to
deliver application information and provide more contact with state
customers.*

*It is difficult to answer this because we do not receive much service
from ED, through technology or otherwise, other than what we initiate.*

*Offer it from time to time. My only interaction has been a couple of
times, so we need more of it.*

*Program office does not tend to communicate with states. Communication
with USDOE program office comes through SETDA or other sources. The
USDOE Ed Tech office should be providing State Ed Tech directors with
updates and information regarding what is going on. Technology is a
support to ensure this communication happens\--it is 50+ contacts for
USDOE\--not unmanageable. I should not have to attend a national SETDA
meeting to find out what is happening with the OET or OESE departments
at USDOE*

*Provide CVC\'s monthly*

*They need to know how to use it. Also, the EETT program doesn\'t
collect information electronically except through EDEN. 21st Century
collects all annual reports electronically.*

*Update capacity to support IP-based videoconferencing. Post monthly or
quarterly information (via email or online newsletter) to provide
program updates, describe/facilitate connections with other programs and
program providers, cite proven practices, etc.*

***Q8 (EdTech). Please describe how your working relationship with EETT
could be improved.***

Be honest and straightforward with us, being timely with information.

***Q9 (EdTech). What can EETT do over the next year to meet your state's
technical assistance and program improvement needs?***

*Advocacy for continued educational technology funding!*

*Assist with getting clearance at to what exactly will be needed for
EDFacts.*

*Bi-monthly email/newsletter of relevant issues or deadlines that are
coming up.*

*Clearly define 8th grade technology literacy*

*CONTINUE TO WORK WITH EDEN ON REPORTING REQUIREMENTS.*

*EDEN is very hard to understand - please make it less technical for
those of us actually having to collect the information.*

*EETT has done an excellent job in keeping the states abreast of the
current happenings in Washington. An improvement to me would be more
EETT sponsored national meetings were the states participates directly
with ED staff and not through any association.*

*EETT should provide ongoing communication to State Ed Tech
Directors\--including program updates. EETT should not rely on SETDA or
other organizations to provide communication to State Ed Tech
Director\'s, but should develop ongoing communication channels on its
own. The information should be published and/or sent directly to the
states in case states are unable to attend the in-person meetings. EETT
office needs to work in conjunction with the OET to ensure a common
vision. EETT and OET need to work together to develop and incorporate
21st century skills within the \'work\' of these programs.*

*First and foremost it must be fully funded to have any impact. If fully
funded, help states explore, discuss, share ways of evaluating impact of
formula grant program and overall EETT evaluation.*

*Frequent communication and increase technical assistance available.*

*Give definitive answers in writing when we reach out for assistance.*

*Give examples of what you believe are adequate representations of how
states can meet the 8th grade technology literacy requirement, the
teacher technology literacy and integration expectations in EDEN.*

*Given the new NETS for students, most states including mine are
struggling to find/develop an effective tool for measuring technological
literacy. Existing tools are aligned to the old standards and therefore
not addressing the indicators that we need. Any assistance in this area
would be helpful.*

*I don\'t think any at this time.*

*I think consistently communicate.*

*Increase it\'s funding. We have had a lot of reduction in funding over
the past few years abd it makes it hard, so restore the funding back to
the initial level to where we started.*

*Increase the budget.*

*It would be helpful if we could tie Title IID formula funding to
reporting on 8th grade tech literacy.*

*It would be nice to have a place where states can visit to gather
implementation ideas or view best practices regarding EETT.*

*It would be very helpful, now that the program had had a few years of
operation, for USDOE to update the program guidance and provide a
frequently asked questions section.*

*It would help if there was more consistency and cross knowledge on data
items such as 8th grade tech literacy. Thanks.*

*Keep up keeping us informed.*

*Letting states know exactly what is needed for reporting an evaluation
purposes. Possibly sending a cable with timelines and expected
deliverables for reporting purposes.*

*Monthly phone conference and/or CVC for FAQ*

*More contact. Additional person-to-person contact.*

*More publications regarding guidance and procedures. It seems like we
see more correspondence from SEDTA than we do from the Department. SETDA
is a nice way to get the info out however due to declining funds some
states will be unable to attend SETDA meetings and if the Department
could get this information out in other ways and not solely rely on
SEDTA that would be beneficial.*

*Not much unless they have the ability to increase these funds
significantly.*

*Offer more meetings and guidance about monitoring and accountability in
EETT and the expectations in the future of the program.*

*Provide additional funding.*

*Provide examples from other states that will assist with program
improvement.*

*SETDA is a good regular source of info. As long as EETT shows up and
gives the best info available they will be able to meet our needs.
Clarify the role of EDEN within EETT that would be good.*

*Share more promising practice rather than talking about the process.*

*Stop providing \'politically correct\' answers to program questions.
Responses given at meetings are typically vague enough to leave us
knowing little more than we did previously.*

*They have been hesitant to give us information. Update their
information and make it more timely.*

*We, in Illinois enjoy the patience that ED has given to the new
staffers. Thank you for being there and taking the time to ease our
understanding of EETT.*

***Q35. Finally, please describe how ED can improve its service to
you.***

*Communication*

*Additional and more frequent contact.*

*I answered this question in the EETT question, but a more direct
involvement with training and sharing among states directors sponsored
by ED and not through the any association. Although the associations
have a place in informing states and representing states on EETT issues
on the hill, ED should have the direct responsibility of having all
states to meet, train, inform, and share information.*

*More sharing of what other states are doing.*

*They have been really good about working with us. Just more consistent
communications. The reauthorization of funding and keeping us abreast.*

*Through detailed information and timelines for reporting purposes.*

*Web site*

*Create a useful, easy to navigate Web site. Post information regularly.
Maintain Web site so resources are timely.*

*Make items easier to find via the search mechanism. Communicate
available services. We don\'t really know what services ED provides.*

*Technical Assistance*

*Again, I think if there could be more TA, guidance, etc that is
provided through webinars, conference calls, and publications would be
great. It is great that ED uses other resources to get their information
out however I don\'t think that is the sole way to share information
with the States. Only having one or two meetings a year is difficult
when funding may not be available for attendees. (Not asking for more
funding simply asking that the DE consider lack of funding when setting
up trainings that require travel).*

*By providing accurate and timely information and provide additional
technical assistance.*

*Clarity on Requirements*

*ED does a very good job in responding to questions and offering
clarification on most issues. There does seem to be confusion, however,
over the reporting requirements of programs because of the different
types of reports that are collected. If an effort would be made to
consolidate that information between the GPA, Performance Reporting, and
EDEN, so that we could understand the purpose of each, that would be
very helpful.*

*Meetings*

*ED needs to call a Title IID meeting separate (or adjacent) to the
SETDA Meetings. One-hour meetings to barely touch upon one topic is not
enough. It would be great to have states work together on strategies for
effectively measuring technological literacy or the increased capacity
of teachers to integrate technology into teaching and learning. Or a
meeting to thoroughly discuss the findings from the 10 ESETP grants and
how they could impact the work each of us is doing in our state.*

*Meetings, conference calls and or training more frequently.*

*Significant advance notice of meetings is especially helpful. States
are very slow to approve meetings. Meetings often fall during peak
travel periods (late June and late November or early December). Consider
scheduling for slower periods.*

*Timeliness*

*Reduce turnover of stats. Provide more timely information. Ask us more
what we need.*

*Guidance*

*ED should follow up with states on feedback from virtual site visits;
ED should provide guidance and interpretations that are accurate related
to EETT.*

*Issuing program guidance a little earlier in the implementation
period.*

*Just keep moving forward in guiding us through what may be ATTAIN*

##### Other

*Get rid of the extreme testing requirements because we just don\'t have
the resources to do it all! We want to help our schools to teach their
kids, not test, test, test. We need a more balanced approach. I can\'t
think of anything at the moment.*

*I have actually experienced an increase in support from the USDOE
office.*

*I have no complaint with the technology office. My only complaint is
dealing with EDEN.*

*I realize that EETT has been slated for elimination for the last 3
years. Does that mean that the program is treated as an orphan that
should not be supported? It appears that way when I see the services
that are available through ED to other areas of K12 education programs.
That is a difficult position to work from, but until the program is
dead, it should be treated the same as other programs.*

*Less political posturing/bias.*

##### 

#### OESE -- State Title I Directors

***Q9. Please identify your state's best example of collaboration across
offices that you would offer as a model for ED.***

*A team of state staff from Title I, II, and III plus Special Education
and District Improvement offices are collaborating on the LEA
Improvement process beginning with a fact-finding on-site visit. Results
are used to develop technical assistance and the improvement plan,
including selection of the corrective action for those LEAs in the 3rd
year of improvement.*

*Consolidated NCLB application of eight formula programs*

*Migrant and SASA*

*Monitoring of our LEAs across all NCLB program areas in one process and
one instrument.*

*Regular, scheduled program meetings between program offices/directors.*

***Q21. Please describe how ED could better use technology to deliver
its services.***

*Having conference calls/webinars for Title I folks in general*

*I find the conference calls awkward. If they could do a web cast or a
pod cast where I could log on to. A blog would be a better use of
everyone's. It would have a written format, which would be easier all
the information would be in.*

*It is very difficult to find information on your Web site. Redesign it!
Improve your search engine. Get a users group to help you. Possibly have
a portal for State and district level users that is different from
public users. Let the user decide which portal to enter.*

*Podcast or other type of delivery of information so that it can be
accessed at any time*

*Provide more audio and Web conferences*

*Sometimes the documents are read to us on the call instead of
discussion taking place.*

***Q8 (Title I). Please describe how SASA could better use technology to
provide information.***

*On conference calls, send out information ahead of time and don\'t read
the information to us. We can read it. Plan these conference calls with
the input of a few of the reliable state directors so that the calls
will actually meet our needs. e.g. the differentiated accountability
conference calls just began to address some of the real issues by the
end of the third call. the political agenda appeared to get in the way
of clear answers. At the end of the first call give send an email to
Title I directors with an email address to send in questions. Sort the
questions into frequently asked questions that would be of value to many
(and address these in a follow up call), and state specific questions
that can be answered to the state asking the question. Then publish a
list of the answers maybe to all questions.*

*Overcome time zone differences*

*Use Webcasts, etc. on a more frequent basis. Right now the only use of
technology on a regular basis are the few emails we get from the
Director of SASA\--which we appreciate.*

***Q10 (Title I). What can SASA do over the next year to meet your
state's technical assistance and program improvement needs?***

*Cite my state for insufficient staff in title I and not meeting the
state\'s maintenance of effort requirement where we have almost no state
funded positions or initiatives. Schedule a series of Just In Time
conferences at times of the year that coincide with what we should be
doing at that time of year, e.g. PNP Planning guidance, Program Planning
supplant issues, Program evaluation, final expenditure report
Monitoring, etc. Give examples of best practices, and strategies to work
within the state government.*

*Continue to use technology to distribute information and to give state
directors a chance to participate more frequently in meetings without
the need to travel. Continue to work with other sections to streamline
and coordinate the data collection so that EDFacts & the CSPR are
aligned and are collecting the appropriate data. We strongly support
electronic data collection, but it\'s been a bumpy road.*

*Continue with current practices. Title I Directors have been waiting
for guidance for Targeted Assistance programs. \[Name deleted\] does a
good job as director. Communication and information is good. Keep Title
I directors in the loop as information is provided to Chiefs, assessment
directors, Title II A directors, etc. as much of the information is
directly related to Title I requirements.*

*Decrease amount of time it takes to receive response from email or
phone questions.*

*Hold more \'WebEx\' kinds of conference*

*I am very satisfied with the level of assistance my state receives. The
responses to our questions are consistent regardless of the staff
called.*

*Information on a more timely basis. Forget the rule making that\'s too
late.*

*More direct contact\--such as teleconferences and other means of
communicating information without having to travel. Also, I am pleased
that ED is sponsoring meetings on RTI and the new math research. Thanks*

*Provide faster response to questions and concerns.*

*Provide more electronic updates and interactions on all the program
areas. Develop a consistent process and agenda for program reviews.*

####### Provide more guidance

*\[Names deleted\] are SASA staff that I work with frequently. They are
extremely helpful and very knowledgeable.*

*To upon request put responses to questions in writing.*

*What there doing is ok. No comment.*

*While costs on travel are high, the complexity of the issues relating
to Title I services to SEAs and LEAs are of concern. Face-to-face
meetings and networking to solve problem is still my preferred method.
Hearing the information from the source with multiple ED staff available
for specific issues is helpful.*

*Work more emphatically with state\'s to ensure alignment of
technologies being used to provide technical assistance and to collect
program data.*

***Q35. Finally, please describe how ED can improve its service to
you.***

***Consistency***

*Consistency would be good\...and someone to answer our questions in a
timely manner.*

*I find that the Homeless Coordinator receives conflicting information
as to the use of Title I funds for homeless students. Please provide
more clarity. Also, we need much more assistance with neglected and
delinquent.*

##### Availability

*Just have staff available to answer questions by phone and e-mail.*

Timeliness

*Continue to provide timely, relevant information.*

*Just information on a more timely basis*

*Loosen policy oversight of technical assistance and information that
could be provided by program staff. Response to questions take too long
because of the red tape that has to be processed before answers can be
given.*

*Provide faster responses to questions and concerns.*

*Speed up the response time to emails and phone calls*

*\
***Web site**

*Redesign their Web page so it\'s more keyword search. A keyword search
that works.*

Publications

*ED limits the number of publications a state can receive, specifically
the number of EDGAR copies a state can get because it is on the Web.
However, grantees do not always have access to the Web. We should be
able to obtain as many copies as are needed in order to share with
grantees. I also think all grant programs should have training for new
directors on a yearly basis. Training should not only include program
requirements but should also include regulations and OMB Circulars
requirements as well.*

[]{.mark}

Communications

*Having more policy memos that they put out, so they don\'t update them
as much as they need to. I\'d like the policy letters sent to states to
available to states, I'd like there to be database of policy questions
and answers.*

*Providing models and examples to the greatest extent possible.*

*Really appreciate the listservs, Webcast, Eden, etc. as well as the
informal emails received directly from the SASA Director.*

[]{.mark}

Guidance

*Overall, the service from the SASA office of OESE has been very good.
We are still waiting for new guidance on Title I targeted assistance
programs, but other guidance recently issued on Schoolwide and fiscal
has been helpful. Other sections of ED have been less helpful. There has
been little concrete guidance from Title III and information has often
been less than complete. The new proposed interpretations should be
helpful in clarifying the expectations nationwide. The other large
concern is the timing of the CSPR data collection requirements and the
coordination with the EDFacts data collection. The CPSR data
requirements are being finalized AFTER the year is over and it is too
late to ask LEAs to collect data in a different way. At the same time,
the EDFacts data collections are being finalized for the NEXT school
year and sometimes the two data collections are still not in alignment,
especially for LEP data and teacher quality data. We strongly support
electronic data collections, but the data elements need to be determined
BEFORE the school year starts and they must be aligned between CSPR and
EDFacts.*

##### []{.mark}

**Other**

*I am very pleased with my contacts at SASA. I feel that there are very
responsive to my needs. They are to be commended.*

*This is a difficult one because I understand ED\'s limitations in terms
of providing guidance that is responsive to the needs of any particular
state. That being said, however, in too many instances the needs of
individual state\'s are not addressed when in fact if ED extended
itself, they may be able to do so.*

#### OESE -- State Title V, Part A Directors

***Q9. Please identify your state's best example of collaboration across
offices that you would offer as a model for ED.***

####### Interagency meeting and collaborative initiatives and projects concerning student learning.

*The technical assistant oversight team.*

***Q21. Please describe how ED could better use technology to deliver
its services.***

*We could receive more information electronically.*

*We had a \'virtual\' monitoring in November of 2006. We are still
waiting for the written response. Ridiculous\...*

***Q8 (Title V). Please describe how your working relationship with the
Title V, Part A program office could be improved.***

*\[NO RESPONSES\]*

***Q9 (Title V). What can the Title V, Part A program office do over the
next year to meet your state's technical assistance and program
improvement needs?***

*Continue to clarify the implications of the \'zero\' appropriation.
Continue efforts to restore funding or discontinue the legislation*

*I don\'t know, because there are not going to be any funds next year.
Help fund the program.*

*It\'s not going to funded next year.*

*Keep us informed of developments*

*Provide clear and concise information to the states. Be more accessible
and responsive.*

*Satisfied*

*The Title V Part A program office is doing all that can be reasonably
expected for a zero funded program. I don\'t know what more would be
needed or prudent in the absence of refunding.*

*When they perform an audit, or a monitoring visit, please make sure the
written synopsis of the event is completed by 30-60 days.*

*Work to get Federal funding back into the Title V, Part A Program by
promoting the things that Title V, Part A provides for states and
monitor how states are doing during the coming year of zero funding.
Continue holding national meetings to assist state (SEA) program
managers through this difficult time.*

***Q35. Finally, please describe how ED can improve its service to
you.***

***Technology***

*Continue to utilize technology and develop guidance incorporating
questions from the field*

##### Responsiveness

*Be more responsive and accessible.*

*\
**Web site***

*Easier Web site to find guidance etc*

*ED\'s Web site is very difficult to work with.*

*I recognize and appreciate that ED folks work hard and try their best.
I believe that they, like us, carry more workload than is reasonable,
and this prevents more state-level support. I imagine ED staff turnover
affects their ability to understand programming over time just as it
does for us. Many of the resources ED provides are outdated, the Web
sites are difficult to navigate, and the emphasis on
\'research/evidenced based\' \'best practices\' has reached absurd
levels. Many of these identified exemplary programs have startlingly
limited value across regions, cultures, communities, and with other
subpopulations. Moreover they often require a level of fidelity and a
length of implementation that defy the realities of today\'s schools. It
is frustrating to guide districts and schools away from localized
practices toward programming that will likely not accomplish what it
advertised as being capable of accomplishing.*

*The website should be improved since it is difficult to locate
information by category and the search function is almost useless; i.e.
often over 500 items result from a specific search.*

Staff

*Unless the program is funded unless they start funding. They're not
going to have staff. If they do then they need more staff. We are more
knowledgeable than they are.*

Other

*Get the entitlement dollars out more timely.*

*If every office was as responsive and as communicative as the Title I
office, you\'d have an amazing federal agency!*

*Satisfied*

**OSERS/OSEP -- Lead Agency Early Intervention Coordinators**

***Q9. Please identify your state's best example of collaboration across
offices that you would offer as a model for ED.***

*National Transition Initiative*

*Our state uses our state interagency coordinating council and
committees of the council to assure all agencies and many parents and
providers are involved with the program and issues. There is no similar
method at the federal level and it needs to be followed nationally to as
many of the states issues are really federal level items that have never
been aligned across funding sources. (CMS, Samsa, Title IV E, etc.)*

***Q21. Please describe how ED could better use technology to deliver
its services.***

More opportunities. If you miss the call, most of the time you just miss
it. If you have a small staff and a busy schedule, you cannot always
join in. More notice\...you can\'t always drop everything for a call
that\'s on short notice. Calls have a limited scope, so you don\'t
always get a chance to ask your burning questions.

Need one place to look on line or have them all linked together. We have
to look at multiple sites, which is not realistic or helpful for easy
references. Teleconference and materials need to be holistic and not so
short time related. OSEP needs to make realistic expectations as need
requirements, data points, and processes are added. We spent time on
month phone call that have different or ongoing updates to materials
which meant inches of paper and that many changes. Figure out a process
and stay with it. To do so work with state for realities of work load
and what can be done within the current funding levels. Part C is
monitored as heavily as part b. However, Part B has much higher funding
levels. There needs to be a balance in process expectations and clear
and concise directions, and materials. Information needs to be received
by states with enough lead-time to meet the requests. This does not
currently happen and states are in constant run mode to try to meet the
OSEP expectations. We do not see the outcomes for children and families
being improved but rather must shift to provide OSEP with constant and
changing demands. Put balance into service delivery and monitoring
activities.

Use of go-to-meeting or other Web conferencing technology to enhance
conf. calls

***Q9 (OSERS). What can OSEP do over the next year to meet your state's
technical assistance and program improvement needs?***

*1. Consolidate the requirements so that they are not scattered thru TA
call notes, correspondence, and other guidance materials. 2. Decrease
the reporting requirements, they are out of the scope of what a
government administrative office can manage and are bordering on
research projects. States which do not have the resources required to
manage them are floundering. 3. Accept a state\'s self-assessment of its
deficits, without forcing sanctions and penalties to local programs,
which may be out of the local program\'s control.*

*Better and more widely available TA around general supervision and
monitoring. This could be a combination of on-line, site visits,
conferences. What\'s currently available is not enough given the
emphasis on accountability.*

####### Continue the good work!

*Continue to provide timely responses to questions*

*Finalize Part C regulations. Seek increased appropriations for Part C.
Stabilize interpretations and stop changes in accountability reporting.
Fund data system development.*

*Get more state contacts. With the SPP/APR process being so cumbersome
and time-consuming, state contacts just cannot handle the increased
workload effectively.*

*I generally find those responsive to helping me with any issues.*

*Implement a one-year moratorium on annual reporting in order to allow
states to \'catch up\' with what OSEP expects! Implementing changes to
State Plans does not occur overnight and with personnel turnover a
critical factor, time is needed.*

*Increase use of technology to the disseminate information such as
Webinar and Web meetings. They should schedule conference back to back
so we don\'t have to fly so much for example data and accountability
meetings.*

*Issue a letter stating that the Baltimore meeting and the December
Washington meeting are required for Part C Coordinators. Our states\'
department directors are getting very tight on approving travel to
national conferences and it is critical that we be there to learn from
each other and meet with OSERS staff and technical assistance staff.
Without the requirement, I worry that fewer and fewer of the
coordinators will get approval to attend and will miss out on these
extremely important meetings.*

*It is very difficult to attend all the teleconferences\--it keeps me
from getting work accomplished as sometimes they are not relevant to my
needs*

*Our RRC is slowly developing more interest and expertise in Part C (it
took a very long time). And, although they don\'t have anyone on staff,
they are very good about seeking out expertise from other TA providers
or other individuals to help us. I think this is probably the way to
go.*

*Provide more national publications/training materials that are directed
to the Part C systems*

*Publish Part5 C Regs*

*Put the funding back to TA systems to meet service deliver issues and
needs and such high degree of TA funding on OSEP monitoring tasks. If it
takes all the TA then OSEP should see that the process is not realistic.
States do not have more resources and can\'t shift the funding to write
reports. Therefore the on going TA for the service provides that used to
be provided has shifted to monitoring activities. This is not supporting
that state and local program partnerships needed to keep up with family,
parent, and children needs or resources.*

*Relax a little and let the work happen; access more resources; fund
data systems work*

*Strengthen NECTAC and stop pushing the RRC to provide TA in early
childhood, which they do not specialize in.*

*They are providing all kinds of training so nothing at this point.*

*Timely distribution of information to accurately address APR indicators
more timely.*

*Visit the state to learn the states programs and circumstances.*

*Whatever OSEP can do to facilitate the development of Part C federal
regulations would be greatly appreciated.*

*When providing TA over Webinars, teleconference calls, etc. provide
follow-up in writing in a timely manner. If states cannot participate in
one of these TA events than critical information is missed.*

***Q35. Finally, please describe how ED can improve its service to
you.***

*Communication*

*Bring states, families, and SICC chairs together to assure
accountability and data collection. Reduce paperwork, as we cannot keep
this workload up. Work with congress through DOE secretary and others in
high chain of command for real funding and resources needed for EIS. The
cost saving for all needs to be used to demonstrate why funding
increases are critical. Stop changing things each month, quarter or
year. Obtain consistency. Write clear and concise rules and regulations.
If is takes OSEP attorneys to write them and then interpret them, what
does that mean for families and states who are trying to use them??? The
massage should be clear that too much regulation becomes as ineffective
as not enough or none. Recognize the urgency and take reduction action
now.*

*Come and learn about the states program.*

*Sometimes its responses are \'legalese\' and don\'t really answer a
question\--be more specific*

*The correspondence needs to be more concise and clearer. I think the ED
staff try to be clear but the ED legal staff get hold of it and make it
less clear. The Part C regs have not been finalized 3 1/2 after the IDEA
statute was passed. This causes confusion as we have a new statute but
old and draft regs. They need to get the regs out ASAP*

*Mentoring*

*I am new to the position and it just seems foreign to me\--I would have
benefited from a mentor\--*

***Reporting Burden***

*I understand the legal review process is necessary but it makes all
relations and communications stilted and often not very practical.
Lighten up.*

*Reduce the burden of increasing requirements. Finalize the Part C
regulations.*

**Technical Assistance**

*More specific focus for Part C education-lead states*

*More targeted TA around general supervision.*

Timeliness

*Provide more timely feedback and uniformly apply interpretations and
implementation of regulations across states.*

Web site

*I find the search feature of the Website difficult to use. Perhaps
organizing it more like the NECTAC site would make it more
user-friendly.*

*Well I'd like to see a few more tweaks in the portals would be great.
They are getting better and much more user-friendly.*

***Other***

*Since we are not a Dept. of Education, we don\'t actually see too many
products from ED related to infants and toddlers since the focus is
usually NCLB or Part B. So, my ratings reflect mostly a lack of services
or products related to Part C. Do we have Part C regs yet? No. But a
second set of Part B regs are on their way to OMB.*

##### 

**OSERS/OSEP -- State Directors of Special Education**

***Q9. Please identify your state's best example of collaboration across
offices that you would offer as a model for ED.***

*I can\'t think of one. They are continually asked to collaborate but it
appears that they do not take the time to get to know each other's
requirements. I have sat in more than one meeting when they have been
told that things do not align with NCLB to be ignored.*

*NCLB (AYP) and Voc Rehab (transition)*

*OSEP office and their legal*

*Project based cross division teams*

*Response to Intervention team contains members from Special Education,
Curriculum, Reading First, Early Childhood, and Bilingual Education.*

*RtI brings a number of our divisions together to explore ways in which
we can work toward ensuring students are accessing appropriate services
and best practices are available to all schools. We brought together
representatives of a wide swath of educational professions to set a
direction for us to move in implementing RtI at all levels.*

*Some work being done between OSEP and RSA on summaries of performance
for transitioning students.*

*Special education, migrant, Title 1, and ELL collaborating to provide
guidance on meeting the needs of culturally or linguistically diverse
students without inappropriately identifying them as students with
disabilities.*

*The Division of Exceptional Children Services and Division of Federal
Programs (Title 1, etc.), attempt to conduct joint monitoring of local
school districts. We have developed a cross-walk to identify
commonalities to prevent duplication and to maximize our limited state
level resources.*

*The partnership that they between IDA OESE Title I.*

*Title 1 and IDEA are regular collaborators in our office.*

*We have a monthly NCLB team meeting that includes special education,
Title, state assessment, data collection, HQT, and the secretary and/or
deputy secretary of education. We discuss all relevant topics related to
NCLB and IDEA.*

*Work on our SPDG (creation and implementation of an integrated
three-tiered model of intervention, known as the MIM).*

***Q21. Please describe how ED could better use technology to deliver
its services.***

*Calls are only offered at one time and not staggered and in our state
we cannot access because of state level meetings that are scheduled. It
would be nice to vary the schedule or if the information is important to
have more than one call.*

*ED introduces new points in conference calls and does not always
provide written confirmation of new information. Calls, in particular,
have had poor connections, interrupting conferences, etc.*

*Ensuring that all conference calls provide a muted line. Most
conference calls require participants to mute their own lines, which
doesn\'t happen very often, making it extremely difficult to hear the
message.*

*Just the frequency of the use and also to allow more interaction. It
would be better to divide states up to be more customer friendly and to
allow for more interaction.*

*Lots of other technologies available, e.g., podcasts; satellite
conferencing; video streaming. Conference calls are not the only
technology available.*

*Not sure.*

***Q9 (OSEP). What can OSEP do over the next year to meet your state's
technical assistance and program improvement needs?***

*Become more aware of the problems states face on a regular basis and
remove the unreal expectations placed upon states.*

*Continue the assignment of my current state contact. I have been
through many state contacts in the past three years. While I know it is
difficult to predict how long a person will remain in a position, I
think some analysis of the rationale for the frequent turnover may be in
order. Having to educate each contact assigned to the specifics on my
state each time a new assignment is made is time consuming and
frustrating, especially around the complexities of my state\'s
assessment system.*

*Cut back on the Indicator work. Working with OSEP is overly legalistic
and far too often delayed responses just to check with OGC \-\-- this is
supposed to be a service program, not a legal exercise.*

*Expand Communities of Practice Continue efforts to enhance
collaboration of technical assistance providers Implement electronic
reporting for AT programs*

*It is important for federal program managers to understand the unique
needs of states. It is assumed that each and every state is unique but I
can assure you that Alaska\'s schools are rural and remote unlike any
other states definition of rural and remote. We would appreciate the
opportunity to educate and provide learning experiences to our state
contact. We believe that such opportunity only move to bring the state
and federal government closer together on the issues that we might
mutually encounter.*

*Just be sure to read the time lines correctly.*

*Less TA centers and more dollars to the RRC\'s who care about meeting
your individual needs and are responsive.*

*Major Need: Improve the timeliness and consistency of information
provided by senior staff to the SEA Directors. We still learn of
\'what\'s going on\' or \'going to happen\' from other sources who are
told these things in the context of other meetings well before we or our
chiefs are given the information. This is not a good practice.*

*Not change State contact; be responsive to requests; provide TA when
requested; no additional data reports and reduce current reports;*

*Over all I have been satisfied. Consistency of targets and know
priority areas as soon as possible.*

*Provide clarity on what requirements are.*

*Provide more guidance around disproportionally and use of early
intervening funds and reporting.*

*Provide specific guidance on how to address underrepresentation data
for SPP \-- provide clear guidance on suspension expulsion indicator \--
develop technical assistance center focused on bilingual special
education \-- provide clear guidance regarding target population for 2
percent assessment*

*Provide timely information*

*Put into writing the information that is currently delivered only
during teleconference calls, so that if the call occurs at a time the
desired participant is unavailable, the information can still be
obtained.*

*Reduce non-mandated data collection requirements; make items more
meaningful; cease changing data items with less than 2 years advance
notice; provide greater technical assistance resources*

*Spend less money on the large number of TA centers and give more to
RRCs to provide specific state targeted TA. The centers are very
difficult to work with and are all about publishing and name-dropping*

*States more targeted technical assistance that goes beyond giving us
Web site information. We need contact with specific activities that have
worked in other states regarding student progress and improvement state
indicator data. We need to get our feed back quicker. Your opportunity
to work on the feedback gets here to late. By the time we get feedback
the opportunity to act on it is already gone.*

*They need to be more responsive in a timely manner and provision of
technical assistance online resources is not effective.*

*We could use more assistance in helping to find quality technical
assistance to bring to the state to assist districts in meeting targets
on the SPP.*

*With all of the regulations they should still make available e-mail
address\'s to experts who will respond to specific questions that are
not addressed in the technical assistance documents.*

*Work with officials responsible for NCLD to promote consistent data
definitions and reporting requirements.*

***Q35. Finally, please describe how ED can improve its service to
you.***

##### 

***Technical Assistance/Guidance***

*Again, I think targeted technical assistance that is specific to states
needs. Technical assistance that results in improved data and improve
students outcome.*

*Provide clearer guidance; allow states more time to prepare to
implement new data collection requirements*

*Provide expectations and guidance up front before we are required to
report instead of requiring a report on a nebulous item and then telling
us we did it wrong. Keep reporting requirements to exactly what is
required by law and no more.*

##### Consistency

*Consistency of liaison. We had four different ones in the past two
years.*

*Consistency in answers to questions about the collection and reporting
of data for the State Performance Plans. Please note, the technical
assistance offered in conjunction with the SIG project is outstanding.*

*Just refer back to the previous comment in being more timely and
consistent with information.*

##### Technology

*Continue to use technology to deliver information; Assure consistency
in the assistance/information provided by staff to each state (i.e.
SPP/APR).*

##### Reporting Burden

*Decrease bureaucracy, paperwork, and allow states to identify 2 or 3
FAPE in the LRE Indicators to focus on and do not require all 8. Allow
states to choose six general supervision indicators rather than all 12
to focus on \-\-- for the purposes of reporting. We understand we are
responsible for compliance in all areas of the law, but the burden of
reporting in the exact manner you wish, every year, on 20 Indicators is
considerable.*

*OSEP has required and changed so many things with the SPP that it is
difficult to keep up. They seem to change data requirements and ignore
the input from the public. The requirement for the parent indicator On
the SPP is not founded in law anywhere and it costs us the most to
implement and tells us very little. I hear a lot from/about the
monitoring section and very little about the work of the research to
practice division. What do they do?*

***\
Web site***

*If there could be more links to physical questions and answers that
would be greatly appreciated.*

##### Communication

*Listen; take the time to understand the states issues when implementing
so many different Federal programs that are not aligned. Figure out who
is your customer. It is never clear who you believe is your audience.
When given honest feedback it is often ignored because it does not match
your mental model. Be respectful to the States. At the last meeting I
attended there were a lot of new staff with the wrong attitudes. Most of
the time it appears that they forget that this work is about serving the
needs of children. They never report how they believe all their efforts
are making the system better for infants, toddlers, and students with
real data that their efforts are resulting in improvement.*

*Provide opportunities for key stakeholders to provide input into ED\'s
policies and mandates before implementation! ED could learn a lot by
asking the people that have to implement their mandates!*

*Read the information from my previous narrative within this survey.
Additionally, understand the SEA is a catalyst for change and many times
we are attempting to change systems and practices to improve the
achievement of students with disabilities. Oftentimes, those who are
needing to change most are those who are not reluctant to call USDOE.
When OSEP leadership engages in conversations with individuals, such as
higher education, the SEA deserves some professional courtesy.*

##### 

##### Timeliness/Responsiveness

*More timely responses to legal and regulatory questions; significantly
simplify data collection requirements; increase cross agency
collaborations -e.g., there is little evidence of policy discussions
between OSEP and higher education where issues of students with
disabilities should be a priority*

*Provide answers in a more timely manner. Make reporting more user
friendly, especially the Annual Performance Report. Eliminate redundant
information submission between programs such as Title 1 and IDEA.*

*Responsiveness is key; it must be timely, accurate, and relevant.*

*Timely responsiveness to questions and issues raised. Listening to
stakeholders and seeking input from stakeholders before setting policies
that require more time to implement than states are given or that have
consequences that were unforeseen by ED staff but could have been
foreseen by state agency staff or other stakeholders had they been asked
for input prior to issuing the policy.*

*We are having a hard time getting any questions answered regarding
EDEN/Ed Facts. We send emails and no one responds. How are we supposed
to have accurate and timely data when no one will respond to our
emails?*

***Other***

*My concerns, along with those of colleagues in other SEAs, have been
shared verbally and in writing with senior staff in the US DOE
OSERS/OSEP office; however, we have not filed a \'formal complaint\' as
we think it would be directly detrimental to our relationship to the
OSERS/OSEP office and staff, which only serves to hurt the States.
Retaliation is not uncommon.*

*My staff and I are pleased with the support and encouragement that we
receive from our State Contact. Any opportunity to provide better
understanding regarding the provision of services in Alaska to our state
contact would be appreciated. I would encourage OSERS/OSEP to allow
staff to provide face-to-face technical assistance to states as well as
the opportunity to gain insight and understanding about that provision
of services through direct experience.*

*None that I can think of.*

**OPEPD - EDEN/ED*Facts* Coordinators**

***Q9. Please identify your state's best example of collaboration across
offices that you would offer as a model for ED.***

*We have a data governance model that makes program areas
responsible/accountable for the information they collect and report. It
prompts collaborative responses to data quality issues to increase
alignment across program areas and ensure the Department responds
uniformly to LEAs on data issues.*

***Q21. Please describe how ED could better use technology to deliver
its services.***

*Provide more WEBEX trainings and/or meetings to keep state staff up to
date on best practices, future plans, etc. Use this technology to have
monthly EDFacts \'town meetings\' to keep stakeholders engaged and
current on developments (EDFacts goals/plans/upcoming concerns, etc).*

*Several of the meetings could be done via videoconference or Web ex,
with more frequency and less travel. One face-to-face annual meeting
versus 2 for example - both with NCES and IES Grantees.*

***Q12 (EDEN/EDFacts). What has been the most significant change to your
state data collection and submission process as a result of the EDFacts
work?***

*A vast increase in the detail of data we are required to report,
(especially on small student subgroups such as migrant, ELL, and
homeless) has prompted a rapid transition from legacy aggregate
collections to student-level collections. Far more program area staff
have had to become involved in data submission.*

*Attention to data accuracy and organization of federal reports.*

*Consolidation of data sources. Consolidation of identification
sources.*

*Consolidation of some Federal Collections into EDFacts.*

*Data collection has changed very little to date, but will increase as a
result of EDFacts' more detailed requirements. The submission process is
more complex, detailed, and time consuming than prior to EDFacts. Data
that was reported as a single number in the CSPR is now an EDEN file
with many new data elements. Submission work has increased
significantly.*

*Data quality from the LEA to the SEA has improved. More collaboration
within the SEA has also been a result.*

*EDFacts has driven us to look much deeper at our data collection
systems, data elements collected, interconnectivity of our datasets, QA
processes, and take steps to improve in all aspects how we view and
handle data.*

*Finding out who submits what and how it is collected. Trying to explain
why information is collected through EDFacts and reconcile differences
in reporting.*

*Higher standards. Reduction in redundancy. Better and more consistent
data.*

*in the short term has added another layer of processing/development,
etc and has been very resource intensive. We are hoping that in the
long-term we will see some benefits. It has shifted responsibility in
reporting from several teams within the department to a single team\...
and will make reporting more centralized.*

*Increase in EDFacts submissions due to work with USED.*

*Intra-departmental communication has improved, and more offices are
showing appreciation of each others efforts. Collection redundancies are
being identified and managers are designing more efficient data
collection guidelines.*

*It has become more automated.*

*It was the catalyst for us to implement Data Steward and Data
Governance programs. These programs have since been important to our
work with master data management and data warehouse initiatives.*

*More coordinated with better timelines.*

*More coordination and collaboration between program offices at state
level and between Michigan and USED.*

*More coordination between programs.*

*More detailed attention to the data submitted.*

*More process, more processed collections and data submissions.*

*My job has changed 100%.*

*Not very much has changed for us.*

*Significantly increased the whole staffs workload.*

*Streamlined/reorganized the submission plan*

*The EDFacts requirements are helping us better understand our data
collections and metadata.*

*There are a lot more data checks.*

*We have been able to eliminate unnecessary items from our data
collections.*

*We just received an lds grant. With out the help the questions would
have been answered 2 or 3 instead of 8 or 9 the longitudinal data system
grant.*

*We\'ve had to spend a lot more time and effort on data collection and
changes to what is required to report.*

*While I believe in the long run, EDFacts will be a good thing, the data
collection/submission responsibilities have significantly increased the
workload for our Data Management staff. We are a small state with
limited resources and at times find it difficult to hire people with the
technical qualifications needed.*

*Work has been directed from the program offices to the data management
shop without commensurate funding. Program offices do not yet trust data
management\'s reports, so double the work has to be done to prove the
results are correct. EDFacts is not relevant to our data collection
process.*

***Q13 (EDEN/EDFacts). How can the Department's EDFacts team be most
helpful to you in meeting federal mandates for data collection,
submission analysis, and reporting in the coming year?***

*1) Voice to ED our need for additional staff/funding resources to
accomplish the HUGE task of federal reporting; 2) Continue to enhance
and improve the tools (Submission Plan spreadsheet, EDFacts Portal
elements, etc.)*

*Additional resources would be great.*

*Be more sensitive to the fact that many of use have silos of data that
exist in many formats. We are not always able to send in the files in
the formats requested, as we do not have the resources. The files should
be acceptable in other formats until we have the capabilities to submit
them in the EDFacts format.*

*Better communication to various programs about the collections.*

*Clarity and consistency of data definitions.*

*Clear and concise communications.*

*Continue to be flexible in negotiating data submission timelines with
states.*

*Continue to provide timely information regarding reporting requirements
and changes*

*Crystal clear guidelines and file specs for data collection.*

*Eliminate technical glitches in the EDEN system, e.g., EDEN-DANS
discrepancy analysis; eliminate the 0-filling requirement in EDEN files;
provide more on-line resources, e.g., re-introduce the Charter School
report; start allowing states to use more of the EDEN data for reporting
needs a la CCD\'s table building functionality.*

*Give states funding to buy tools, such as software and training;
provide us with online metadata tool; and facilitate establishment of
user groups for states with like systems.*

*I find the trainings and the meetings very helpful, so more
continuation of those.*

*I really need some assistance in staffing here.*

*I think they are doing a pretty good job as they are.*

*It would help to have more emphasis and reference to submitted data.
Offices tend to be less resistant when they see value to collection
rather than just for the sake of it.*

*Keep changes to a minimum.*

*More one-on-one training with state staff responsible for this
process.*

*More support in terms of federal dollars. A full time coordinator
position for each state. The full time coordinators capacity for. More
coordination of our federal program efforts offices federal state.*

*More timely of the materials received from ED. We still don\'t have all
the file specifications yet we are required to tell what we can and
cannot submit for the coming year. We receive materials from ED (like
IDEA congruency and submission/transition plan) with little turn around
time for us. The IDEA congruency we have less than two weeks to get that
figure out and back to ED. We don\'t have enough staff to do all this.*

*On-going funding. Changes to submission plan have been useful ( with
expected dates of when data is first used).*

*Please fund an EDFacts coordinator for each state. Please continue to
work with federal program offices to continue to refine data needs.
Please examine the utility of the data collected for evaluation and
reporting.*

*Provide at least two years lead time for the introduction of new data
groups or elements before they become mandatory; fund the EDFacts
Coordinator position; provide the specifications at least three months
prior to the collection deadline; resolve the race/ethnicity
implementation process questions and differences between US ED offices.
Provide SEA data quality training courses online(similar to the LEA
courses now offered by CCSSO, SIFA, and Central Susquehanna Intermediate
Unit).*

*Provide realistic lead-time to complete tasks. Improve communication of
known issues. Improve communication regarding use of data by federal
program areas. Communicate long-ranges plans and goals. Eliminate
surprises. Provide more frequent EDFacts status updates. Explore ways
the EDFacts team and state staff can have direct dialogue. Continue to
seek funding to provide to states.*

*Providing the funding we have been promised over the last two years and
have never received.*

*Releasing changes and file specifications earlier. Developing more
reports that coordinators can run from the EDFacts Reporting site.*

*The EDFacts team needs to initiate and continue conversations with US
DOE program area offices regarding the quantity and especially the data
categories/permutations required by each file specification. These
discussions should ensure that only data categories and permutations of
these data categories that US DOE program offices are actively using to
make programmatic and policy decisions should be collected. Often, the
data categories are the reason we cannot submit a complete file.*

*To keep the communications channel clear.*

***Q35. Finally, please describe how ED can improve its service to
you.***

***Accessibility***

*Better access to the data provided by our state (for data quality
review purposes).*

*The ED EDFacts staff members are exemplary in their professional
knowledge, support, and assistance. Provide multiple access licenses to
the EDFacts reporting system on a free or fee basis.*

##### Web site

*I use the ED Web site often to reference material for policy, and
referral for Standard Practice. It would help to have more advanced
features in Ed Search Engine. Being able to provide efficient searches
by exact phrases would make my research less time consuming.*

##### Responsiveness

*Continue to be responsive and give weight to requests from the State
Coordinators\...As you have done in the past.*

##### Reporting Burden

*Extended timelines from when we receive materials and when they have to
be completed and returned to ED. Also, less changes in the reporting
requirements. We are scrambling every year to change our data systems to
meet yet another ED requirement.*

##### 

##### Documentation

*In getting the documentation data documents out earlier.*

*They could streamline the reporting system. More consistent
documentation. The program offices do a better job of sending about any
projects information about what's going on.*

*To ensure States can plan for and fulfill the requirements of EDFacts,
ED must release the file specifications for the entire school year of
files in advance of the school year. In addition, there needs to be
greater collaboration and uniformity in reporting areas that cross
programs (e.g., Highly Qualified, AYP reporting) so States don\'t have
to report the same type of information in numerous ways. That said, the
EDFacts initiative has prompted positive change in many SEAs regarding
data collection, quality, and reporting.*

##### Training

*In the area of EDFacts, we would like to see more one-on-one training.
With limited staff and resources it is hard to send our people to
national trainings/conferences.*

*\
**Funding/Resources***

*Provide more funding for EDFacts to the states.*

*Provide more tools, funds to purchase tools you can\'t provide, and
access to national conferences.*

*Provide staff assistance here at the State level.*

***Consistency***

*Speak one message with one voice, no matter what program area is
speaking. Recognize that just because ED says so, doesn\'t make it so
here in the trenches! Provide funding to allow states to provide AT
LEAST 1.0 FTE of staff dedicated exclusively to EDFacts Reporting
duties.*

##### Other

*I am pretty happy with what they are doing, so keep doing what there
doing.*

*I can\'t think of anything off the top of my head.*

*I only deal with ED on EDEN/EDFacts and have no complaints at this
time.*

*No recommendations. They do a good job.*

*No I cannot think of anything specific at this time.*

#### OVAE -- Career and Technical Education State Directors

***Q9. Please identify your state's best example of collaboration across
offices that you would offer as a model for ED.***

*We work in teams on most things with representatives from secondary,
postsecondary, data, finance, compliance and instructional programs -
folks all work together on projects so all perspectives are taken into
consideration.*

***Q21. Please describe how ED could better use technology to deliver
its services.***

*Provide electronic application completion option; hold Webinars rather
than conference calls; hold live interactive TAV meetings; provide
Web-based training regarding FAUPL definitions.*

***Q11 (OVAE). What can OVAE do over the next year to meet your state's
technical assistance and program improvement needs?***

*Be clearer up front about interpretations of legal limitations before
field-based work groups spend valuable time developing strategies that
are later determined to be non-compliant. I recognize this requires
getting the OAG to cooperate, but it would be very respectful of
efficient use of limited resources.*

*Be flexible - we are working toward requirements, but it will take a
while to get there.*

*Continue collaboration with NASDCTEC and NAPE, and continue use of DTI,
MPR, and AED as technical assistance providers.*

*Continue to provide opportunities for one-on-one TA visits and
conference calls.*

*Continue to support state implementation of Perkins IV - with
flexibility yet consistency as appropriate for individual states.*

*Guidance on Technical Skill Assessment*

*Improve the CAR reporting system by having us report later in the
school year*

*Just continue to have patience with us as we work through the
transition from Perkins III to Perkins IV, especially in the Technical
Skill Attainment area. With the ability to sanction states
administrative funds if they do not meet Performance Standards and
changing how things are to be measured on those Standards, states must
be given a grace period for the change. The OVAE staff that I have
worked with have been outstanding and very helpful. I always feel they
are there to help!*

*Maintain consistency in staff that are assigned to a state. Provide
second and third lines of support when the primary contact cannot be
reached. Acquiring immediate feedback to urgent issues is usually
impossible.*

*More information involving Programs of Study. Collect and disseminate,
on a regular basis, updates as to how states are doing in their efforts
to create Technical Assessments.*

*Our primary focus of need is in student assessments and in development
of programs of study.*

*Provide clarity on definitions, esp. skill attainment*

*Provide more extensive guidance and support for technical skill
attainment.*

*Share best practices as determined through OVAE visits to states*

*We suggest that monthly (or as needed) videoconferences are scheduled
to keep states informed about key issues, and provide guidance on new
projects and work expectations.*

***Q35. Finally, please describe how ED can improve its service to
you.***

***Funding/Resources***

*Advocate for additional funding for our programs.*

*Please make sure that the Research Center is given proper support for
their projects. They do a great job. The research is very relevant to
what we are doing on a daily basis. Overall, ED is very customer
oriented and is doing an effective job. By way of clarification, I
almost never would give a 10 on this type of survey, so 8 is close to my
highest rating.*

##### Consistency/Accessibility

*As previously stated: States need a Single Point of Contact for their
technical assistance needs. It is usually an ever-evolving list of
staff; always under revision and reorganization. Furthermore, when an
issue or a question requires an immediate response there is no
(understood) means of accelerating an issue. Typically, on a Friday
afternoon, NO ONE is available to answer the phone or respond to an
email at OVAE despite the time differences that states are functioning
within. All this said: CONSISTENCY AND ACCESSIBILITY are the themes
intended to be conveyed.*

*Not be inconsistent with responses. Improve the Website. It is
impossible to find anything you are looking for without spending hours
going through old postings. Categorize things better*

*Please make certain that whenever we call, there is at least one staff
member available to respond so we do not have to leave voice messages
that are returned 3-4 days later. That is too late.*

*Provide consistent information on FAUPL definitions; formalize the
negotiation process, allow for attainable percentage targets.*

##### Reporting Burden

*We need more lead time when decisions are made on collecting data. It
takes a minimum of two years if we are making changes in data collection
at the state level and sometimes it takes even longer at the local
level. In order to be effective at the state and local level, we need
definite decisions and ample time to implement requirements for
standards.*

##### Communication

*Provide greater clarity in the email and paper correspondence; many
times we have to sift through the government jargon to get to the real
information.*

*Some additional clarification under Perkins IV - technical assistance*

*\
**Training***

*Continue to provide \'best practice\' information about secondary and
postsecondary Perkins-funded programs from other states; give us
frequently updated, quality information on \'gold standard\'
certifications; make all workshops and conference sessions available via
Webinars.*

##### Timeliness/Responsiveness

*Probably the only issue to resolve is returning phone or email
questions in a more timely manner.*

*Quicker turnaround response to states*

##### Other

*Develop an alternative to OMB\'s PART system, which is deeply flawed
and useful for very little except political manipulation*

##### 

**OVAE -- Directors of Adult Education and Literacy**

***Q9. Please identify your state's best example of collaboration across
offices that you would offer as a model for ED.***

*Collaboration WIA WITH HIGHER EDUCATION.*

*Implementation of a bridge to employment program between and
cooperation on costs (English literacy funded by ABE and employment
related preparation classes and experiential learning activities funded
by WIA).*

*Shared Youth Vision Project*

*The MN Adult Ed. office is collaborating well with the MN College
office and the MN Labor office on the implementation of a Joyce
Foundation transition to post secondary grant.*

***Q21. Please describe how ED could better use technology to deliver
its services.***

*Make Web site more user-friendly*

*National Directors\' meetings; Webcast of these would allow state
directors to set up appropriate state groups to be involved in relevant
sections of these meetings and to have facilitated discussions at the
state and local level.*

*Teleconferencing to replace annual meetings.*

***Q12 (DAEL). What can DAEL do over the next year to meet your state's
technical assistance and program improvement needs?***

*1. Share sample policies and program descriptions that are contributed
by states around certain high-profile topics e.g. - distance education
policies. 2. Share actual \'findings\' text from federal monitoring
reports - remove the state names from the document first.*

*Become more customer-service oriented. Currently OVAE\'s tenor is
regulatory and punitive.*

*Continue the practice of inviting state ABE directors to help plan the
annual meetings.*

*Continue the training in all areas; share the results from initiatives
that limit participation due to funding, help with training that will be
needed for the new series of the GED test for our professional
developers to teach our teachers like we did before in 2002 in Florida.
Keep up the high quality and standard you have set.*

*Continue to do what they have been doing. Also, try to convene regional
meetings for state directors.*

*Continue to offer online resources, online trainings to assist if
meetings cannot be attended, face-to-face meetings and continued
technical support.*

*Current legislation information about act. Reauthorization, assistance
with performance measures and more advanced knowledge new requirements.*

*Designate a single person to do a single training for all of the states
an all of the regionals, so a regional training would be a good idea.*

*Determine how to provide assistance to small states as well as the
larger states. It seems that the focus is often on the large states.*

*ESL-ABE transition*

*Help with teaching writing.*

*I think that it would be worthwhile for OVAE to construct a model
funding formula for states to consider. I think states would welcome a
general not too complex of a formula to consider and build on based on
each states individual needs\...of course this would be optional.*

*It would be nice to have regional meetings; at which we could discuss
priorities and needs.*

*More assistance in the teaching of math and continued support in
transitioning students from GED to post secondary*

*None at this time*

*Offer at a low cost.*

*Provide funding to states that are participating in national activity
projects as a pilot state.*

*Provide section in Thursday Notes on helpful tips from OVAE staff
regarding elements of accountability or policy issues - updates on
national projects.*

*Provide some guidance on the transition of ESL students to ABE
services*

*Provide state specific results and outcomes in regards to performance
based funding, content standards implementation, transition to
post-secondary, and partnership-building with Title I (one-stops). Has
student performance improved as a result of these efforts?*

*Publish a policy and procedures manual.*

*Standards and curriculum align*

*Technical assistance is great. Wyoming is having trouble with the data
provider and the field understanding what is required.*

*They need to work on the issue of the sixty-hour requirement in the
testing. It shouldn\'t be that high. There are policies being put into
place without any national research to verify the validity.*

*To make the Web site more user friendly. To add additional information
and to find information.*

*We\'ll need lots of assistance with a new State Plan whenever
reauthorization occurs.*

***Q35. Finally, please describe how ED can improve its service to
you.***

##### Documentation

*A policy manual would be very helpful.*

*Develop a prototype RFP.*

##### Timeliness/Responsiveness

*A timely response and accurate information. It needs to be
unambiguous.*

*Regular updates on special projects; timely dissemination of resources
to those not in the projects.*

*Respond quickly when we have a question or a concern. Have them do more
research on the western region programs or have them be better informed
as to what the adult education needs are.*

##### 

##### Meetings/Training

*It would be nice to hold more Webinars and not have to travel as often
to face-to-face meetings. It is hard to get out of state for the
meetings but I need the information.*

*More facilitated opportunities to communicate with other state
directors would be helpful.*

*Provide some training through distance learning formats since travel is
being curtailed due to budget concerns and reduction of state funding.*

*We do some more work on distant learning. More effort in distant
learning.*

##### Communication

*Proactively issue policy briefs to provide further clarification to
areas of the legislation where there are frequently asked questions.*

*Work as closely as possible with the National Council of State
Directors of Adult Education, our advocacy group.*

##### Consistency

*Provide consistency and clarity in definitions and expectations of
standardization between states regarding the interpretation/clarity of
definitions and expectations.*

##### Other

*Be an advocate for the re-authorization of WIA to be more amenable to
the current societal needs.*

*ED has provided satisfactory services, and is prompt in responding to
questions and/or concerns. Sometimes the responses are not we expect to
hear as ED is far behind with the political arrangements of the Freely
Associated States with the U.S. Government. ED should also comply with
the AEFLA as they expect grantees to do so.*

*ED is doing the best they can with the resources available. Thank you.*

*Just continue doing as they have been doing in the past. They return
phone calls and emails very promptly and I appreciate that. Are always
friendly and appear glad to assist us.*

*Replace the DAEL Director, \[name deleted\]. Replace punitive and
arrogant attitude with one of customer-service.*

*Services at all levels have been very satisfactory in my 6-month tenure
as a state director.*

###### APPENDIX E: Explanation of Significant Difference Scores

###### 

The charts depicted throughout this report compare 2008 to 2007 scores
and note significant differences. The following provides some background
on how CFI calculates and reports significant differences.

Whether a significant difference exists between two scores (mean scores
reported on a 0 to 100 scale) depends on the sample size, the standard
deviation and the level of significance selected. CFI employed a 90%
level of confidence to check for significant difference on all
questions. This is the standard level used in most of our studies.
However, standard deviation and sample size vary from question to
question. Therefore, some questions may show a small difference in
scores as being significant, while others show a much larger difference
not being significantly different.

In CFI's studies standard deviation, which is a measure of how dispersed
scores are around the mean, typically ranges from 15 to 30 points for
any given question as reported on a 0 to 100 scale. A higher standard
deviation results in a larger confidence interval around a score, so a
larger difference in scores would be required to be significant. To
further illustrate how the dispersion of scores affects significance
testing between two sets of scores, two scenarios are provided. *Assume
350 responses were collected in both year one and year two and a 90%
level of confidence is used. In the first instance, the standard
deviation is 15 points in both years, so scores were fairly uniform
without much dispersion around the mean. In this case, a difference in
scores between years one and two of less than 2 points would be
significant. However, if the standard deviation were 30 points instead
of 15 in both years, so scores were not as uniform and much more
dispersed around the mean, nearly a four-point (3.7) difference in
scores between years one and two would be necessary to be significant.*

With respect to sample size, larger sample sizes result in smaller
confidence intervals. Thus, larger sample sizes require smaller
differences in score to be significant. At a program-level, State Title
I, Title III State Directors, State Education Technology Directors,
Directors of Adult Education and Literacy, and Career and Technical
Education State Directors had more responses and larger sample sizes.
Conversely, Chief State School Officers, State Title V, Part A
Directors, and Lead Agency Early Intervention Directors had lower
response rates and smaller sample sizes.