U.S. DEPARTMENT OF EDUCATION

The U.S.-China E-Language Project: A Study of 

a Gaming Approach to English Language 

Learning for Middle School Students

The U.S.-China E-Language Project:

A Study of a Gaming Approach to 

English Language Learning for Middle 

School Students

Prepared by:

Patricia J. Green

Mandy Sha

Lu Liu

RTI International

Prepared for:

U.S. Department of Education

Office of Planning, Evaluation and Policy Development

Policy and Program Studies Service

2011
A Study of a Gaming Approach to English Language Learning for Middle School Students iv

This report was prepared for the U.S. Department of Education under Contract Number ED-04-CO0036/0002. Brian Fu served as the contracting officer‘s representative for Task Order 8, Evaluation of the 

E-Language Learning System Named: Open Language Learning Initiative for English (OLLI English). The 

views expressed herein do not necessarily represent the positions or policies of the Department of 

Education. No official endorsement by the U.S. Department of Education is intended or should be 

inferred.

U.S. Department of Education

Arne Duncan

Secretary

Office of Planning, Evaluation and Policy Development

Carmel Martin

Assistant Secretary

Policy and Program Studies Service

Stuart Kerachsky

Director

May 2011

This report is in the public domain. Authorization to reproduce the report in whole or in part is granted. 

While permission to reprint this publication is not necessary, the suggested citation is: U.S. Department of 

Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service, 

The U.S.-China E-Language Project: A Study of a Gaming Approach to English Language Learning for 

Middle School Students. Washington, D.C., 2011.

This report is also available on the Department‘s website at

http://www.ed.gov/about/offices/list/opepd/ppss/reports.html.

On request, this publication is available in alternative formats, such as Braille, large print, 

audiotape, or computer diskette. For more information, please contact the Department‘s 

Alternate Format Center at 202-260-0852 or 202-260-0818.
A Study of a Gaming Approach to English Language Learning for Middle School Students iii

Contents

Exhibits and Tables....................................................................................................................... v

Acknowledgments....................................................................................................................... vii

Executive Summary..................................................................................................................... ix

Chapter 1. Purpose and Background ......................................................................................... 1

1.1 Purpose..............................................................................................................................1

1.2 Background.......................................................................................................................2

Chapter 2. Description of the Project and Early Development................................................ 5

2.1 Overall Structure of the Project ........................................................................................5

2.2 Initial Planning..................................................................................................................6

2.3 Overview of The Forgotten World ...................................................................................6

2.4 Pilot Study.........................................................................................................................8

Chapter 3. Study Design and Methodology.............................................................................. 13

3.1 Research Questions.........................................................................................................13

3.2 Study Design...................................................................................................................13

3.3 Data Collection ...............................................................................................................15

3.4 Data Processing...............................................................................................................16

3.5 Data Analysis and Statistical Methods ...........................................................................17

Chapter 4. Implementation........................................................................................................ 19

4.1 Teacher Training.............................................................................................................19

4.2 School and Student Participation....................................................................................22

4.3 Implementation of The Forgotten World........................................................................25

4.4 Teachers‘ Classroom Use of The Forgotten World ........................................................26

Chapter 5. Results....................................................................................................................... 29

5.1 Student Results Based on the Assessments and Questionnaires.....................................29

5.2 Teacher Attitudes and Experiences.................................................................................33

Chapter 6. Summary and Implications .................................................................................... 39

6.1 Summary of Findings......................................................................................................39

6.2 Implications for the Future..............................................................................................39

References ................................................................................................................................... 43

Appendix A. Open Language Learning Initiative Evaluation Planning Group Meeting

Summary............................................................................................................. A-1

Appendix B. The Forgotten World Quick Reference Guide.....................................................B-1

Appendix C. English Proficiency Tests Under Consideration, October 2008 ..........................C-1
Contents 

A Study of a Gaming Approach to English Language Learning for Middle School Students iv

Appendix D. Plan for Pilot Test of The Forgotten World in Gansu Province, April–

June 2009 ............................................................................................................ D-1

Appendix E. Evaluation Plan Update, July 2009 (Revised) .....................................................E-1

Appendix F. Data Collection Procedures and Materials: 2009 Pre-Test Test 

Administrator Manual, Attached Informed Consent Forms, and 

Questionnaires......................................................................................................F-1

Appendix G: Data Collection Procedures and Materials: 2010 Post-Test Test 

Administrator Manual, Attached Informed Consent Forms, and 

Questionnaires..................................................................................................... G-1

Appendix H. Supplementary Tables......................................................................................... H-1
A Study of a Gaming Approach to English Language Learning for Middle School Students v

Exhibits and Tables

Exhibit 1. Organizations Involved in the U.S.-China E-Language Project and 

Their Roles.......................................................................................................... 5

Exhibit 2. Schedule of Study Activities............................................................................. 11

Exhibit 3. Characteristics of Participating Middle Schools............................................... 14

Exhibit 4. Agenda for Teacher Training............................................................................ 20

Exhibit 5. Participants‘ Ratings of Teacher Training Workshop ...................................... 21

Exhibit 6. Training Participants‘ Self-reported Use of the Internet................................... 22

Exhibit 7. School and Student Participation...................................................................... 23

Exhibit 8. Characteristics of Students................................................................................ 24

Exhibit 9. Characteristics of Teachers............................................................................... 25

Exhibit 10. Reported Use of The Forgotten World by Students and Teachers ................... 26

Exhibit 11. Mean Performance on Assessments.................................................................. 29

Exhibit 12. Performance on Assessments, by Use of The Forgotten World ....................... 30

Exhibit 13. Importance of and Interest in Learning English ............................................... 31

Exhibit 14. Percentage of Students Who Reported That Motivation to Learn 

English Improved and That Learning English Is Fun....................................... 31

Exhibit 15. Student Experience With The Forgotten World ............................................... 32

Exhibit 16. Teacher Experience With The Forgotten World............................................... 34

Exhibit 17. Teacher Comments About Why They Are Likely or Not Likely to Use 

The Forgotten World in the Future ................................................................... 35

Exhibit 18. Teacher Suggestions About Improving The Forgotten World ......................... 36

Exhibit 19. Teacher Experience With The Forgotten World............................................... 37

Exhibit 20. Teachers‘ Comments on How This Approach to Language Changed 

the Way They Think About Teaching .............................................................. 38

Exhibit D-1 Evaluation Questions for Gansu Pilot.............................................. D-6

Exhibit D-2 Test scores for IPT and MAC from Gansu December Pretest.................. D-7

Exhibit D-3 Prerequisite Vocabulary Words Presented in Textbooks by Grade Level.... D-8

Exhibit D-4 Matched Vocabulary Count by TFW Episode and Textbook Grade Levels.. D-9

Exhibit E-1 IPT 2004 Reading Test Average Raw Score...................................... E-4

Exhibit E-2 MAC II Listening Test Average Raw Score....................................... E-5

Exhibit E-3 IPT 2004 Writing Test Average Raw Score....................................... E-5

Exhibit E-4 Reading and Listening Tests Combined Average Raw Score.................. E-5

Exhibit E-5 Reading and Listening Combined Raw Score by School Score of EighthGraders................................................................................. E-6

Exhibit E-6 Correlations Between Tests......................................................... E-6
Contents 

A Study of a Gaming Approach to English Language Learning for Middle School Students vi

Table H-1 Hierarchical Linear Models to Confirm Significant Findings for Test Score

Gains Among Low-performing Students and Improved Motivation.......... H-3

Table H-2 Mean and Design-corrected Standard Errors for Motivation, for Treatment

and Comparison Groups............................................................. H-4
A Study of a Gaming Approach to English Language Learning for Middle School Students vii

Acknowledgments

This study is the result of a collaborative partnership of people and organizations in 

China and the United States. The authors would like to thank Adriana de Kanter and Brian Fu of 

the Policy and Program Studies Service at the U.S. Department of Education. Their 

determination to see this project through to fruition, through unexpected twists and turns, 

provided the solid foundation on which the project was built.

The Ministry of Education in China supported the study and provided the team with 

opportunities to work in middle schools for preliminary testing and to conduct this evaluation 

project. Education officials at the ministry—including Zhang Xiuqin, Fang Jun, Wei Liqing, Cen 

Jianjun, Yang Jun, Jing Wei—paved the way for the project by identifying language experts and 

consultants and by identifying sites to participate in the project. Yang Jun participated in the 

teacher training workshop and conveyed the ministry‘s enthusiasm for the project to provincial 

officials, headmasters, and teachers.

Zheng Dawei and Zhang Xuemei of the National Center for Education Technology 

facilitated the project team‘s efforts in China, arranging introductions, attending initial visits, and 

linking the team to the staff of the Provincial Center for Education Technology (PCET) in Gansu 

Province. Yang Xiaojian, the director of PCET, and Cui Liang, also of PCET, facilitated all 

contacts with the schools, assisted in arranging data collection and site visits, and served as 

intermediaries between the schools and the project team. The implementation of the project 

would have been impossible without the help of Gang Liu of the City Center for Education 

Technology, who worked tirelessly to troubleshoot technological problems and provide workable 

solutions. 

The importance of teachers and teacher development in implementing a novel approach 

to language learning was evident throughout the project. Professor Zhang Lianzhong of the 

National Center for Educational Research provided guidance to the project team from the very 

early days of development and remained committed throughout the project. The project was also 

fortunate to involve Professor Wu Heping of Northwest Normal University and Zhang Xiaoling 

of the district education institute in providing ongoing assistance to teachers participating in the 

project, through periodic meetings and ongoing online communication.

This project would not have been possible without the support of the William and Flora 

Hewlett Foundation, which supported the development of The Forgotten World. Under the 

leadership of Phoenix Wang and later Barbara Chow, the ideas for this program were translated 

into reality. The initial grant for the development of The Forgotten World was made to Coastline 

Community College. Dan Jones was the project director; he worked with staff members Jin 

Fang, Michelle Nieman, Toan Tran, and Meg Yanalunas to create The Forgotten World. Alex 

Chisholm, one of the initial advisors on the project, joined the software development team along 

with colleagues at Learning Games Network, and he led the training and implementation in 

China. Sherri Wright of Learning Games Network worked with colleagues at Coastline 

Community College to develop the teacher manual and supporting materials. Damon Anderson 

the U.S. Department of State‘s Beijing-based Regional English Language Officer provided

advisory services.
A Study of a Gaming Approach to English Language Learning for Middle School Students viii

Finally, the important contribution of Professor Zhu Yiming of East China Normal 

University must be acknowledged. Dr. Zhu‘s partnership with RTI International in conducting 

the evaluation was instrumental to its success. The collaborative relationships that he established 

with teachers, schools, and provincial and national officials enabled the evaluation activities to 

proceed smoothly. 
A Study of a Gaming Approach to English Language Learning for Middle School Students ix

Executive Summary

Purpose and Description

In 2001, the U.S. Department of Education and the Ministry of Education in China 

entered into a bilateral partnership to develop a technology-driven approach to foreign language 

learning that integrated gaming, immersion, voice recognition, problem-based learning tasks, and 

other features that made it a significant research and development pilot project for study.

The purpose of this report is to describe the evaluation of a key outcome of this bilateral 

partnership, The Forgotten World. This application, which was developed by Coastline 

Community College and the Learning Games Network through a grant from the William and 

Flora Hewlett Foundation, uses the integrated approach set forth in the U.S.-China bilateral 

agreement. This program was implemented as a supplementary activity in middle school 

classrooms in western China to teach the English language and American culture to eighth-grade 

students.

The evaluation was conducted in five treatment schools and five comparison schools

during the 2009–10 school year and included approximately 3,500 students. The ten schools 

were identified by provincial authorities and then randomly selected for either the treatment or 

comparison groups. All eighth-grade classrooms were asked to participate; in two schools all 

classes participated, but in the other three schools only about half of the classes participated.

Thus, there may be some selection bias in the sample. However, eighth-grade students in the 

treatment and comparison schools on average started the school year with similar initial levels of 

English-language performance.

Teachers in the treatment schools attended a three-day training workshop in August 2009 

to become familiar with The Forgotten World and with the approach to teaching and learning on 

which the application was based. At the end of the training, each school developed a plan for 

implementing the program in the school. Schools were asked to begin using the program in 

supplementary classes in late September 2009 and to allow students access to The Forgotten 

World in school computer labs for at least one class period each week. 

Reading and listening tests were administered at the beginning and end of the year to

eighth-grade students in schools participating in the study. In addition, questionnaires were 

completed by teachers and students in the schools participating in the study. Teachers were also 

asked to complete logs describing each class session in which the program was used. 

Several implementation issues encountered during the study significantly reduced the 

amount of time students could spend on The Forgotten World. The expected level of 

implementation was not realized until halfway through the study, after information technology

infrastructure improvements were made at treatment schools. Even then, only about 70 percent 

of students reported that they used The Forgotten World at least once a week. This partial 

implementation during one semester represents a fairly limited treatment in comparison with 

other education studies of this type.
Executive Summary 

x A Study of a Gaming Approach to English Language Learning for Middle School Students

Findings

Students in treatment schools with relatively low levels of initial proficiency, who scored 

in the bottom half of all students on the fall tests, evidenced larger gains in reading and listening 

than similar students in comparison schools. There were no differences in performance between 

treatment and comparison groups among students with higher levels of initial proficiency (those 

scoring in the upper half of all students on the fall test). Overall, students in treatment and 

comparison schools performed similarly on the proficiency tests administered at the beginning 

and end of the school year.

Students in the treatment group also reported higher levels of motivation to learn English

than students in the comparison schools. At the end of the year, 51 percent of students in the 

treatment schools reported that their motivation to learn English had improved, compared with

40 percent of students in the comparison schools. Teachers in treatment schools also reported 

that students‘ motivation improved. This is an important finding because motivation to learn a 

foreign language has been identified by Chinese language experts as a problem in Chinese 

schools and as a barrier to learning.

Almost all of the teachers in the treatment schools (95 percent) who participated in the 

project reported that their use of The Forgotten World changed the way they think about 

teaching. The approach to language teaching and learning that forms the basis for The Forgotten 

World is learner-centered and interactive and presents students with problem-based tasks and 

immediate feedback. The teachers appreciated the fact that the program provided students with 

fun opportunities for practicing listening and speaking skills, with exposure to native English 

speakers. However, some teachers expressed concern that their primary responsibility was to 

prepare students for the English-language portion of the high school entrance exam that students 

take in ninth grade and that they did not feel that use of The Forgotten World would improve 

student performance on the exam.

The statistically significant positive results of using The Forgotten World for the lower 

performing students along with the positive effects on student motivation is encouraging for an 

intervention that was only partially implemented. Further research could be conducted to help 

understand why the program produced these benefits, how to overcome implementation 

problems in schools lacking sophisticated computer technology, and why The Forgotten World

did not produce positive benefits for the initially higher performers.
A Study of a Gaming Approach to English Language Learning for Middle School Students 1

Chapter 1. Purpose and Background 

1.1 Purpose

The Policy and Program Studies Service (PPSS), within the Office of Planning, 

Evaluation and Policy Development (OPEPD), conducted a study of the E-language Learning

System1named Open Language Learning Initiative for English (OLLI English).

2

The program

developed under this initiative, The Forgotten World, 3 was designed to help students in middle 

schools in China learn English. This study was conducted in Chinese middle schools for two 

semesters from September 2009 to May 2010.

The original online language learning project began as a joint partnership between the 

U.S. Department of Education (Department) and the Ministry of Education of the People‘s 

Republic of China to create a highly innovative research and development (R&D) product to test 

the effectiveness of using online technology to deliver foreign language instruction within a realworld environment, consistent with integrating emerging theories of effective language learning

practices. The original design called for the Department to develop 36 online lessons to teach the 

English language (reading, writing, speaking, and listening) and American culture to students in 

the People‘s Republic of China.

The Department did not complete building the online language learning system. Instead, 

the Department made existing materials that had been developed using Department funds,

including scripts and source code, available to the public via the World Wide Web.

Subsequently, the William and Flora Hewlett Foundation committed to building upon the 

Department‘s work. In February 2007, the Hewlett Foundation awarded a grant to Coastline 

Community College to build a complete application to teach the English language and American 

culture to students in China. The application, The Forgotten World, is part of OLLI English, a 

program sponsored by the Hewlett Foundation to develop open education resources for English 

language learning.

This report presents information about the implementation of The Forgotten World

program in middle schools in western China during the 2009–10 school year. The original 

purpose of the study was to evaluate the efficacy of the language learning approach on 

motivation and student language proficiency (outcomes), believing that proof of concept of the 

approach would be applicable to students learning a second language in the United States.

Because of difficulties in implementation, the efficacy of the program could not be thoroughly

evaluated. Nevertheless, the report discusses challenges to implementation and user experiences 

with the language approach.

 

1 E-Language Learning System (ELLS) is the name of the joint project that was launched as a result of the 

memorandum of understanding between the U.S. and China signed in 2002

2 The Open Language Learning Initiative (OLLI) is the name given to the William and Flora Hewlett 

Foundation initiative to provide open source high quality language courseware at little or no cost to language 

learners worldwide

3 The Forgotten World is the name of the English language learning program designed for Chinese middleschool students that was developed by Coastline Community College and Learning Games Network with funding 

from the Hewlett Foundation
Chapter 1. Purpose and Background

2 A Study of a Gaming Approach to English Language Learning for Middle School Students

1.2 Background

During President George W. Bush‘s trip to Shanghai in October 2001 for the Asia-Pacific 

Economic Cooperation (APEC) Leader‘s Meeting, the White House announced a series of 

e-learning initiatives. One of these initiatives involved the development of an online language

learning system applying advanced Web-based language instruction technologies.

One year later, the U.S. secretary of education and the Chinese minister of education 

signed a bilateral memorandum of understanding for a joint research and development project, 

known as the U.S.-China E-Language Project. The Chinese minister noted that the project was 

the largest government-to-government education initiative since the normalization of relations 

between the two countries in 1979.

The crux of the agreement was that each country, China and the United States, would 

develop online language learning programs for middle school students in the other country. The 

project aimed to create an innovative online language learning R&D program in English and 

Chinese that was

 targeted at middle school American and Chinese students;

 constructed as a joint collegial effort by top experts from China and the United States

in the areas of language instruction, technology, evaluation, and dissemination;

 delivered to students in China and the United States via the Internet; and 

 presented to students in an interactive gaming environment with speech recognition.

The design parameters used to guide development of the programs included a number of 

innovative features that made it a significant R&D pilot for evaluation. The Chinese and 

English programs were to

 incorporate the research-based benefits of teaching a foreign language within a realworld context to enhance the learning and transferability of communicative language 

skills;

 use a model-practice-apply framework consistent with the theory of learning a foreign 

language by hearing it, practicing it, and using it;

 provide for extensive practice essential to developing language fluency;

 be learner-centered in that it builds systematically on prior vocabulary, word 

difficulty, and communicative complexity;

 encourage language learning through creating a gaming environment within a realworld context that rewards students for making language progress;

 present cultural cues as part of language instruction along with expanded 

opportunities to learn about particular features of the culture and way of life;

 adopt a learning management system that integrates problem-based learning tasks 

throughout lessons to provide the learner with continuous feedback on progress and 

for instruction to reinforce weaknesses; and

 include voice recognition to authenticate and provide feedback on the learner‘s 

speech reproduction capability.
Chapter 1. Purpose and Background

A Study of a Gaming Approach to English Language Learning for Middle School Students 3

In addition to teaching language, the online language-learning systems were to promote the 

learning of each country‘s culture, incorporating real-world illustrations that depict current 

lifestyles, including living habits, transportation, buildings, dress, and music. 

China completed development of an online program for U.S. middle school students, 

named Chengo, in 2004. Access to that program can be found at www.elanguage.cn. Chengo

was evaluated through a series of focus groups conducted by the Department and documented in 

a previous report.

The Department did not complete development of the online language-learning system.

Instead, it decided to turn responsibility for development of the system over to the private sector.

The Department released the following system components into the public domain by publishing 

them on the Department‘s Web site:

 the source code for the technology platform, minus proprietary software components 

for voice recognition and learning management;

 thirty-six English-language scripts written by English as a second language (ESL) 

experts and aligned with English language learning content standards;

 a review of the system by the contractor; and 

 findings from the market research request for information.

When the Department released these materials into the public domain, it indicated that it 

would evaluate completed systems that carried out the original vision for the system. In February 

2007, the William and Flora Hewlett Foundation awarded a grant to Coastline Community 

College to build a completed system to teach the English language and American culture to 

students in China. In May 2007, the Department and the Hewlett Foundation convened a meeting 

of experts to discuss the development of the program as well as plans for an evaluation.

This report is focused on the study of the English language and culture program

developed for Chinese middle schools, with funding from the Hewlett Foundation, and does not 

include information on the Chinese language program developed for U.S. middle schools.

A Study of a Gaming Approach to English Language Learning for Middle School Students 5

Chapter 2. Description of the Project and Early 

Chapter 2. Development 

2.1 Overall Structure of the Project

One of the unique features of this study is that it is not an evaluation of a preexisting 

program. Rather, it is a study of the implementation of a new program that was sponsored and 

developed by a number of partners. Exhibit 1 presents information on the major partners 

involved in the overall project in the United States and China. The Department entered into the 

agreement with the Chinese Ministry of Education to develop and study a new language learning

approach. As stated earlier, the Department released the specifications for the program and 

agreed to fund an evaluation of the program meeting the major design requirements. Thus, when 

the Hewlett Foundation awarded a grant to Coastline Community College (and, later, Learning 

Games Network) to develop the program, the Department contracted with RTI International (and 

its partner in China, East China Normal University [ECNU]), to evaluate implementation of the 

approach (via online technology) in China. The Chinese Ministry of Education tasked the 

National Center for Education Technology to facilitate the U.S. team‘s work in China by 

identifying sites for the study.

Exhibit 1. Organizations Involved in the U.S.-China E-Language Project and Their Roles

Organization Role

U.S. Department of Education Collaborated with the Ministry of Education on the overall design requirements

of the U.S.-China E-Language Learning System. Funded contract to RTI 

International to study implementation.

Ministry of Education of the 

People’s Republic of China 

Collaborated with the U.S. Department of Education on the overall design of 

U.S.-China E-Language Learning System. Assigned responsibility for 

facilitating the project and identifying province for implementation to the 

National Center for Educational Technology.

William and Flora Hewlett 

Foundation

Funded the development of The Forgotten World as part of the Open Language 

Learning Initiative.

Coastline Community College, 

Learning Games Network

Funded by a grant from the Hewlett Foundation, developed instructional 

software, The Forgotten World, using a video-gaming approach. Also 

responsible for teacher training and implementation in schools.

RTI International Funded by the U.S. Department of Education to study use of the program in 

Chinese middle schools.

East China Normal University Partnered with RTI to design and conduct a study of the program when 

implemented in Chinese middle schools (subcontractor to RTI).

National Center for Educational

Technology (NCET), 

Provincial Center for 

Education Technology (PCET) 

and the City Center for 

Education Technology (CCET)

NCET was assigned by Ministry of Education to facilitate project activities in 

China and identify a province to implement The Forgotten World. PCET 

identified schools, served as primary contact with schools regarding 

participation, and was liaison with schools for arranging dates for teacher 

training and data collection. CCET provided hardware support and 

troubleshooting.

Northwest Normal University

(China)

Partnered with Learning Games Network to provide support for initial teacher 

training and ongoing support for teachers throughout the year. 

Exhibit reads: Multiple organizations and agencies in the United States and China partnered to develop, implement, and evaluate 

new software designed to help Chinese middle school students learn English.

Source: The U.S.-China E-Language Project.
Chapter 2. Description of the Project and Early Development

6 A Study of a Gaming Approach to English Language Learning for Middle School Students

The software development team, led by Coastline Community College and Learning 

Games Network, was responsible for implementation and teacher training and support. Locally,

they partnered with faculty at Northwest Normal University for teacher support and the City 

Center for Education Technology (CCET) for technology support. Thus references in this report 

to the software and implementation team refer to Coastline Community College, Learning 

Games Network, Northwest Normal University, and CCET, all funded by the Hewlett

Foundation. References to the evaluation team refer to RTI International and ECNU. Because 

the Department funded only the evaluation effort, that is the primary focus of this report.

2.2 Initial Planning

An Evaluation Planning Group (EPG) meeting was held in May 2007 in Beijing as part 

of a larger Technical Working Group (TWG) meeting to discuss the development of the program 

and the design of the evaluation. Experts in English language learning, educational evaluation, 

and educational gaming from China and the United States attended the meeting. At the meeting, 

officials from the Chinese Ministry of Education stated that they would like to implement the 

program in the middle schools in the western provinces of China because qualified English 

teachers are less available in those areas than in eastern and central China. At the time of the 

meeting, a Modern Distance Education Project was planned that would equip 35,000 middle 

schools in western China with new computers and Internet access, and the software development 

team was encouraged to design an application that would make full use of this advanced 

technology.

Chinese scholars in the group expressed concern that middle schools might be reluctant 

to devote instructional time to experimental teaching methods like The Forgotten World program 

because of the importance of student performance on high-stakes entrance exams. Students in 

ninth grade must pass entrance exams to be accepted into high school, and English is one of the 

subjects included on the exam. Experts were also skeptical about the amount of time parents 

would allow students to use any type of gaming program at home, even if it was instructional 

gaming. Thus, the group consensus was that the system be designed for supplemental instruction, 

to be used in addition to the regular English classes. Because of the emphasis on high school 

entrance exams in ninth grade, the group suggested that the program be developed for seventhor eighth-grade students and suggested that participating schools be selected from a single 

district to limit variation based on differing curriculum, textbooks, and length of English

language instruction.

The EPG recommended that a random assignment experiment be conducted with 25–

35 treatment schools and an equal number of comparison schools to answer the research 

question: Do students exposed to the English program demonstrate increases in proficiency when 

compared with students without access to the program, given the same curriculum and under 

conditions that allow reasonable access to The Forgotten World? The group suggested that a mix 

of performance measures be used, including existing assessments, performance on district 

assessments, and specially designed assessments, if possible. Measurement of student motivation 

and attitudes was also recommended. Minutes of the EPG meeting are provided in Appendix A.

2.3 Overview of The Forgotten World

The Forgotten World was developed by Coastline Community College (a major provider 

of higher education distance learning courses) and the Learning Games Network with funding 
Chapter 2. Description of the Project and Early Development

A Study of a Gaming Approach to English Language Learning for Middle School Students 7

provided by the William and Flora Hewlett Foundation. Following the guidelines initially set 

forth by the Department (described in Section 1.2) and the recommendations of the TWG, the 

program was designed for the target population of seventh- and eighth-grade students with prior 

exposure to English but limited vocabulary. The initial grant funded the development of the first 

12 episodes of a potential 36-episode series. The program was designed to be delivered to 

schools via the Internet for use in school computer labs. Students could also access the program 

from home computers using their assigned login credentials. A detailed description of the 

program can be found in Appendix B, a Quick Reference Guide to The Forgotten World.

One unique feature of the program is that it has been designed as an open education 

resource that can be freely used, adapted, and repurposed by others, if desired. The underlying 

software is structured so that it can be easily modified for English language learners whose 

native language is not Chinese. For example, the components of the program (the story and 

activities) remain the same, but the dictionary can be changed from English-Mandarin to 

English-Spanish, enabling Spanish-speaking students learning English to use the program. The 

combination of open source code, video gaming, and easily modifiable dictionaries has the 

potential for creating derivative products for a wide range of audiences.

The Forgotten World is presented in a Web browser. Learners are required to complete a 

simple registration process before becoming immersed in a comic-book adventure where the 

learner is one of the characters. A basic vocabulary assessment is woven into the first episode.

The assessment is presented as a basic puzzle game in which the application gathers information 

about the learner‘s proficiency of a vocabulary set drawn from approximately 500 words 

representing the baseline awareness expectation of the low-novice learner for whom the program

has been designed.4The story then continues to unfold through an interactive graphic-novel 

format across 12 individual episodes, including the following:

 Comic book story panels—Between 15 and 20 screens per episode communicate 

the story through full-color art, interactive dialogue bubbles, and audio clips; the story 

panels provide the learner with his or her first exposure to the vocabulary words and 

pronunciations.

 Activities—Clickable activities and games enable the learner to interact with specific 

screens to support reading and listening skills development.

 Games—Nested within each episode are two Flash-based games designed to support 

reading, listening, and speaking skills development. The games use vocabulary and 

simple phrases introduced in that or previous episodes as the ―learning objects‖ to 

support learning goals; after a series of three to four episodes, games provide an 

opportunity to create formative evaluations from larger sets of content.

 Resources—A personal digital assistant (PDA) with scoring and progress reporting 

function, episode log, dictionary, and games is displayed on screen; the PDA is 

designed to enable the learner to break out of the linear graphic-novel presentation 

and access previously viewed episodes and previously played games.

 

4 The American Council on the Teaching of Foreign Languages (ACTFL) has developed a set of proficiency levels 

that include novice, intermediate, advanced, and superior. Within the novice category, the low-novice reading level 

is described as ―able occasionally to identify isolated words and/or major phrases when strongly supported by 

context‖ (https://www.languagetesting.com/assessments_academic.cfm#).
Chapter 2. Description of the Project and Early Development

8 A Study of a Gaming Approach to English Language Learning for Middle School Students

 Progress Reports—Scores are provided to the learner through the onscreen PDA 

interface. Teachers and parents have access to separate evaluation reports that convert

the learner‘s score to information about his or her achievement of specific learning 

goals. 

The learner has the freedom to navigate the screens within each episode, moving forward 

and backward at his or her own pace. Within the context of the narrative, games are launched to 

create opportunities for the learner to rehearse and apply vocabulary he or she has been exposed 

to or mastered in previous episodes plus newly introduced words and phrases.

Because the level of each vocabulary word is coded within the application and presented 

at a specific time in an episode, the application adapts to the learner‘s progress and performance. 

Following the learner‘s completion of the preliminary evaluation, the application captures a set 

of variables that define the user‘s starting level. From there, words and phrases that are 

appropriate to that level of performance as part of each episode populate the games in that 

episode. Depending on the learner‘s achievement and scoring, the system can either raise the bar 

and introduce more difficult words and phrases or remediate by repopulating the games with the 

learning objects to reinforce the basics. The application is designed both to move the learner 

within a specific level across episodes so he or she can complete the application and to advance 

the learner‘s performance through increased difficulty as he or she demonstrates proficiency and 

mastery of learning goals.

2.4 Pilot Study

Instrument identification.The EPG identified several potential tests that might be used 

in the evaluation. The list below contains the major tests reviewed for possible use:

 The Test of English for International Communication (TOEIC) Bridge is a product of 

ETS and was developed to measure emerging English language competencies among 

beginning and intermediate learners. The paper-and-pencil tests take approximately 60 

minutes each. The tests were primarily designed for business use and contain many 

practical items.

 Maculaitis Assessment of Competencies II, Test of English Language Proficiency 

(Touchstone Applied Science Associates), MAC II, is used by some states to measure 

English language proficiency among ESL students.

 IDEA IPT (Ballard and Tighe) is another test used by some states to measure English 

language proficiency among ESL students.

 Stanford English Language Proficiency (SELP) Test, published by Harcourt Assessment, 

Inc., is another exam used by states to measure English language proficiency among ESL 

students.

 Cambridge Young Learners Exam (YLE) is a criterion-referenced exam designed for use 

with elementary school aged children. A member of the EPG stated that this exam is used 

and respected in China.
Chapter 2. Description of the Project and Early Development

A Study of a Gaming Approach to English Language Learning for Middle School Students 9

 ACTFL Oral Proficiency Interview (OPI). This is a new proficiency exam developed by 

the American Council on the Teaching of Foreign Languages (ACTFL) for 

administration via phone or Internet. The Internet connectivity of participating schools 

may be an issue for this methodology.

These assessments were reviewed to determine the level of English assessed, skill areas assessed, 

potential to measure growth, time required for administration, special requirements for 

administration, appropriateness for age group, cultural appropriateness, cost, option to adapt the 

instrument, and vocabulary overlap with the English textbook used in the pilot school and The 

Forgotten World.

Based on the preliminary review, the TOEIC Bridge was rejected as being inappropriate 

for this age group. The publishers of the SELP were unwilling to work with organizations not 

affiliated with school districts, and the Cambridge tests were criterion refererenced and not 

designed to yield gain scores. The ACTFL test was considered, but because of concern about the 

fact the test was still in development, it was not selected for this study. The assessments that 

were chosen for testing in Gansu included the IPT reading, writing, and combined 

listening/speaking tests, and the MAC II reading, writing, and listening tests (see Appendix C for 

more detail).

December advance visit. In November 2008, the Ministry of Education identified a 

province, the western province of Gansu, and two pilot schools in that province willing to 

participate in the project. A meeting in the provincial capital was scheduled for December 9–11, 

2008. Representatives from the U.S. Department of Education, Coastline Community College, 

Learning Games Network, RTI International, and East China Normal University met with 

officials from NCET, the provincial Education Bureau, PCET, and principals and teachers of the 

two schools that volunteered to participate.

The purpose of the visit was to explain the overall project, provide demonstrations to 

teachers, and gather feedback from the schools. Arrangements were also made to administer 

English language tests to a handful of students to provide preliminary information on the level of 

students‘ English and the appropriateness of tests identified as potentially suitable for the study.

Upon returning to Beijing en route to the United States, the study team met at the 

Ministry of Education to discuss arrangements for the pilot study. The study team expressed 

some concern about the level of technology observed in the labs in the two schools. The director 

of NCET noted that funding provided by NCET to provinces and schools was not restricted to 

the purchase of certain types of hardware configurations. Localities could choose how to spend 

the money they were allocated and thus lab configurations varied across schools. And given the 

recent earthquake in Sichuan, monies were prioritized for the schools in the affected areas, and 

additional funds to upgrade computer labs for this study were unavailable.

A decision was made to conduct the pilot and main study in Gansu. The level of English 

instruction in the middle school was consistent with the program, and provincial officials, 

headmasters, and teachers were enthusiastic and willing to participate. The software 

development and implementation team decided that it could provide limited upgrades, such as 

increased memory for machines and headsets, that would enable the labs to be used for the study.

Following the visit, extensive analysis was conducted comparing the vocabulary used in 

the tests with the vocabulary in the textbooks used by seventh- and eighth-grade students, Go 
Chapter 2. Description of the Project and Early Development

10 A Study of a Gaming Approach to English Language Learning for Middle School Students

For It. In addition, the appropriateness of individual items was examined by identifying items 

that few students answered correctly, and examining whether the vocabulary was covered in the 

textbooks used, or if perhaps some element of the question was culturally inappropriate. The 

results of this analysis suggested that the MAC II listening test should be piloted in April, along 

with the IPT reading and writing tests. More information on these analyses can be found in 

Appendix D. 

April–June pilot. The main pilot activities occurred from April through June 2009; a 

complete list of activities for the pilot and main study is provided in Exhibit 2. In the initial 

statement of work for this study, a yearlong pilot was scheduled for 2008–09. However, given 

delays in the development of the program, the pilot consisted of a three-month implementation of 

the program in two pilot schools. During the April–June pilot, The Forgotten World was 

implemented in seventh- and eighth-grade classrooms in both schools.

The purpose of the pilot data collection in April was to determine:

 whether the software was ready for a full-scale implementation in the fall,

 whether the English language reading, listening, and writing tests identified in 

December were appropriate for use with this population, and

 the suitability of beginning-of-year questionnaires for students and teachers.

Assessment instruments and questionnaires were tested in two pilot schools in the 

province in spring 2009; more than 1,800 students were tested. The IPT Reading and IPT 

Writing assessments were used, along with the Mac II Listening assessment. The IPT Reading 

assessment was used without modification; the other two assessments were adapted based on 

preliminary work in China in December. For the writing test, some of the picture prompts were 

substituted using pictures from other IPT tests that were more culturally appropriate for Chinese 

middle school students. The listening test directions were modified so that the directions were 

delivered to students in Chinese, and the answer form was modified to make it more 

understandable to students. All of the listening instructions were professionally recorded to 

ensure comparability across sites.

One of the major purposes of the pilot was to make sure that the tests were suitable for 

use among this population and that the modifications made to the tests did not alter the 

equivalency of the pre- and post-test forms. The tests were administered at two pilot schools to 

seventh- and eighth-grade students. Because the delays in developing the program meant that a 

full year pilot was impossible, end-of-year seventh-grade students were tested to represent scores 

that might be expected from beginning eighth-grade students. It was not possible to measure 

growth in proficiency for individual students.
Chapter 2. Description of the Project and Early Development

A Study of a Gaming Approach to English Language Learning for Middle School Students 11

Exhibit 2. Schedule of Study Activities

Date Activity Conducted by:

Fall 2008 Identification of potential tests RTI International

December 2008 Meeting with provincial officials and pre-pilot 

testing

U.S. Department of Education, RTI

International, East China Normal 

University (ECNU)

April 2009 1.5 days of training of teachers in pilot 

schools

Coastline Community College and Learning 

Games Network, Professor Zhang 

Lianzhong

April 2009 Pilot of tests and pretest questionnaires, all 

seventh- and eighth-grade students in 

pilot schools; meeting with provincial 

officials to outline requirements for 

2009–10 study

RTI and ECNU train test administrators to 

administer pilot tests and questionnaires 

to seventh- and eighth-grade students in 

two pilot schools (30+ classrooms)

June 2009 Data entry and scoring ECNU; RTI conducted reliability scoring

June 30–July 3, 2009 End-of-pilot posttest questionnaires 

administered in six classrooms; focus 

groups and interviews with subset of 

students, teachers, principal, support 

personnel, and provincial center

ECNU team: Professors Zhu Yiming, Wang 

Jiqing, and Dr. Yang Guangfu of ECNU

June 25, 2009 Schools selected for inclusion in 2009–10 

study

Provincial Center for Education Technology 

(PCET)

August 2–5, 2009 2.5 days of training for teachers; computer 

labs at participating schools visited and 

computers checked

U.S. Department of Education, Ministry of 

Education (People’s Republic of China), 

William and Flora Hewlett Foundation, 

Coastline Community College, Learning 

Games Network, Northwest Normal 

University, RTI, ECNU, invited guest

August–September 

2009

Implementation plans negotiated with 

schools

ECNU

September 24, 2009 Use of The Forgotten World begins in 

classrooms

September 28–

October 13, 2009

(No testing first week 

of October;

national holiday)

Tests and questionnaires administered ECNU and RTI train test administrators and 

administer tests in 10 schools 

(95 eighth-grade classrooms)

October 12–13, 2009 Observation of The Forgotten World classes 

in two schools

RTI

December 2009 Dell agrees to donate a new server for each 

school

January 2010 New servers delivered

January 2010 Site visit ECNU

January–February, 

2010

Schools closed

March 2010 New servers installed and refresher training 

launched

Learning Games Network, PCET, Northwest 

Normal University, ECNU

June 20–26, 2010 End-of-year tests and questionnaires 

administered

ECNU

July 2010 Data entry and scoring ECNU; data quality checks at RTI

Source: The U.S.-China E-Language Project.

Results for each of the major tests are presented in Appendix D. For both the reading and 

listening tests, the two forms (pretest and posttest) were roughly equivalent. There were no 
Chapter 2. Description of the Project and Early Development

12 A Study of a Gaming Approach to English Language Learning for Middle School Students

statistically significant differences in the raw scores for the two reading test forms. A significant 

difference was found for the mean raw scores on the two forms of the listening test; however, 

conversations with the publisher and published manuals show that the equivalence between 

forms is achieved through adjustments in transforming the raw scores into standard scale scores.

Also, comparing results for seventh- and eighth-grade students, eighth-grade students 

consistently performed better than seventh-grade students on both the reading and listening tests, 

and the differences were statistically significant. The writing test showed the weakest results. A 

large number of students left the writing test blank; in focus groups, students reported that they 

were unfamiliar with the test format (using picture prompts) or had little or no experience writing 

English or both. Because of this, and the fact that proficiency in writing was not a goal for the 

e-language approach, a decision was made to drop the writing test from the final battery.

Test results were also validated by comparing them with scores on district tests provided 

by the schools. As expected, students scoring well on district exams performed well on the 

proficiency assessments. The scores on school exams were more correlated with reading and 

writing assessments, and less correlated with the listening assessment. This finding was

consistent with the rationale for the study; teachers and students in rural areas have little

opportunity to interact with native English speakers; consequently, their listening (and speaking) 

skills are frequently weaker than their reading skills.

In June, end-of-year questionnaires for students and teachers were administered and a 

series of focus groups and interviews were conducted with administrators, teachers and students 

to gather feedback on their experiences with The Forgotten World. Results of these interviews 

and focus groups indicated the following:

 Both pilot schools valued the experience and asked to continue using The Forgotten 

World and to be included in the full-scale evaluation.

 Participants complained about the slowness of the software. As a result, the software 

was modified and compressed by the software developers.

 Teachers stated that they needed more training. As a result, a three-day training 

workshop was developed for the full-scale evaluation. Periodic meetings and 

trainings throughout the year for participating teachers were also scheduled for 

teachers participating in the 2009–10 study.

 Teachers indicated that they would like more clarity on goals and expectations. Plans 

for the August training were modified to provide more information for teachers and 

principals, including clear expectations for the use of the program.

 Students expressed concern about the dictionary, and some were concerned that the 

speech used was too fast for some to comprehend. The software development team 

took this matter under consideration, because experts had suggested limiting the 

amount of native language available and using conversational speed consistent with 

that of native speakers.

 When asked about the writing tests, students said that they were unfamiliar with 

writing English and also unfamiliar with picture prompts. As a result, a decision was 

made to drop the writing test from the full-scale study.
A Study of a Gaming Approach to English Language Learning for Middle School Students 13

Chapter 3. Study Design and Methodology

The original intent of this project was to conduct a random assignment experiment to 

evaluate the impact of this new approach to language learning on student proficiency in English.

Numerous factors intervened to make this design unfeasible, primarily the difficulty in 

identifying schools that had computer labs that met the minimum requirements for the study. The 

research questions and study design were modified as a result and are described below.

3.1 Research Questions

In June 2009, the provincial authorities determined that there were not enough schools 

that met the technological requirements for participation in the project. They were able to 

identify 10 schools, with a combined enrollment of approximately 6,000 eighth-grade students,

that met project requirements:

 schools with computer labs capable of running The Forgotten World;

 schools committed to allowing eighth-grade students access to the computer lab at 

least one hour per week;

 schools using Go For It, the English textbook used in the pilot schools;

 schools with teachers able to attend training in early August 2009; and

 schools willing to allow testing in September 2009 and June 2010.

While the number of schools was lower than requested, a decision was made that the 

study would continue and would use the 10 schools that met the requirements (A move to a 

different province would delay the study by one year or more.) An RTI statistician randomly 

assigned five schools to the treatment group and five schools to the comparison group. It was 

acknowledged that this design did not allow the statistical precision required of a hypothesis 

testing study but would be a solid foundation for a hypothesis-generating study. Using a smaller 

set of schools would allow more support for implementation, and retaining some schools for 

comparison would help provide a frame of reference for the revised research questions:

 What challenges do schools face in implementing The Forgotten World?

 Do findings suggest that students who use The Forgotten World frequently improve 

their English proficiency more than students who use The Forgotten World less 

frequently or not at all?

 Does The Forgotten World increase student motivation to learn English?

 What are students‘ and teachers‘ attitudes toward the use of The Forgotten World?

3.2 Study Design

Exhibit 3 provides information on the 10 schools that were identified by provincial 

officials as meeting the criteria for participation. The evaluation team discussed options for the 

assignment of schools and teachers with the Department. Concern about the viral nature of some 

gaming applications and the potential for ―contamination‖ across classes indicated that random 

assignment should occur at the school level, rather than the classroom level. Thus, the schools 
Chapter 3. Study Design and Methodology

14 A Study of a Gaming Approach to English Language Learning for Middle School Students

were randomly divided into two groups by RTI; the schools selected to use The Forgotten World

are highlighted. All eighth-grade classes in the highlighted schools were asked to participate, and 

eighth-grade English teachers were invited to attend training in early August 2009. However, not 

all teachers and classrooms in treatment schools participated in the study, due to the capacity of 

labs, scheduling issues, and the voluntary nature of the study. In two schools all eighth-grade 

classrooms participated, and in the remaining three about half of the eighth-grade classrooms 

participated; thus about 64 percent of classrooms in treatment schools participated (see Section 

4.2 for more information about participating classes). Participating classrooms in the treatment 

schools were identified through implementation plans prepared by the schools at the end of the 

August training and then updated in the fall when implementation began. Classrooms were 

selected for inclusion by the schools based on teachers‘ willingness to participate and scheduling 

of computer labs. Teacher logs were collected monthly from participating classes to confirm 

active participation, and site visits were made to observe classes. No classrooms in treatment 

schools dropped out of the study. All eighth-grade classrooms in comparison schools were 

included in the fall data collection; a random sample of all classrooms was tested at the end of 

the year.

Exhibit 3. Characteristics of Participating Middle Schools

School type Enrollment 

Number of teachers Number of classes Number of 

computer 

rooms

Number of 

computers 

Total for students

English

teachers Total

Grade 

7

Grade 

8

Grade 

9

City 2,300 168 23 40 12 12 16 2 150

City 1,700 130 16 25 8 8 9 4 172

City 1,400 90 15 24 8 8 8 1 80

City 635 66 13 16 6 5 5 2 131

Rural 1,300 126 17 24 8 8 8 2 135

Rural 1,300 36 8 19 8 6 5 1 50

Rural 2,701 117 19 36 13 13 10 2 61

Rural 2,550 114 18 33 12 12 9 2 96

Rural 1,990 94 16 30 12 12 6 1 35

Rural 1,203 52 10 18 8 10 0 2 96

Exhibit reads: Middle schools in China chosen for inclusion in the project have large enrollments and large class sizes.

Source: The U.S.-China E-Language Project.

The middle schools were located in two areas within the province. As shown in Exhibit 3, 

most of the middle schools selected for inclusion in the study enroll large numbers of students.

The smallest school enrolled 645 students; the remaining 9 schools enrolled between 1,200 and 

2,700 middle school students. All schools appeared to have the requisite computer equipment, 

although later it was found that in many schools the Internet connectivity was unstable and the 

average class size for middle school English classes exceeded the capacity of the computer labs.

Instruments tested or developed during the pilot were modified slightly for use in the 

2009–10 study and included the following:

 the IPT 2004 Reading Test—Forms 1C and 1D for use at the beginning and the end 

of the school year (Amori, Dalton, and Tighe 2008);
Chapter 3. Study Design and Methodology

A Study of a Gaming Approach to English Language Learning for Middle School Students 15

 the MAC II Listening Test—Forms A2 and B2 (Maculaitis 2001) for use at the 

beginning and the end of the school year (minor modifications were made to Form 

A2, based on pilot results);

 student and teacher questionnaires for the beginning and the end of the school year; 

and 

 weekly teacher logs.

No speaking test was used because the logistics required to implement one-on-one testing 

in English would have required additional resources. Also, the final version of The Forgotten 

World did not incorporate speech recognition, although students using it were asked to produce 

and practice speech.

An effort was made to limit the amount of time required for students to complete the tests 

and questionnaires. The total time required in each class was just under two hours: 10 minutes to 

introduce the study and distribute the forms, 60 minutes for the reading test, 30 minutes for the 

listening test, 15 minutes for the questionnaire, and 5 minutes to collect the forms.

In addition to collecting the quantitative data, study staff made three site visits to 

participating schools. The first site visit, in October 2009, following the fall data collection, was 

made by a researcher from RTI to identify any issues with early implementation. Two interim 

site visits were made by staff from ECNU to observe implementation in the schools in winter and 

spring 2010.

Teachers were also asked to complete logs describing each class session. Research has 

shown that teachers are accurate reporters of classroom activities (Rowan and Correnti 2009). 

These logs were submitted monthly to ECNU, and feedback was provided to teachers as needed. 

See Appendix E for information on the revised evaluation plan.

3.3 Data Collection

Data collection was conducted in the 10 participating middle schools described in 

Section 3.2. Data were collected in late September 2009 and in late June 2010. Chinese test 

administrators (many from the local university) were trained to distribute consent forms, check 

the class rosters, administer the test, and collect tests and questionnaires from students and 

teachers (see Appendix F for the fall training manual and questionnaires and Appendix G for the 

end-of-year training manual and questionnaires). RTI staff were present and supervised the 

September data collection activities.

In both the September 2009 and June 2010 data collections, students were read a 

statement about the voluntary nature of the study and then were asked to complete the reading 

test and listening test and to fill out a paper-and-pencil student questionnaire. The listening test 

was administered using a tape recorder or, in a few cases, the school‘s public address system.

(Administration using a tape recorder is authorized by the test publisher.) All participating 

teachers were asked to fill out a paper-and-pencil teacher questionnaire. 

All eighth-grade classrooms in treatment schools were included in the fall 2009 testing 

because it was unclear as to which teachers and classes would participate; all eighth-grade 

classes in comparison schools were also tested in the fall. The total sample size for the fall 2009 

data collection was 6,073 students within 95 classrooms, including 3,102 students from five 
Chapter 3. Study Design and Methodology

16 A Study of a Gaming Approach to English Language Learning for Middle School Students

treatment schools and 2,971 students from five comparison schools. The response rate for the fall 

was 98.6 percent in the treatment schools and 99.1 percent in the comparison schools. A student 

was considered a respondent if he or she completed at least one of the assessments.

In the June 2010 post-test, only eighth-grade classrooms that used The Forgotten World

were tested in the treatment schools, along with an approximately equal sample of classrooms 

from the comparison schools. In the fall, there were 3,938 students in 62 classrooms that were retested at the end of the year: 1,967 students from treatment schools and 1,971 students from 

comparison schools. In all, 3,640 of these students completed at least one assessment at the end 

of the year, for an overall response rate of 92.4 percent; within the treatment schools, 93.4 

percent of students in participating classes completed the spring assessment compared to 91.4 

percent of students in the comparison schools).

The smaller number of students tested in June 2010 does not represent attrition; rather, a 

decision was made to test all students in treatment schools in fall 2009 because not all schools 

had provided final implementation plans, and the data collection contractor did not know exactly 

which classrooms would participate. The testing materials were shipped approximately one 

month before data collection, and at that time the participation status of some schools was 

unclear.

Teachers of eighth-grade English were also asked to complete questionnaires. All of the 

48 teachers identified in the fall (24 each in the treatment and comparison schools) completed the 

questionnaires for a 100 percent response rate. In the spring, the 22 participating teachers in the 

treatment schools and 26 teachers in the comparison schools (including new teachers) completed 

questionnaires, for a 100 percent response rate.

3.4 Data Processing

The final dataset used for most of the analysis includes only those records that appear on 

both the September 2009 and June 2010 files, a total of 3,640 records. The final dataset includes 

test IDs, student test scores, records of the amount of time spent using The Forgotten World (as 

recorded by the software developer on U.S. servers), student questionnaire response data, and 

teacher questionnaire response data. RTI provided the data entry templates to ECNU to 

implement the initial data entry. RTI staff performed quality comparison checks to ensure the 

data quality. 

The student test ID is a nine-digit number, which is composed of a three-digit school ID, 

a three-digit classroom ID, and a three-digit student ID. The teacher questionnaire ID is a

six-digit number including a three-digit school ID and a three-digit teacher ID. The teacher 

record also includes the class IDs for the classes taught by that teacher. The IPT1 reading test has 

24 questions, and the MAC II listening test has 53 questions. Data from the beginning and endof-year questionnaires are also included in the data files for teachers and students.
Chapter 3. Study Design and Methodology

A Study of a Gaming Approach to English Language Learning for Middle School Students 17

3.5 Data Analysis and Statistical Methods

Data analysis was conducted using SAS. Because of the relatively small number of 

schools included in the study, no attempt was made to estimate complex models. Because the 

purpose of the study was primarily descriptive, t-tests were used to compare results across 

groups. Also, because of the clustering of students within schools, design-corrected standard 

errors were computed and used for significance testing, using the SAS Surveymeans procedure.

In this report, no results are statistically significant unless specifically stated.

A Study of a Gaming Approach to English Language Learning for Middle School Students 19

Chapter 4. Implementation

4.1 Teacher Training

The workshop for training teachers was held August 3–6, 2009, on the campus of a local 

university in Gansu province. Headmasters, teachers, and school computer lab directors were 

invited to attend. Forty-four participants completed the training; four of the schools sent all 

eighth-grade English teachers, while one school sent one teacher (a supplemental training was 

held at this school in October). Participants from distant schools and the project team were 

housed in the campus conference center. Large sessions were held in the conference center, and 

hands-on practice sessions were held in a computer lab on campus. Sessions presented by 

persons from the United States were translated into Chinese.

The agenda for the training session is presented in Exhibit 4. Officials from the U.S. 

Department of Education and the Ministry of Education of the People‘s Republic of China 

described the history and purpose of the U.S.-China E-Language Learning System and 

emphasized the importance of the schools‘ participation in the project. Experts in teaching and 

learning were also asked to speak to the group about the program‘s significance, emphasizing 

that the approach to learning embedded in The Forgotten World differed from the traditional 

approach typically used in middle schools.

Based on the results of the pilot study, the length of the training was extended, and 

teachers were given more hands-on practice with the application. Sessions also included 

demonstrations of how the program had been used in classes during the pilot, a review of the 

teacher‘s guide, and planning time for teachers to develop lesson plans based on The Forgotten 

World. The Ministry of Education and the Provincial Education Bureau presented the 

expectations for program implementation and discussed the practical requirements of the project.

On the final day of training, teachers and headmasters were asked to meet with others 

from their school to develop an action plan for implementing the program at their school. When 

the teachers presented their school action plans, it became clear that there were challenges and 

that some schools might be unable to meet the program implementation requirements. One 

school‘s representative stated that only one eighth-grade class would participate; another 

school‘s representative said that only one teacher would implement The Forgotten World. These 

schools were reminded that the minimum requirement for participation was half of the eighthgrade classes. In two schools, the size of the average class was much larger than the size of the 

computer labs, and not all students would be able to use The Forgotten World during each class 

period. These issues were discussed, and a decision was made to follow up with these schools 

and the Ministry of Education to ensure that students had adequate exposure to The Forgotten 

World.

At the end of the training, participants were asked to complete a questionnaire about the 

training workshop. Forty-four questionnaires were completed. Participants were asked to rate 

major components of the training on a five-point scale, where 1 was ―not good at all‖ and 5 was 

―very good.‖ Mean scores for each component are presented in Exhibit 5. In general, the training 

received ratings of ―good‖ or better with one exception. Participants‘ rating of their own level of 

independent use was rated between ―average‖ and ―good.‖
Chapter 4. Implementation

20 A Study of a Gaming Approach to English Language Learning for Middle School Students

Exhibit 4. Agenda for Teacher Training

The Forgotten World

Teacher Training Workshop

Monday, August 3–Thursday, August 6, 2009

Monday, August 3

1:30–5:00 p.m. Preparatory Meeting for Coastline Community College, Learning Games 

Network, Gansu Provincial Center for Education Technology, National Center 

for Educational Technology, and Northwest Normal University Teams

Tuesday, August 4

8:30–9:30 a.m. Welcome and Introduction

9:30 a.m.–10:00 a.m. Demonstration of The Forgotten World

10:15 a.m.–12:00 p.m. Practice Session 1 in Computer Lab

12:00–2:30 p.m. Lunch and Break

2:30–3:45 p.m. Overview of Teacher’s Guide 

4:00–6:00 p.m. Keynote and Discussion

Playing Games and English Teaching: How to understand and use gaming in English 

teaching (Professor Zhang Lianzhong)

6:00–6:30 p.m. Wrap-Up and Questions 

Wednesday, August 5

8:30–8:45 a.m. Day Two Welcome 

8:45 a.m.–9:45 a.m. High Technology in Language Learning

Presentation on the value of informal, media- and technology-based applications in 

the teaching and learning of language (Professor Wu Heping)

9:45 a.m.–10:45 a.m. Practice Session 2 in Computer Lab

11:00 a.m.–12:00 p.m. Sharing the Pilot Experience 

Presentation and discussion on how the learning object was used in class and what 

worked in engaging students in learning (teachers from pilot schools)

12:00–2:30 p.m. Lunch and Break

2:30–3:15 p.m. Classroom Demonstration

Teachers who participated in the spring pilot act as model teachers while workshop 

participants act as ―students‖ in the demonstration 

3:15–4:00 p.m. Discussion

Teachers react to their experience as ―students‖ and explore strategies for how they 

will engage and support students using the learning object

4:15–5:30 p.m. Developing a Lesson

Teachers work in small groups to develop sample lessons based on their review of 

the Teacher’s Guide, experience using The Forgotten World and classroom 

demonstrations

5:30–6:30 p.m. Reporting

Overview of teacher dashboard and student/class reporting features: how to read 

reports and strategies for using the reports to shape classroom teaching and 

individual student instruction

6:30–8:00 p.m. Kick-Off Banquet

Celebration for all teachers, workshop participants, project leaders, and local officials

Thursday, August 6

8:00–8:15 a.m. Day Three Welcome 

8:15–9:00 a.m. Monitoring and Evaluation Program

Timeline from September 2009 to March/April 2010 

9:00–9:30 a.m. Experiences and Lessons From Evaluation in Pilot 

Presentation and discussion of what worked during the pilot, including thoughts on 

how teachers can use the program more effectively as a supplement to their 

current teaching resources and lesson plans (Professor Zhu Yiming)

9:45–11:00 a.m. School Groups Meet to Develop Action Plan

Headmasters and teachers collaborate in school-based small groups to discuss plans 

for working with the program in their classrooms

11:00–11:30 a.m. Sharing and Review of School Action Plans 

11:30 a.m.–12:00 p.m. Conclusion 

12:00–12:30 p.m. Closing Ceremony

Source: The U.S.-China E-Language Project.
Chapter 4. Implementation

A Study of a Gaming Approach to English Language Learning for Middle School Students 21

Exhibit 5. Participants’ Ratings of Teacher Training Workshop

Exhibit reads: Teachers were positive about the training experience but were critical of their own level of expertise in using The 

Forgotten World.

Note: TWF = The Forgotten World

Source: The U.S.-China E-Language Project.

On the training evaluation forms, teachers were also given the opportunity to share 

thoughts and concerns about using The Forgotten World in their classes. Three major concerns 

were expressed. First, teachers stated that they were worried about the condition of the 

computers and the Internet connection at their schools. They also asked for continued 

professional guidance and help, including meetings with experts and continuing opportunities for 

training. Finally, several participants expressed a desire to build exchanges and connections 

outside their schools and said they hoped to have more communication with American experts. A 

few stated that they would like to learn more about American middle schools. While some of 

these activities were beyond the scope of the project, it is clear that this project could be 

extended to promote exchanges between Chinese and U.S. educators.

Because the software and implementation team was interested in potentially sponsoring 

Web chats for the teachers, the questionnaire also includes an item asking participants how often 

they used the Internet. Nearly half of the teachers reported using the Internet every day, and 

another quarter reported using it two to three times per week, as shown in Exhibit 6. Based on 

these responses, an online forum was developed and Tencent QQ5 was used by teachers and 

experts to exchange information.

 

5 Tencent QQ is the most popular instant messaging and chat software in China.

4.86

4.11 4.00

3.45

4.07

0

1

2

3

4

5

Content of the 

training

Purpose of the 

program

Understanding of 

TFW

Level of independent 

TFW use

Confidence in 

teaching with TFW

Training elements
Chapter 4. Implementation

22 A Study of a Gaming Approach to English Language Learning for Middle School Students

Exhibit 6. Training Participants’ Self-reported Use of the Internet

Exhibit reads: Nearly half of teachers reported using the Internet every day.

Source: The U.S.-China E-Language Project.

4.2 School and Student Participation

One of the first questions to be addressed is whether the students in treatment and 

comparison schools were similar in English proficiency at the start of the study. Information on 

the number of classes and students in participating schools is presented in Exhibit 7. The first 

column shows the total number of eighth-grade classes in the school. The next four columns 

show that all students in all classes were tested in the fall and present their mean fall scores in 

reading and listening. The mean reading score of all students in treatment schools was 201.17 

compared to 200.08 in the comparison schools, while the mean listening scores were 176.08 and 

175.91, respectively. The scores for the reading scores range from 159 to 256, and the listening 

scores range from 79 to 267 [data not shown in exhibit]. The differences in mean scores between 

treatment and comparison schools were not statistically significant.

A second question that can be answered by this table is whether students in the sample of 

classrooms that participated in the study were representative of all the students in their schools.

Not all the classrooms in the treatment school used The Forgotten World. In two treatment

schools, all eighth-grade English classes participated; in the remaining three schools, about half 

the eighth-grade classes participated. Teachers within the treatment schools were not randomly 

assigned to participate; rather, teachers volunteered to participate in the study. The final sample 

includes only those classrooms in treatment schools that used The Forgotten World and a random 

sample of classrooms selected from the comparison schools.

The columns labeled ―Final sample‖ present the numbers of classes and students 

participating in the study and their mean scores on the fall test. It appears that the performance of 

Everyday

43%

2 to 3 times per 

week

30%

About once a week

11%

About once every 

other week

9 %

About once a month

5 %

Never

2 %
Chapter 4. Implementation

A Study of a Gaming Approach to English Language Learning for Middle School Students 23

students in classrooms participating in the study—those that used The Forgotten World—is

similar to the student performance in all eighth-grade classrooms: the mean reading score across 

all students in treatment schools was 201.17, while the mean in the participating classrooms in 

the treatment schools was 201.31. The mean listening scores were also approximately equal. The 

differences between the total eighth grade mean scores in reading and listening and the sampled 

eighth-grade classrooms were not statistically significant. Only in Treatment School 5 does there 

appear to be the possibility that higher-performing classes were selected to participate in the 

study and use The Forgotten World, but these differences were not statistically significant. In the 

comparison schools, the total mean reading and listening scores of students in the sampled 

classrooms closely matched the mean scores of the schools.

Exhibit 7. School and Student Participation

Total 

8thgrade 

classes

All Final sample Difference

Classes Students

Mean fall scores

Classes Students

Mean fall scores t values

Reading Listening Reading Listening Reading Listening

Treatment schools 

subtotal

School 1 6 6 269 196.69 172.55 6 243 196.96 173.02 .11 .05

School 2 8 8 596 197.84 166.61 8 552 198.13 167.04 .12 .08

School 3 13 13 956 203.11 179.55 6 414 204.51 181.93 .53 .96

School 4 12 12 932 205.14 180.79 6 439 205.64 182.08 .67 .68

School 5 8 8 405 195.32 173.51 4 190 199.22 177.16 1.17 .61

Total/mean score 47 47 3,158 201.17 176.08 30 1,838 201.31 175.81 .05 -.05

Comparison schools 

subtotal

School 6 8 8 572 196.64 162.45 6 394 196.69 162.57 .61 .03

School 7 12 12 748 198.56 172.48 7 374 199.08 174.02 .58 .41

School 8 8 8 487 204.76 187.32 6 345 206.42 192.51 .60 .69

School 9 8 8 287 196.64 170.92 4 130 196.05 165.81 -.96 -1.52

School 10 12 12 781 202.38 183.73 9 559 203.19 186.06 1.27 .92

Total/mean score 48 48 2,875 200.08 175.91 32 1,802 201.03 178.21 .40 .31

Difference treatment 

and comparison 

schools t =.45 t =.03 t =.10 t =-.34

Exhibit reads: Treatment and comparison schools were well matched, and students in the final sample of participating classes performed about the same on fall 

assessments as all the students in their schools. The possible scores for the reading scale range from 159 to 256, and the possible scores for the listening scale range 

from 79 to 267.

* p < .05

Source: The U.S.-China E-Language Project.

The characteristics of students and teachers in the treatment and comparison schools can 

be examined using data collected in the fall questionnaire. Exhibit 8 presents information on key 

student demographics and on the students‘ experience using English. The middle schools studied 

had more male students (55 percent) than female students (45 percent) , and most were born in 

1995 or 1996, which makes them slightly older than typical eighth-grade students in the United 

States.

Most students (nearly 70 percent) in the treatment group reported that they began 

learning English in third grade. Typically, students are introduced to English in elementary 

school; the frequency and intensity of instruction increase when students begin middle school in 

seventh grade, and the majority of students, more than 80 percent, expected to attend college.
Chapter 4. Implementation

24 A Study of a Gaming Approach to English Language Learning for Middle School Students

Approximately half of the students reported having computers at home, having Internet access, 

and playing computer games once a week or more.

Exhibit 8. Characteristics of Students

Percentage of students who report:

The Forgotten World

schools Comparison schools

Percent Number Percent Number Difference t value

Sex

Female 44.61 3,080 47.03 2,815 -2.42 -1.23

Year of birth 3,040 2,795

Before 1995 10.07 9.23

1995 51.41 52.77 -1.36 -.45

1996 36.94 36.92 0.02 .00

After 1996 1.58 1.08

Grade began learning English 3,029 2,792

Before third grade 4.72 16.12

Third grade 68.67 59.46 9.21 1.22

Fourth through seventh grade 14.16 13.47

Eighth grade 12.45 10.96 1.49 1.12

Expecting to attend university 83.71 3,063 80.53 2,815 3.18 1.02

Having computer at home 50.65 3,068 42.93 2,816 6.42 -.41

Having Internet access at home 43.11 3,055 49.22 2,812 -6.11 -.45

Playing computer games once a week or more 58.15 3,068 54.38 2,803 3.81 .71

They have ever been tutored in English 90.76 3,075 88.36 2,819 2.4 .70

Reading English-language books, magazines, 

newspapers, and websites once a week or more 70.54 2,872 66.69 2,594 3.85 1.46

Using English to speak with friends/family 43.9 2,861 48.42 2,594 -4.52* -2.12

Playing computer games with English once a week 

or more 41.33 3,075 39.52 2,816 1.81 .40

Watching English TV once a week or more 40.11 3,069 37.13 2,817 2.98 1.02

Chatting online in English once a week or more 15.34 3,070 13.55 2,820 1.79 .77

They have traveled to English-speaking country 3.45 2,872 5.74 2,594 2.29 -1.21

Exhibit reads: Characteristics of students were similar in treatment and comparison schools.

* p < .05

Source: The U.S.-China E-Language Project.

Several questions were asked of students to determine their prior experience in learning 

and using English outside of school. About 90 percent of students reported that they had received 

tutoring in English at some time, and 70 percent reported that they read books, magazines, 

newspapers, or websites in English. Approximately 40 percent sometimes use English to speak 

with family or friends, and a similar percentage play computer games with English or watch 

English-language television programs at least once a week. Students in treatment schools were 

significantly less likely than students in comparison schools to speak English with family and 

friends. Only a small number, 3 percent, reported having traveled to an English-speaking 

country.

English teachers in this study were predominantly female and have an average of 

13 years‘ experience teaching English, as shown in Exhibit 9. The majority (about 79 percent) 

have four-year degrees in English. The average age is mid-thirties. Significantly more teachers in 

the treatment schools were female than in the comparison schools
Chapter 4. Implementation

A Study of a Gaming Approach to English Language Learning for Middle School Students 25

Exhibit 9. Characteristics of Teachers

Characteristic

The Forgotten World

schools Comparison schools

Percent Number Percent Number Difference t value

Percent of teachers whose highest degree 

is four-year degree or above in English 78.95 19 86.36 22 7.42 t=-1.04

Mean age of teachers 36.38 24 33.58 24 2.79 t=1.47

Percent female 91.67 24 70.83 24 20.83* t=2.6

Mean years teaching English 13.33 24 10.46 24 2.87 t=1.19

Exhibit reads: Characteristics of teachers were similar in treatment and comparison schools.

* p < .05

Source: The U.S.-China E-Language Project.

4.3 Implementation of The Forgotten World

Early in the fall, it became apparent from teacher logs and e-mails to support staff that the 

Internet connections in the schools were insufficient to support large groups of students using 

The Forgotten World simultaneously. While the software development team had visited all the 

computer labs in August 2009, when school was not in session, this problem surfaced in late 

September 2009, when the system was used simultaneously by large numbers of students.

Students would log in to the application but would often lose their connection and be forced to 

log in a second time. Personnel in the province worked to diagnose the issue, but the problem 

could not be fixed without the purchase of new servers for the schools. The budget for 

implementation provided by the William and Flora Hewlett Foundation did not include the 

purchase of equipment. In December 2009, the Dell Corporation agreed to donate servers to the 

schools. During fall 2009, some teachers had continued to use The Forgotten World, but system 

performance was frustrating to teachers and students, as documented by site visits and teaching 

logs.

Chinese schools typically have a long holiday to celebrate the New Year. In this 

province, schools closed January 18, 2010, and reopened March 1, 2010. Thus, the new servers 

were not used until March 2010. Because of the initial difficulties and the long break, the project 

team decided that the project needed a jump start, and the lead U.S. software developer traveled 

to China to visit schools and conduct supplementary training in early March 2010. Reports from 

the training were positive, and a site visit by ECNU confirmed that teachers were enthusiastic 

about using the application now that the Internet connections were stable and response times had 

improved.

Information on the frequency of use of The Forgotten World in participating classes was 

collected from multiple sources, as presented in Exhibit 10. Two sources of information are 

presented in this table: student reports of how frequently they used The Forgotten World both in 

and out of class and teacher reports of how frequently they used The Forgotten World in class 

and how they organized students to use it outside of class.
Chapter 4. Implementation

26 A Study of a Gaming Approach to English Language Learning for Middle School Students

Exhibit 10. Reported Use of The Forgotten World by Students and Teachers 

The Forgotten World schools subtotal

Percentage of students and teachers who used 

The Forgotten World once a week or more

In class 

(reported by 

students)

Outside of class

(reported by 

students)

In class

(reported by 

teachers)

Outside of class

(reported by 

teachers)

School 1 78.92 17.30 100.00 33.33

School 2 70.32 12.12 50.00 0.00

School 3 71.96 24.32 80.00 40.00

School 4 84.24 20.52 100.00 50.00

School 5 84.95 27.03 75.00 25.00

Total n 1,728.00 1,725.00 22.00 22.00

Exhibit reads: Over 70 percent of students reported that they used The Forgotten World at least once a week in class.

Source: The U.S.-China E-Language Project.

Schools participating in the study were asked to provide students with the opportunity to 

use The Forgotten World at least once each week in class. Over 70 percent of students reported 

that they used The Forgotten World once a week or more in schools, and about 20 percent 

reported that they used it once a week or more outside of class. About 81 percent of teachers 

reported that they used The Forgotten World once a week or more in class, and about 29 percent 

of teachers reported they asked students to use the program outside of class. It is not surprising 

that student reports of use are somewhat lower than teachers‘ reports, because the size of the labs 

required students to share computers at times, and because students may not have participated in 

each session.

In June 2010, teachers from each school reported on their progress in using The 

Forgotten World. One school had completed all 12 episodes, all schools had completed at least 7 

episodes, and most had completed 8–10 episodes.6

Clearly, the implementation of The Forgotten World was incomplete. Because of 

technological limitations, use during the first semester was limited. However, use during the 

second semester approached project goals, with nearly three-quarters of all students reporting 

that they used the program at least once a week.

4.4 Teachers’ Classroom Use of The Forgotten World

The teacher logs provide information on how The Forgotten World was used in classes

from March through June 2010. Typically, teachers reported using The Forgotten World once a 

week with classes, for approximately 40–45 minutes. Time spent on completing one episode 

varied across schools and teachers: the reported range was from one to six class periods to 

complete one episode of The Forgotten World.

Before class, teachers reported spending time to get familiar with the stories and games 

and design questions and activities to help students learn the story, the language points, the 

sentence structures, and practice conversations. Teachers would also check network connections

 

6 Teachers in treatment school 5 had finished teaching all 12 episodes. Treatment school 1 reached 

Episode 10; classes at treatment school 4 had reached Episodes 8–10; treatment school 3 was at Episodes 10–11; 

and treatment school 2 was at Episode 7 in Class 7; Episode 9 in Classes 1, 2, 3, and 8; and Episode 12 in Classes 4,

5, and 6.
Chapter 4. Implementation

A Study of a Gaming Approach to English Language Learning for Middle School Students 27

and website log-in conditions before class. Students were reminded before the start of class that 

their user names should not be shared with other people.

In class, teachers typically began by demonstrating the story on the teacher‘s computer

projected to a large screen, while explaining the stories to the students, pointing out specific

knowledge points and sentence structures, and teaching students how to play the games. After 

the teacher‘s demonstration, students would learn the stories, mimic and practice conversations 

on their own, and play the games on their individual computers. At the end of class, teachers 

sometimes would assign homework to the students. 

The teacher logs indicate that students became enthusiastic about the program after the 

network condition improved. Students were interested in the stories and games, and in carrying 

out conversations. Teachers reported that some students found games more interesting than 

stories. The majority of students were involved in the class, but they progressed at different rates, 

making classroom management difficult. Teachers sometimes asked students who went faster to 

form small discussion groups so students learning at a slower pace could have time to finish the 

story. 

A Study of a Gaming Approach to English Language Learning for Middle School Students 29

Chapter 5. Results

5.1 Student Results Based on the Assessments and Questionnaires

Did students in classrooms that used The Forgotten World improve their English reading 

and listening skills more than students in classrooms in comparison schools that did not have 

access to the program? Overall, students participating in the treatment schools did not increase 

their proficiency more than students in the comparison schools. However, it appears that students 

with relatively low levels of proficiency (those who scored below the median on the fall 

proficiency tests) made greater gains in the treatment schools than in the comparison schools.

Exhibit 11 presents information on the student scores at the beginning and end of the year on the 

reading and listening assessments. This table includes all students who were in the final sample, 

regardless of their reported use of The Forgotten World. The mean gain in reading for lowerperforming students was 8.5 in treatment schools compared with 5.7 in comparison schools. In 

listening, the lower-performing students gained 9 points in treatment schools and 5.5 points in 

the comparison schools. These differences were statistically significant. This finding was 

confirmed by use of a hierarchical linear model, presented in Appendix H.

Exhibit 11. Mean Performance on Assessments

The Forgotten World schools Comparison schools

Difference t value

Beginning 

of year

End of 

year

Mean 

individual 

gain or 

loss

Beginning 

of year

End of 

year

Mean 

individual 

gain or 

loss

Fall reading score

Bottom half (below the 

median) 194.45 203.05 8.48 194.46 200.23 5.70 2.78* 2.28

Top half (above the 

median) 207.41 214.65 7.44 207.19 215.37 7.96 .52 -.04

Reading mean 201.31 209.28 7.92 201.03 207.85 6.85 1.07 .88

Total n 1,810 1,782

Fall listening score

Bottom half (below the 

median) 164.39 173.53 9.03 164.50 170.09 5.47 3.56* 2.75

Top half (above the 

median) 185.61 188.99 3.35 191.59 195.80 4.15 -.8 -.51

Listening mean 175.81 181.79 5.97 178.21 183.01 4.80 1.17 .87

Total n 1,824 1,786

Exhibit reads: Students who began the year in the bottom half of the sample increased their English proficiency more in treatment 

schools than in comparison schools.

* p < .05

Source: The U.S.-China E-Language Project.

Did students who reported using The Forgotten World more often exhibit increased 

English language proficiency? Exhibit 12 examines student performance on beginning and endof-year assessments by how frequently they reported using The Forgotten World. The table 

suggests that the students who used The Forgotten World more frequently may have made 
Chapter 5. Results

30 A Study of a Gaming Approach to English Language Learning for Middle School Students

slightly more gains in reading and listening proficiency than students who never used the 

program or used it only once or twice. While the results are in the expected direction, with this 

sample size, the results are not statistically significant.

Exhibit 12. Performance on Assessments, by Use of The Forgotten World

Used The Forgotten World . . .

The Forgotten World schools

Beginning of year End of year

Mean gain or loss

Never (n = 82)

Reading score 197.59 201.47 3.89

Listening score 172.63 172.56 −0.07

Once every few months (n = 101)

Reading score 197.23 202.93 5.74

Listening score 171.92 174.86 3.16

Once or twice a month (n = 221)

Reading score 198.99 206.69 7.68

Listening score 172.26 176.75 4.49

Once or twice a week (n = 1283)

Reading score 202.69 211.22 8.53

Listening score 177.63 184.61 6.96

Three to four times a week (n = 32)

Reading score 202..09 209.25 7.15

Listening score 173.06 184.21 11.16

Every day (n = 9)

Reading score 199.66 206.33 6.67

Listening score 168.33 173.89 5.56

Exhibit reads: Relationship between frequency of use and improved performance on assessment is not statistically 

significant.

Source: The U.S.-China E-Language Project.

Did using The Forgotten World have an effect on students’ interest in or motivation to 

learn English? Language experts and teachers in China repeatedly told the study team that 

language learning is not popular in China, even though it is important for success in school. 

Research on teaching English as a foreign language has identified motivation as an important 

factor (see, e.g., Norris-Holt 2002), and there is a long tradition of research that links student 

motivation and academic achievement (Dalton 2010). Thus, one of the goals of this study was to 

examine whether this approach to language learning might have an effect on student interest in 

and motivation to learn English.

Almost all middle school students, approximately 95 percent, who participated in this 

study believed that studying English is somewhat or very important, as shown in Exhibit 13. 

However, fewer students expressed interest in learning English, and it appears that interest 

waned for both groups as the year progressed.
Chapter 5. Results

A Study of a Gaming Approach to English Language Learning for Middle School Students 31

Exhibit 13. Importance of and Interest in Learning English

The Forgotten World schools Comparison schools

Beginning of year End of year Beginning of year End of year

Percent Number Percent Number Percent Number Percent Number

Importance of learning English 96.44 1,798 92.87 1,794 94.60 1,778 90.19 1,784

Interest in learning English 70.47 1,798 60.90 1,793 67.60 1,778 54.53 1,779

Exhibit reads: Nearly all students reported that learning English is somewhat or very important.

Source: The U.S.-China E-Language Project.

At the end of the year, students were asked about their motivation to learn English, and 

how much fun it was to learn English. Fifty-one percent of students in the treatment schools said 

that motivation to learn English had improved, compared to 40 percent of students in comparison 

schools, as shown in Exhibit 14. From the outset of the study, some Chinese experts suggested 

that increasing student motivation to learn English might be one of the most important benefits 

of using The Forgotten World. Given the relatively limited exposure that students had to the 

program, the suggestion that this might be the case is encouraging.

Exhibit 14. Percentage of Students Who Reported That Motivation to Learn English Improved

and That Learning English Is Fun

Exhibit reads: A higher percentage of students in schools that used The Forgotten World reported that motivation improved 

compared to students in comparison schools.

Source: The U.S.-China E-Language Project.

Note: Percentage motivated (.5092; SE = .01753) in treatment schools is higher than percentage motivated (.3978; SE = .0406) in 

comparison schools; t = 2.52 and p < .05. Difference in fun between groups is not significant.

What were student attitudes toward The Forgotten World? About 40 percent of 

students reported that they never used The Forgotten World outside of school, and about the 

same number reported that they used the program outside of school once a week or more, as 

shown in Exhibit 15. Students also provided mixed reports on the helpfulness of the program: 

50.9 50.9

39.8

44.9

0

10

20

30

40

50

60

70

80

90

100

Motivation to learn 

English improved

Learning English is fun

TFW Schools

Comparison schools
Chapter 5. Results

32 A Study of a Gaming Approach to English Language Learning for Middle School Students

46 percent said that it was somewhat or very helpful, 42 percent said it was not helpful, and 

12 percent said they didn‘t know or hadn‘t used it much.

Students reported that they faced a number of challenges in using The Forgotten World. 

The major challenges, reported by over half the students, involved not having enough time to 

engage in the episodes (64 percent) and problems with the computers at school (57 percent). 

Concerns about the possible negative effect of The Forgotten World on test scores were voiced 

by a small number of students, about 10 percent.

Exhibit 15. Student Experience With The Forgotten World

Percent Number

Use of The Forgotten World outside of school (Q14) 1,722

Never 40.29

Twice a month or less 19.19

Once a week or more 40.52

How helpful was The Forgotten World? 1,723

Don’t know or didn’t use much 12.19

Not helpful 41.96

Somewhat helpful 37.55

Very helpful 8.30

What was biggest challenge? (multiple answers allowed)

Insufficient time for using the program 64.60 1,723

Limited support from school 23.51 1,723

Problems with computers at the school computer lab 57.05 1,723

Concerns with negative impact on test scores 9.81 1,723

Other challenges 14.14 1,722

Exhibit reads: Students reported mixed opinions on whether The Forgotten World was helpful.

Source: The U.S.-China E-Language Project.

Students were given the opportunity to specify other challenges they faced in using the 

program. The challenges can be categorized into four groups, related to (1) the program content 

and structure, (2) lack of access to the software, (3) hardware and connectivity issues, and 

(4) personal issues. Challenges mentioned by more than 10 students each included the following:

Program content and structure issues

 Difficulty understanding sentences

 Too many unknown vocabulary words

 Insufficient Chinese translation

 Speech too fast to understand

 Dictionary slow to use, not always helpful

Lack of access issues

 Teachers did not always hold class

 Insufficient computers for all students in class

 Not enough time to use application

 No computer at home
Chapter 5. Results

A Study of a Gaming Approach to English Language Learning for Middle School Students 33

Hardware and connectivity issues

 Internet speed was slow.

 Computers sometimes were automatically logged out.

 Computers sometimes froze.

 Computers sometimes failed to log in.

5.2 Teacher Attitudes and Experiences

What were teacher attitudes toward The Forgotten World? At the start of the project, 

the TWG and team discussed how The Forgotten World should be delivered to students. One 

group felt that it was best as a standalone application that could be accessed directly by students; 

the other group felt that teachers‘ use of the program was essential. Both groups were concerned 

about teachers‘ receptivity to the new approach but thought that implementing this approach in 

classrooms would afford teachers the opportunity to think about their teaching in novel ways.

Teachers‘ opinions on the helpfulness of The Forgotten World were mixed, as were 

student opinions. As presented in Exhibit 16, some 59 percent reported that it was somewhat 

helpful, and 41 percent reported that it was not very helpful. No teachers chose the more extreme 

options of ―very helpful‖ or ―not at all helpful.‖

A majority of teachers said they were unlikely to use The Forgotten World, or any other 

technology approach, in the future: only 14 percent said they were likely to use The Forgotten 

World again, and only 18 percent reported that they were likely to use some other software.

(Reasons teachers gave for these responses are presented in Exhibit 17.) 

Teachers reported noticing a change in students‘ motivation to learn English, with only 

5 percent reporting that they noticed no change. This is consistent with the finding that students 

in treatment schools reported higher levels of motivation than students in comparison schools. In 

light of some of the comments from students about the difficulty of the program, it is interesting 

that more than 70 percent of teachers said that they would recommend that The Forgotten World

be used for eighth-grade students, rather than seventh- or ninth-grade students.

Teachers were asked follow-up questions to two of the items reported in Exhibit 16: to 

explain why they were likely or not likely to use The Forgotten World in the future, and to 

provide suggestions about how The Forgotten World might be improved. Exhibit 17 presents 

some of the reasons that teachers are likely or unlikely to use the program in the future. Among 

those who said they were not too likely to use it again, the most common reason cited was that it 

represented too much additional content and vocabulary to cover, given the time available. A few 

also mentioned that preparing students for entrance exams was a priority. Among those teachers 

who said they were likely to use the program again, the major reason cited was that it generated 

interest and enthusiasm on the part of students.
Chapter 5. Results

34 A Study of a Gaming Approach to English Language Learning for Middle School Students

Exhibit 16. Teacher Experience With The Forgotten World

Percent

How helpful was The Forgotten World? (n = 22)

Not helpful at all 0.00

Not very helpful 40.91

Somewhat helpful 59.09

Very helpful 0.00

How likely to use The Forgotten World in future (n = 22)

Not likely at all 0.00

Not very likely 59.09

Somewhat likely 27.27

Very likely 13.64

How likely to use other software (n = 22)

Not likely at all 0.00

Not very likely 40.91

Somewhat likely 40.91

Very likely 18.18

Has student motivation changed since using The Forgotten World? (n = 22)

No change 4.55

A little more motivated 81.82

Somewhat more motivated 13.64

A lot more motivated 0.00

For what grade level would you recommend The Forgotten World? (n = 22)

Seventh grade 19.05

Eighth grade 71.43

Ninth grade 9.52

Exhibit reads: Teacher reaction to helpfulness of The Forgotten World was mixed, but they did report noticing that students were 

more motivated.

Source: The U.S.-China E-Language Project.

Specific suggestions for improving The Forgotten World are presented in Exhibit 18. It is 

clear from the comments that the quality of the schools‘ computer laboratories and the 

connectivity to the Internet were major issues for the teachers. The size of the laboratories, 

relative to the number of students in class, was also problematic. Perhaps because the program 

was only implemented for a single semester, teachers reported that students did not have enough 

time to use the program. There also appear to be a number of areas in which the software 

application might be improved, including allowing teachers and students to adjust the speed of 

the speech contained in the program, provide clear and simple instructions for the games, and 

provide more support for the vocabulary presented in the program. A few teachers also 

commented on the quality of the story and episodes.
Chapter 5. Results

A Study of a Gaming Approach to English Language Learning for Middle School Students 35

Exhibit 17. Teacher Comments About Why They Are Likely or Not Likely to Use The 

Forgotten World in the Future 

How likely are you to use The Forgotten World in the future, and why?

Not likely

Comment 1 None

Not too likely

Comment 1 Some network issues impacted the teaching activities.

Comment 2 The games and content of the software are rarely related with the test. School and 

parents have the requirement for students to do well on the test scores.

Comment 3 The game design could not attract student's attention when using; there was a difficulty 

in organizing the class.

Comment 4 The Forgotten World does not have a systematic knowledge structure, so it cannot be 

part of the curriculum.

Comment 5 There was a lot of content and not enough class hours, The Forgotten World could only 

be used as a supplementary material.

Comment 6 Teaching tasks were too much. There were not enough class hours. Work load was too 

much. There were too many students in each class.

Comment 7 1. There is already a lot of content in student textbooks; 2. Majority of the contents in 

this software are in reading.

Comment 8 The textbooks which we are currently using are comprehensively designed, plus there 

were not enough class hours for The Forgotten World, and it could only be used as 

a supplementary materials.

Comment 9 Too much new vocabulary, colloquial, not suitable for exam preparation.

Comment 10 Factor of high school entrance exam.

Comment 11 Since there are many limitations, The Forgotten World is very likely to be used only as 

supplementary curriculum.

Somewhat likely

Comment 1 It attracts students by combining computer online operation with English; Stories are 

intricate and abundant, which fit the age group of the students.

Comment 2 It significantly increased the enthusiasm of students in learning English; it could 

increase the comprehensive quality of students in full scale.

Comment 3 To great extent The Forgotten World helps the students develop and increase their 

interest in learning English. Meanwhile, it is a very effective supplementary means 

of teaching.

Comment 4 The Forgotten World is a beneficial supplement of current English-language teaching.

Comment 5 English teaching should fuse language teaching into the context of language.

Very likely 

Comment 1

Students are very interested in it. It is an inevitable trend to improve out-dated 

hardware.

Comment 2 The program is a combination of information technology and test preparation.

Exhibit reads: Teachers’ reasons for using or not using The Forgotten World in the future.

Source: The U.S.-China E-Language Project.
Chapter 5. Results

36 A Study of a Gaming Approach to English Language Learning for Middle School Students

Exhibit 18. Teacher Suggestions About Improving The Forgotten World

Comment 1 1. School must provide appropriate hardware so as to guarantee every class goes smoothly. 

Otherwise, it will be like the current situation where a majority of students had to log in 

repetitively, which discouraged students; 2. In the software, the pronunciation, tone, and 

speed of some sentences are very difficult; 3. Teachers must communicate and cooperate;

Avoid individual and separate work; 4. Teachers need to make efforts to familiarize the 

texts; 5. Students must have sufficient time to use computer.

Comment 2 Improve outdated teaching equipment; obtain support from parents; increase opportunity and 

time for students to practice speaking on certain subjects out of class.

Comment 3 1. Excited students’ learning interest by explaining the stories; 2. Help students learn new 

vocabulary and phrases by playing games; 3. It would be better if the hardware at school 

was improved.

Comment 4 1. Increase time to practice and learn; 2. Design The Forgotten World to be actual listening and 

speaking curricula so as to excite students so that they will take it more seriously; 3. 

Provide more training and learning opportunities to teachers in order to increase teaching 

level and professional quality.

Comment 5 1. Ensure the network runs smoothly; 2. Make stories more exciting so students will be more 

interested in it; 3. Make the stories shorter; design quiz after each episode; 4. The games 

should be simpler so students can enjoy success.

Comment 6 1. Some episodes/stories are too long. It will be better to divide it into several shorter episodes; 

2. Decrease the amount of vocabulary in games such as TRIGANGLE so that majority of 

the students could enjoy each class and increase their interest and confidence learning 

English; 3. It will be better if the stories are closer to real life and practice.

Comment 7 The software design should stay closer to the textbook to be its extension, so that students can 

grasp the textbook contents when playing the games. The software should start from lower 

level. It's hard to correct eighth-grade students’ pronunciation; the conversation speed in 

the games is pretty fast and students have limited time to use computer, so it's hard for 

students to improve.

Comment 8 The episodes and games should match with the textbooks, which would be beneficial to 

improving students’ test scores and the teachers’ rate of using the software.

Comment 9 1. The instructions for games should be more clear and specific; 2. If episodes were designed 

with a cartoon effect, it would be easier.

Comment 10 1. Need to have collation of English and Chinese. There should be some Chinese notes for the

difficult sentences; 2. Listening speed should be adjustable from slow to fast; 3. Correct 

instructions should be provided for the games.

Comment 11 There should be simple textbooks to help students in learning English designed to help 

students learn English while playing. Need to help students use computer with purpose.

Comment 12 1. The contents should stay close with actual lives and current time; 2. There should be a 

system of grammar; 3. The difficulty level should go from easy to hard gradually; 4. The 

story should be vivid and interesting; 5. It will be better to design it as a cartoon; 6. Games 

should combine knowledge and ease of playing to help student learn.

Comment 13 1. Need to make sure there is more time to use computer; 2. Enforce supervision and 

management when students use computers.

Comment 14 Provide students enough time and provide teachers enough freedom and time to look into the 

contents.

Comment 15 Students must be self-disciplined and willing to participate; there should be enough time every 

time to finish the task.

Comment 16 Must make sure every student has a computer; There should be at least 3–4 classes every 

week; the electronic dictionary should provide entries that cover all the new vocabulary 

words students will encounter when learning the stories.

Comment 17 It is better to design the game in animation, make the stories more interesting to increase 

student interest.

Comment 18 There should be computers available in the classrooms which can be used at any time.

See notes at end of table. Cont’d next page
Chapter 5. Results

A Study of a Gaming Approach to English Language Learning for Middle School Students 37

Exhibit 18. Teacher Suggestions About Improving The Forgotten World—Continued

Comment 19 There is a conflict between high school entrance exam and language application; the sections 

on grammar and vocabulary in the high school exams are very detailed, and the proportion 

of the score based on listening and oral communication is relatively small. The Forgotten 

World provides an opportunity for the students to practice listening skills and develop 

language sense.

Comment 20 The interface should be more user friendly. The games should be more interesting. Need to 

make the software run more smoothly. The problems encountered during usage should be 

addressed in a timely manner.

Exhibit reads: Teachers provided suggestions about improving The Forgotten World.

Source: The U.S.-China E-Language Project.

One of the hopes of the Chinese experts who advised the project team was that The 

Forgotten World would provide with teachers an opportunity to collaborate and to rethink their 

approach to teaching. As shown in Exhibit 19, some 77 percent of teachers who participated in 

this project reported collaborating with other teachers at least once a week. Even more striking is 

the fact that 95 percent of participating teachers reported that it changed the way they think about 

teaching. Note that the fact that teachers reported that it changed the way they think about 

teaching may be the result of participating in a new project, and may not necessarily be due to 

the particular aspects of this approach.

Exhibit 19. Teacher Experience With The Forgotten World

Percent

How often did you collaborate with other teachers while using The Forgotten World? (n=22)

 Never or once every few months 0.00

Once or twice a month 22.73

Once or twice a week 54.55

Three times a week or more 22.73

Has using The Forgotten World changed the way you think about teaching? (n=22)

Yes 95.45

No 4.55

Exhibit reads: Nearly all teachers reported that using The Forgotten World changed the way they think about teaching.

Source: The U.S.-China E-Language Project.

Teachers were asked a follow-up question to explain how using this approach changed 

the way they think about teaching. Exhibit 20 presents their answers. Many teachers referred to 

the affective impacts of the approach, using words and phrases like ―spontaneous,‖ ―happy 

learning experience,‖ ―class became dynamic,‖ ―encourages students,‖ ―apply games and 

entertainment in teaching,‖ ―nurture interest,‖ and ―feel native English.‖ A secondary thread 

through the teacher comments emerges in the references to students‘ active role in learning:

―reciprocity between teachers and students,‖ ―they learn from experience and application,‖ ―we 

should return the classroom to the student.‖
Chapter 5. Results

38 A Study of a Gaming Approach to English Language Learning for Middle School Students

Exhibit 20. Teachers’ Comments on How This Approach to Language Changed the Way 

They Think About Teaching

If The Forgotten World has changed the way you think about teaching, please explain

Comment 1 Innovation of teaching concepts. The Forgotten World combines and connects 

with current task-oriented teaching method to reflect the teaching process 

where teachers take the lead, students act as major component, and there is 

reciprocity between teachers and students.

Comment 2 In English learning, students attempt to transition to learning spontaneously in 

specific language contexts. They learn from experience and application.

Comment 3 The Forgotten World at least promotes us to change perceptions away from 

traditional teaching models; it’s applicable to current education reform system; 

it provides students a new experience.

Comment 4 The Forgotten World provides me new teaching concepts; by letting students 

have a happy learning experience, the whole class became dynamic based 

on this new teaching concept.

Comment 5 It encourages students to learn English for applying it.

Comment 6 We should return the classroom to the students to let them speak more.

Comment 7 Strengthen listening and speaking training.

Comment 8 Infiltrate new teaching concepts to teaching practice; apply games and 

entertainment in teaching.

Comment 9 Audio-visual characteristics in multimedia excited students’ interest in learning 

English.

Comment 10 Improve ability in listening and speaking by imitating native English.

Comment 11 If there is no knowledge atmosphere, there is no language.

Comment 12 Should emphasize speaking and listening practice.

Comment 13 Electronic textbooks attract students.

Comment 14 Suggest promoting this software in developed cities.

Comment 15 Nurture interest in remembering vocabulary, listening, and language sense.

Comment 16 Do not only focus on exams, but let students feel the native English.

Source: The U.S.-China E-Language Project.
A Study of a Gaming Approach to English Language Learning for Middle School Students 39

Chapter 6. Summary and Implications

6.1 Summary of Findings

Several implementation issues encountered during the study significantly reduced the 

amount of time students could spend on The Forgotten World. The expected frequency and 

intensity of implementation was not realized until halfway through the study, after information 

technology infrastructure improvements were made at treatment schools. Even then, only about 

70 percent of students reported that they used The Forgotten World at least once a week. This 

partial implementation during one semester represents a fairly limited treatment in comparison 

with other education studies of this type.

While overall, students in treatment and comparison schools performed similarly on the 

proficiency tests administered at the beginning and end of the school year, students in treatment

schools with relatively low levels of initial proficiency (those who scored in the bottom half of 

all students on the fall tests) evidenced larger gains in reading and listening than did similar 

students in comparison schools. This finding was statistically significant for both reading and 

listening.

Students in the treatment group reported higher levels of motivation to learn English than 

students in the comparison schools. At the end of the year, 51 percent of students in the treatment

schools reported that their motivation to learn English had improved, compared with 40 percent 

of students in the comparison schools. Teachers in treatment schools also reported that students‘ 

motivation improved. This is an important finding because motivation to learn a foreign

language has been identified by Chinese language experts as a problem in Chinese schools and a 

barrier to learning. 

Almost all of the teachers in the treatment schools (95 percent) who participated in the 

project reported that their use of The Forgotten World changed the way they think about 

teaching. The approach to language teaching and learning that forms the basis for The Forgotten 

World is learner-centered and interactive, and it presents students with problem-based tasks and 

immediate feedback. The teachers appreciated the fact that the program provided students with 

fun opportunities for practicing listening and speaking skills, as well as exposure to native 

English speakers. However some teachers expressed concern that their primary responsibility 

was to prepare students for the English portion of the high-school entrance exam that students 

take in ninth grade, and they did not feel that use of The Forgotten World would improve student 

performance on the exam.

6.2 Implications for the Future

Because of the difficulty with implementation, it is difficult to judge the true efficacy of 

The Forgotten World. However, the project team was able to gather information on the issues 

that should be addressed before distributing the program more broadly in China, or elsewhere.

The issues to be addressed fall into three major categories: hardware, teacher training and 

support, and program issues.

Hardware issues. Many times in studies such as this, the implementation of a program 

fails because teachers or administrators are not committed to the project. This was not the case in 

this study; teachers and administrators showed remarkable perseverance in using The Forgotten
Chapter 6. Summary and Implications

A Study of a Gaming Approach to English Language Learning for Middle School Students 40

World, even in the face of major challenges. Two of these—the quality of the computer labs and 

Internet connections, and the size of computer labs relative to average class size—must be 

addressed to make the implementation of this or any other technology-based approach to 

language learning feasible. Neither of these issues has a cheap or easy fix. Thus, for the 

immediate future, only a subset of schools in China can be expected to implement the program 

without a major investment in upgrades to their computer labs. To complicate this matter, in 

many eastern Chinese cities, which have stronger computing infrastructures, English instruction 

begins at an earlier age, and the content of The Forgotten World would be too easy for middle 

school students in those cities. Identifying schools that have the right constellation of conditions 

for implementation is an important but challenging task.

Teacher training and support. Implementing a new approach to teaching and learning 

requires training, support, and time. While teachers were satisfied with the initial training, from 

their logs and answers to questionnaire items it is clear that several areas of training and support 

could be improved. The software development and implementation team prepared a 

comprehensive teachers‘ guide that provided teachers with sample lesson plans as well as other 

practical information, such as the vocabulary used in each episode. However, teachers still felt 

uncomfortable with some aspects of using The Forgotten World in their classrooms.

First, teachers noted in logs, site visits, and questionnaires that classroom management in 

the computer lab was difficult. Because of the large class sizes observed in these middle schools, 

up to 80 students in a single class, managing independent activity by students may be more 

challenging than it would be with a smaller number of students. Students had many questions for 

teachers, and the teachers sometimes felt they could not answer all questions. Also, teachers 

reported that the program was too difficult for some students and that supervising those students 

added to the difficulty in controlling the class. Training specifically designed to help teachers 

work successfully in such environments might be beneficial.

Another related area for additional teacher workshops may be the area of differentiated 

instruction—that is, tailoring instructional content to a diverse group of learners within a single 

classroom. In the United States, a considerable body of literature discusses strategies for teachers 

to target instruction to learners at different levels within the same classroom. There may be 

lessons that can be transferred from the research literature on this topic and applied to this issue.

In the classes described by many teachers in this study, whole-group instruction was sometimes 

favored over more individualized work with small groups; again, this may be due in part to the 

large class size. Finally, in keeping with the teachers‘ own judgments of the initial training, more 

time should be spent on providing teachers with hands-on experience using the software.

Program issues. Given the issues with Internet connectivity, it is difficult to evaluate the 

strengths and weaknesses of the program. However, several issues have been identified 

repeatedly by students and teachers as impediments to students‘ progress through the episodes.

The first issue involves the talking speed of the characters speaking English in the stories 

and games. Teachers and students in the pilot study and full-scale study reported that it was 

sometimes difficult to understand stories and games because characters spoke too quickly to be 

understood. When asked about this after the pilot, the software development team reported that 

U.S. advisors had suggested that the characters speak at a speed that is typical for native English 

speakers. While this may be the ultimate goal, it seems that it would be prudent to incorporate a 

developmental approach to meeting this goal and allow students to initially select slower speeds 
Chapter 6. Summary and Implications

A Study of a Gaming Approach to English Language Learning for Middle School Students 41

and then increase the speed as they progress. At a minimum, it might be beneficial to reduce the 

speed of speech presented in the first few episodes.

Because of the number of new vocabulary words introduced to students in The Forgotten 

World, students and teachers expressed concern about the functionality of the dictionary that is 

available to students within the application. Some students expressed concern that it was 

inefficient and time-consuming to look up each word of a sentence they did not understand and 

suggested that students should be able to access complete translations of each sentence. While 

teachers thought that the eighth grade was the correct target grade for the program, many stated 

that eighth-grade students required support in understanding the new vocabulary and suggested 

that more efficient access to translations of new vocabulary or phrases be provided.

The extent to which translations should be easily accessible is a source of some debate.

Experts consulted by the software development team suggested that translations should be kept 

to a minimum so that students would not be able to rely on Chinese to work their way through 

the stories and games. Given the reaction of students and teachers, it might be advisable to adopt 

a developmental approach again and to provide complete translations of the initial few episodes 

so that students understand the basic story line. However, given that it appears that the lowerperforming students made the largest gains, this may not be necessary.

Finally, the extent to which students and teachers could adequately track progress 

through the program is unclear. One of the advantages of computer applications such as this is 

the ability to provide feedback on student progress to both students and their teachers. Because 

of the difficulty in U.S. servers‘ capturing information on usage, it is possible that feedback to 

students and teachers did not adequately portray students‘ progress through the program. This 

problem is relatively easy to fix. Capturing this information on local servers is probably a better 

option if and when large-scale adoption of The Forgotten World is considered and evaluation 

data are no longer necessary.

The statistically significant positive results of using The Forgotten World for the lowerperforming students along with the positive effects on student motivation is encouraging for an 

intervention that was only partially implemented. Further research could be conducted to help 

understand why the program produced these benefits, how to overcome implementation 

problems in schools lacking sophisticated computer technology, and why The Forgotten World

did not produce positive benefits for the initially higher performers.

Appendix B – The Forgotten World Quick Reference Guide

A Study of a Gaming Approach to English Language Learning for Middle School Students 43

References

Amori, M. A., E. F. Dalton, and P. Tighe. 2008. Technical manual: IPT1 reading and writing, 

grades 2–3, English forms 1C and 1D (2nd ed.). Brea, Calif.: Ballard and Tighe.

Dalton, B. W. 2010. Motivation. In J. A. Rosen, E. J. Glennie, B. W. Dalton, J. M. Lennon, and 

R. N. Bozick. Noncognitive skills in the classroom: New perspectives on educational 

research (RTI Press publication No. BK-0004-1009), 11–38. Research Triangle Park, N.C.:

RTI International. 

Maculaitis, J. D. 2001. The MAC II administration manual: Blue level, form A2. Brewster, N.Y.:

TASA Literacy.

Maculaitis, J. D. 2001. The MAC II administration manual: Blue level, form B2. Brewster, N.Y.:

TASA Literacy.

Norris-Holt, J. 2001. Motivation as a contributing factor in second language acquisition. Internet 

TESL Journal 7 (6): 1–4.

Rowan, B., and R. Correnti. 2009. Studying reading instruction with teacher logs: Lessons from 

the Study of Instructional Improvement. Educational Researcher 38:120–131.

A-1

Appendix A.

Open Language Learning Initiative Evaluation 

Planning Group Meeting Summary

A Study of a Gaming Approach to English Language Learning for Middle School Students A-3

Appendix A:

Open Language Learning Initiative Evaluation Planning Group Meeting Summary

Beijing, China

May 10-15, 2007

Participants:

U.S.A.

David Goodwin, U.S. Department of Education, Chairperson

Adriana de Kanter, U.S. Department of Education

Brian Fu, U.S. Department of Education

David J. Francis, University of Houston

Gary Cziko, University of Illinois

Yong Zhao, Michigan State University

Patricia Green, RTI International

ZhaoHong Han, Teachers College of Columbia University (attended Sunday only)

Steven Stoynoff, Minnesota State University (attended Sunday only)

China

Wang Wenjing, Beijing Normal University

Liu Dailin, China Central Radio and Television University (attended Saturday and Sunday)

Zeng Dawei, National Center for Education Technology (attended Saturday only)

Zhang Lianzhong, China National Institute of Educational Research (attended Sunday only)

The Evaluation Planning Group (EPG) met as part of a larger meeting to discuss the 

development and evaluation of the Open Language Learning Initiative (OLLI), English language 

learning software that is being developed by the Hewlett Foundation and Coastline Community 

College. This project continues the efforts that were initiated as part of the CHENGO project,

7

a 

joint partnership between the U.S. Department of Education and the Ministry of Education of the 

People‘s Republic of China to create a highly innovative R&D product to test the effectiveness 

of using online technology to deliver foreign language instruction within a real-world 

environment, consistent with theory of language learning. In keeping with the agreements 

between the two nations, the U.S. Department of Education intends to evaluate the effectiveness 

of the software for Chinese middle school students.

This document summarizes the key topics discussed by the EPG and presents their 

recommendations as well as concerns.

1. Key evaluation question

The EPG agreed that the evaluation should be designed to study the efficacy of OLLI, and that 

the key evaluation question is: 

 

7 Chengo is the name of the Chinese language learning program that was developed in China for U.S. 

middle school students.
Appendix A – Open Language Learning Initiative Evaluation Planning Group Meeting Summary

A-4 A Study of a Gaming Approach to English Language Learning for Middle School Students

Do students exposed to OLLI demonstrate increases in proficiency when compared to 

students without access to OLLI, given the same curriculum and under conditions that 

allow reasonable access to the software?

The group recommended that proficiency gains be measured over the course of a year rather than 

a single semester. It was also recommended that a field trial of the evaluation be conducted in the 

fall of 2008, with implementation of the full-scale study in 2009. The group agreed that it was 

likely that there will be some technical issues and other problems when the program is first 

introduced in the fall of 2008 and mounting the full-scale evaluation at that time would be 

unwise. The group concurred that it would be best to conduct a field trial of the evaluation in the 

fall of 2008, to ensure that the data collection protocols and assessments are consistent with they 

way OLLI is implemented in China.

2. Dimensions to be measured

OLLI is currently intended to focus on the improvement of students‘ speaking and listening 

skills; thus, the measurement of speaking and listening skills should be the primary focus of the 

evaluation. 

Reading and writing skills will also be assessed. These skills will be measured to determine 

whether (1) gains in speaking and listening enhance other language skills, or (2) gains in 

listening and speaking are made at the expense of gains in other areas. Reading and writing are 

emphasized in traditional classrooms, and skills in these areas are measured on local and national 

examinations. Thus, it is important to evaluate whether OLLI could have an unintentional 

detrimental impact by moving student effort away from these areas, or whether it might have a 

positive impact (or perhaps no impact) on reading and writing skills.

At some point, the EPG will need to have more detailed specification from the developers about 

the subskills of each dimension so that tests can be identified or developed that parallel the 

language skills emphasized by the program.

The EPG recognizes that OLLI may have an effect on student motivation to learn English, and it 

may also increase students‘ awareness of U.S. culture. However, the EPG agreed that those 

constructs are secondary and will be included in study protocols as time and resources permit.

3. Research design

Given the evaluation question, a random assignment study is recommended. The EPG discussed 

assignment at the school level and at the classroom level. Because students are expected to 

sometimes work with OLLI outside of regular school hours, the possibility of contamination 

across classrooms appears to be high. Thus, randomization at the school level seems to be a more 

prudent option. One member of the evaluation group also suggested that it consider a smaller 

group of matched-pair schools, and work more aggressively in these schools to ensure that OLLI 

is fully implemented.
Appendix A – Open Language Learning Initiative Evaluation Planning Group Meeting Summary

A Study of a Gaming Approach to English Language Learning for Middle School Students A-5

Textbooks and curriculum are probably major factors in students‘ English proficiency. This 

suggests that schools be selected from only one or two districts, so that these factors can be 

controlled.

The EPG will need the developers to specify minimum standards in a number of areas for 

schools to be included in the study. The standards should include the following:

 How many hours per week do students need to have access to a computer to make 

meaningful progress using OLLI?

 What are the minimum requirements for computers at participating schools?

 Are there minimum requirements concerning teacher training in the use of OLLI? Do 

teachers need a certain level of experience with IT? 

4. Assessments

The EPG discussed a number of existing assessments with members of the Language Learning 

and Pedagogy Working Group. Assessments that were discussed as possibilities included

Cambridge University Young Learners English Test (Cambridge University), Maculaitis 

Assessment of Competencies II, Test of English Language Proficiency (Touchstone Applied 

Science Associates), and IDEA English Proficiency Test (IPT) 1 and 2 (Ballard-Tighe). Other 

assessments also mentioned as having unique features that might be useful included Phone Pass

(Ordinate Technologies), the assessments developed for PANDA English (Beijing Normal 

University), Stanford English Language Proficiency Test (Harcourt Assessments), and the 

Woodcock Johnson Sentence Repetition subtest. (At a subsequent meeting, another test was 

discussed: the TOEIC Bridge). Problems were noted with each of the tests, including (1) some of 

the tests target TESOL standards rather than ACTFL standards, (2) some of the tests measure 

beginning English but content might be inappropriate because it is aimed at older or younger 

learners; and (3) some of the tests were developed for North American populations and may be 

culturally inappropriate for Chinese populations.

The group also discussed using English exams developed for the Beijing municipality, or exams 

used in the districts where the evaluation will be implemented. These exams would provide good 

measures of reading and perhaps writing proficiency. They would need to be supplemented with 

assessments directed at listening and speaking.

Developing specialized assessments for this project was also discussed. Some suggested that we 

could use the Cambridge Young Learners test as a model and adapt the subtests for the seventhand eighth-grade students that will be targeted in the evaluation; the test is currently 

recommended for students from ages 7 through 12. Others mentioned developing sentence 

repetition tasks and perhaps a task asking students to narrate a series of pictures. The pros and 

cons of natural language samples were also discussed.

The group concluded that a hybrid assessment would need to be used that incorporated

 an existing assessment for use as an anchor and for providing context for the other 

measures;
Appendix A – Open Language Learning Initiative Evaluation Planning Group Meeting Summary

A-6 A Study of a Gaming Approach to English Language Learning for Middle School Students

 a speaking task that includes sentence repetition and picture narration as well as 

straight pronunciation;

 a vocabulary assessment that uses the words that are common to both the national 

curriculum and the OLLI software (Both the comparison and treatment groups would 

have exposure to these words.); and

 an existing assessment developed at the district level.

The group also discussed the importance of the teacher‘s level of fluency in English and 

potential ways it could be measured. One potential way would be to record a short interview with 

the teacher about participation in the evaluation study.

5. Implementation of the evaluation

The EPG recommended that the evaluation take advantage of the new equipment being given to 

schools through the Modern Distance Education Project. As described by Zheng Dawei, this 

project will equip 35,000 middle schools in the western region with new computers. These 

computers will be set up in a computer lab with Internet access.

Twenty-five to 35 schools should be selected to be treatment schools, and the same number of 

schools should be selected to serve as the comparison group.

Students in the first two years of middle school will participate in the evaluation. A decision to 

include primarily seventh-grade students or eighth-grade students will await more detailed 

information on the content of OLLI and selection of districts and an examination of the English 

curriculum in the district.

Chinese scholars consistently expressed concern that teachers will be reluctant to devote 

instructional time to OLLI because of the importance of student performance on exams. They 

also were skeptical about the amount of time parents will allow students to use OLLI at home.

One suggestion was that perhaps time in afternoon classes (devoted to IT or social practice) 

could be allocated to practice using OLLI. The EPG looks forward to learning more details about 

the teacher materials and courseware being developed for OLLI.

The EPG suggested consideration of implementing the evaluation in boarding schools, where 

conditions could be more carefully controlled. In the western regions of China, many of the 

middle schools are boarding schools so that students do not have to travel great distances daily.

If these schools have computer laboratories, it might be possible to arrange for students to have 

greater access to computers (at both comparison and treatment schools) than would be possible at 

day schools.

The study will be designed to have a pretest and posttest. While the majority of the test will be 

administered to the whole class, individual assessments of student speaking will be necessary.
Appendix A – Open Language Learning Initiative Evaluation Planning Group Meeting Summary

A Study of a Gaming Approach to English Language Learning for Middle School Students A-7

6. Roles and responsibilities

It was understood that the Ministry of Education has volunteered to be responsible for 

identifying appropriate schools for the evaluation of the project.

It will be necessary to identify the organization that will be responsible for providing the 

software to schools, along with orientation, training, and maintenance. The National Center for 

Education Technology may play this role.

7. Next steps

The United States will partner with an organization in China to conduct the evaluation. The 

Chinese partner will consult on the design of the assessment, review instruments to make sure 

that they are appropriate, and collect the data at the schools.

It would be useful to obtain more information about the districts, schools, and students that might 

be selected to participate in the evaluation project. We would like data on numbers of schools, 

numbers of classes in schools, and number of students within classes. It would also be useful to 

know when students currently begin English language instruction in these districts. Also, if data 

are available, it would be valuable to obtain data from recent districtwide English examinations.

These data would allow us to estimate the size of the typical gain in proficiency over the course 

of a year and would allow us to examine the variability within and between schools and classes.

B-1

Appendix B.

The Forgotten World Quick Reference Guide

B-3

QUICK REFERENCE GUIDE

Appendix B – The Forgotten World Quick Reference Guide

A Study of a Gaming Approach to English Language Learning for Middle School Students B-5

TABLE OF CONTENTS 

GETTING STARTED.............................................................................................................. B-7

TECHNICAL FAQ................................................................................................................. B-13

FREQUENTLY ASKED QUESTIONS..............................................................................B-13

THE FORGOTTEN WORLD STORY SUMMARIES ...................................................... B-15

EPISODE 1: FLIGHT TO AMERICA................................................................................B-15

EPISODE 2 AND 3: LOS ANGELES, CALIFORNIA ......................................................B-15

EPISODE 4: SAN FRANCISCO, CALIFORNIA ..............................................................B-16

EPISODE 5: DENVER, COLORADO ...............................................................................B-16

EPISODE 6: ROSWELL, NEW MEXICO .........................................................................B-16

EPISODE 7: DALLAS, TEXAS .........................................................................................B-17

EPISODE 8: NEW ORLEANS, LOUISIANA....................................................................B-17

EPISODE 9: DETROIT, MICHIGAN ................................................................................B-17

EPISODE 10: CHICAGO, ILLINOIS.................................................................................B-18

EPISODE 11: WASHINGTON, D.C. .................................................................................B-18

EPISODE 12: NEW YORK, NEW YORK .........................................................................B-18

THE FORGOTTEN WORLD GAME DESCRIPTONS .................................................... B-19

HOW TO ACCESS FILES ON THE TEACHER’S HUB.................................................. B-29

Appendix B – The Forgotten World Quick Reference Guide

A Study of a Gaming Approach to English Language Learning for Middle School Students B-7

GETTING STARTED

PARTS OF A COMIC PANEL (SCREEN)

This teacher aid is a reference for you, the teacher, so you can show your students the parts 

of The Forgotten World comic panels and their functions. This information can also be 

found in the Teacher’s Guide.

Log into The Forgotten World by typing 

in the URL 

http://21xp.com/english

In the beginning, students will type 

in their Username and Password 

and then click the Sign In button.

Students will see the Loader icon 

when the game is loading. Tell the 

students that they should wait until 

this icon goes away and the panel 

loads.
Appendix B – The Forgotten World Quick Reference Guide

B-8 A Study of a Gaming Approach to English Language Learning for Middle School Students

Below, is a typical comic panel (screen) in The Forgotten World. There are many parts to 

the panel that students should become familiar with so they can feel confident navigating 

the game. These parts are highlighted in red and numbered here, but each part separately 

will be described separately on the next three pages.
Appendix B – The Forgotten World Quick Reference Guide

A Study of a Gaming Approach to English Language Learning for Middle School Students B-9

1. The Welcome Student text in the 

upper right-hand corner of the 

panel indicates that the student is 

logged into The Forgotten World.

2. The Logout button in the upper 

left-hand corner of the panel is 

where students can click to log out 

of the game.

3. In the lower left-hand corner of the 

panel, students will see the 

episode/page indicator. This will 

tell them what episode and what 

page they are on.

4. Once every dialog bubble on the 

page plays, the students may move 

on to the next comic panel by 

clicking the Next button. Students 

may also go back to the previous 

comic panel by clicking the Back 

button.

5. When the students roll over a 

dialogue bubble with their mouse, 

they may see certain words 

highlighted in green. These green 

words can be found in the 

Dictionary.
Appendix B – The Forgotten World Quick Reference Guide

B-10 A Study of a Gaming Approach to English Language Learning for Middle School Students

6. When students see that a part of the 

screen is outlined in blue, that 

means a game is present. Students 

should click on the glowing blue 

area to launch the game. 

7. In the lower right-hand corner of the 

screen, students will see the PDA. 

Students may click on the PDA to 

open the PDA menu.

Inside the PDA menu, students will see 

icons:

Episodes

Dictionary

Game

Credits

If students click on the Episodes icon 

on the PDA menu, they will see a list of 

the episodes they have already 

entered. They will not be able to see 

the episodes they have not entered. 

Students can go back to the episodes 

they have entered by clicking on an 

episode icon in the list.
Appendix B – The Forgotten World Quick Reference Guide

A Study of a Gaming Approach to English Language Learning for Middle School Students B-11

Students may click on the Dictionary

icon on the PDA menu to open the 

dictionary and search for words. They 

can read the definition, listen to the 

word aloud, and listen to the word 

phonetically.

Students can navigate to a particular 

game by clicking the Game icon in the 

PDA menu.

Students can watch a short tutorial 

about how to navigate The Forgotten 

World by clicking on the Help icon in the 

PDA menu.

Appendix B – The Forgotten World Quick Reference Guide

A Study of a Gaming Approach to English Language Learning for Middle School Students B-13

TECHNICAL FAQ

FREQUENTLY ASKED QUESTIONS

1. After pressing Next, I get a gray Zuka image and nothing happens. What do I do?

This may be due to a problem connecting to the front-end, and/or back-end server.

a) Note the current page number on the bottom left of the screen (Ep X, Comic Page X)

b) Close the browser and open a new one

c) Navigate back to the application

d) Log in

2. After playing a game, I get a gray Zuka image and nothing happens. What should I do? 

Please follow FAQ #1

3. I want to play a game. How do I do that?

a) Click on the PDA Icon at the lower right of the screen

b) Click on the Games Icon

c) Select the desired game

4. How do I get The Forgotten World to be full-size on my screen?

a) Press F1 (function key)

I want to play a specific game that I've already encountered. How do I do that?

b) Click on the PDA Icon at the lower right of the screen

c) Click on the Games Icon

d) Select the desired game

Appendix B – The Forgotten World Quick Reference Guide

A Study of a Gaming Approach to English Language Learning for Middle School Students B-15

THE FORGOTTEN WORLD STORY SUMMARIES

The complete Teacher’s Guide, scripts for each episode and student worksheets are found 

on the enclosed CD-ROM.

EPISODE 1: FLIGHT TO AMERICA

Story Summary: We meet Uncle Jonesy, a scientist, as he is packing up his laboratory. He 

and his niece, Lia, are returning to the United States of America. Lia has made friends with 

one of your students, and she has invited him/her to accompany her to the United States. 

The student has arranged to meet Lia at the airport and board the plane to Los Angeles. 

While boarding the plane, the student is exposed to both formal and informal greetings. 

During the flight, Lia invites her friend to play video games.

梗概∶一场爆炸之后,我们在Jonesy叔叔收拾实验室时见到了他。Jonesy叔叔是一位科学

家。 Jonesy 

和他的侄女,Lia即将返回美国。Lia和你的一位学生交上了朋友。她已邀请他/她

陪她到美国。这名学

生已安排在机场去接Lia,并将飞往洛杉矶。在登机过程中,这名学生受到正

式和非正式的问候。在飞行期间,Lia邀请她的朋友玩视频游戏。

EPISODE 2 AND 3: LOS ANGELES

Summary: Lia and her friend (your student) arrive at Los Angeles International 

Airport, but they cannot find Uncle Jonesy anywhere. They decide to go to his home, but 

Uncle Jonesy is not there. They call his assistants, Dave and Ray, to help them. While 

unpacking, Lia discovers a mysterious round ball—Zuka. When touched, it tries to 

communicate with them via symbols. Ultimately the players discover that Zuka wants 

them to go to San Francisco, but how will they get there? In Episode 3, Grace arrives and 

says she will drive everyone to San Francisco in her Recreational Vehicle (RV), but first 

they need to stop for supplies.

梗概:

Lia和她的朋友(你的学生)抵达洛杉矶国际机场,但他们到处找不到Jonesy叔叔

。他们决定去 他家,但Jonesy叔叔不在家。

他们打电话叫来他的助手Dave和Ray来帮助他们。打开行李时,利亚发现一个神秘

的圆球—Zuka 

当有人触动它时,它会试图通过符号与人交流。最后他们发现Zuka希望他们去

旧金山,但他们怎么能 到那儿呢?

在第3集中,Grace赶来了,说她会驾着她的运动休闲车(RV),带大家到洛杉矶

,但首先他们需要补充给养。
Appendix B – The Forgotten World Quick Reference Guide

B-16 A Study of a Gaming Approach to English Language Learning for Middle School Students

EPISODE 4: SAN FRANCISCO

Summary: Lia, Ray, Dave, and Grace finally arrive in San Francisco. Zuka tells them to find 

“The Harmony Shop” and Dr. Hu. Dr. Hu gives them more history about Zuka, teaches them relaxation 

exercise, and then sends them on the task to collect potion bottles that will keep them 

safe from their enemies—the Creepers.

梗概: Lia,

Ray,Dave和Grace一行最后抵达旧金山。Zuka告诉指示他们找到“和谐商店”和

胡博士。胡博士给他们讲述了Zuka的由来,并教他们做放松训

练,然后给他们布置了收集魔水瓶的任务,魔水瓶将保证他们免受他们的敌人

- -食人族的侵害。

EPISODE 5: DENVER, COLORADO

Summary: The group is being led by Zuka across the Rocky Mountains to Denver, Colorado. 

They hope that Jonesy is there waiting for them. Once there, they go to NOAA, a national 

weather station, and then the Creepers arrive on the scene and the gang eludes them again. 

The gang searches for the key.

梗概:

Lia一行由Zuka带领,穿过落基山脉到达科罗拉多州丹佛市。他们希望Jones

y在那里等着他们。一到那里,

他们就去国家气象台。紧接着食人族也到了那里。他们又一次成功地摆脱

了食人族。他们到处在搜寻钥匙。

EPISODE 6: ROSWELL, NEW MEXICO

Summary: As the group heads for Roswell, New Mexico, Dave explains some of the rumors 

surrounding the city. After passing through security, Zuka leads everyone to a cave, where they 

find and release Uncle Jonesy. Before they escape, he explains about the earth‘s ―life force‖ 

energy and that the Creepers seek this energy.

梗概:当小分队向新墨西哥州罗兹韦尔市挺近时,Dave向他的伙伴们解释

了有关这个城市的一些谣言。通过安全检查之后,Zuka把大家带到一个山

洞,他们在那里找到并解救了Jonesy叔叔。在他们逃跑时,他解释有关地

球的―生命力‖的能源,并说食人族也在寻找这种能源。
Appendix B – The Forgotten World Quick Reference Guide

A Study of a Gaming Approach to English Language Learning for Middle School Students B-17

EPISODE 7: DALLAS, TEXAS

Story Summary: The group travels to Dallas and attends a Rodeo. The Rodeo is not quite 

right. The weather conditions continue to be out of the ordinary. Information is gathered 

that the Creepers are building a water capture machine.

梗概:该小组将前往达拉斯去观看当地的牛仔竞技。今年的比赛不太好看。

天气还是很反常。他们收集的情报显示,食人族正在建设一个取水机。

EPISODE 8: NEW ORLEANS, LOUISIANA

Story Summary: The group goes to New Orleans. They arrive in the middle of a Mardi Gras 

celebration. Jonesy tells them about strange activities by the local river. The city is full of 

Creepers!

梗概:Lia一行四人去新奥尔良。他们在狂欢节正酣时到达该市。Jonesy给

他们讲述当地河流的奇异之处。这个城市到处都是食人族!

EPISODE 9: DETROIT, MICHIGAN

Story Summary: The group arrives in Detroit to investigate an abandoned car factory. As 

they heard in New Orleans, the city is teeming with Creepers. As they investigate, they find 

what the Creepers are doing there.

梗概:

Lia一行抵达底特律调查一座废弃的汽车工厂。正如他们在新奥尔良所听说

的那样,这个城市到

处都是食人族。通过调查,他们发现了食人族正在做什么。
Appendix B – The Forgotten World Quick Reference Guide

B-18 A Study of a Gaming Approach to English Language Learning for Middle School Students

EPISODE 10: CHICAGO, ILLINOIS

Story Summary: The group continues to chase the Creepers. Their next destination is at 

the Sears Tower in Chicago, Illinois (“The Windy City”). Zuka is trapped in the tower. They 

save Zuka, but he is very sick. They must harness the Earth’s life force to heal Zuka. This life 

force is at four power centers.

梗概: Lia他们一行继续追赶食人族。

他们的下一个目的地是坐落在被称为―风城‖的芝加哥市的希尔

斯大厦。Zuka 被困在大厦里。

他们救出了Zuka,但他病得很厉害。他们必须利用地球生命力量来Z

uka治病。地球生命力来自四个力量中心。

EPISODE 11: WASHINGTON D.C.

Story Summary: They find out that Washington D.C. has important clues and race to get 

there before the Creepers do. Zuka feels better after touching water which becomes one of 

the four power centers. As the journey continues, Lia learns more information about her 

family history.

梗概:他们在华盛顿特区发现重要线索,并在食人族到达之前飞速赶

到那里 。 Zuka在摸了四个力量中心之一的水之后,感觉好多了。

随着旅程的继续,Lia了解到更多有关她家族 历史的情况。

EPISODE 12: NEW YORK, NEW YORK

Story Summary: The group goes into New York City to find the other three power centers. 

Zuka transforms into a beautiful being with more power. They go to the Statue of Liberty to 

synthesize the powers and defeat the incoming Creepers, saving the world!

梗概:这个小组进入纽约市去找其他三个力量中心。

Zuka得到更多的力量之后,变得更加漂亮。他们

到自由女神像去合成这四种力量,打败了正在侵入的食人族,并拯救世

界!
Appendix B – The Forgotten World Quick Reference Guide

A Study of a Gaming Approach to English Language Learning for Middle School Students B-19

THE FORGOTTEN WORLD GAME DESCRIPTONS

Descriptions of the games that occur in the Forgotten World are listed below in 

alphabetical order, along with their occurrences within the episodes. These 

descriptions can also be found in Appendix A in the Teacher’s Guide.

Floating Phrases Game: The challenge for players is to construct valid sentences from 

a series of phrases that are floating across the screen. The players will need to understand 

the meaning of the words as well as the sequences in which the phases must be placed.

Episodes: 1, 5, 11

Directions: Drag the phrases to the empty spaces at the bottom of the screen. Put three 

phrases in the correct order to make an English sentence. When all of the spaces are filled, 

click the Submit button to check your work. To win the game, make five different sentences 

before the time runs out.

将短语拖进屏幕下方的空白处。将三个短语按照正确的顺序进行排列组合,

最后组成一个英文句子, 所有空白处被填满后,请点击提交按钮

(Submit)检查句子是否正确。若能在规定时间内完成五个不

同的句子,就能赢得本节比赛。
Appendix B – The Forgotten World Quick Reference Guide

B-20 A Study of a Gaming Approach to English Language Learning for Middle School Students

Map Game: The player’s challenge is to guide other characters in the story through a map 

or maze to collect items and, ultimately, to reach a final target destination. The player does 

this by constructing a series of directions from a set of words and receiving written and 

audio feedback from the character who is receiving the directions.

Episodes: 4, 6, 8 

Directions: Give Grace the directions she needs to find the bottles of potion. Click on one of 

the tabs at the top of the screen to select the type of sentence you want to use. Then, select 

words from the drop-down menus to complete the sentence. Click on the green button to 

hear your sentence and give the instruction to Grace. Avoid the creepers or you will lose 

the bottles you’ve collected. After you collect five bottles of potion, you can help Grace find 

the clue or you can collect more bottles to earn more points. Find the final clue to win the 

game.

给Grace提供如何找到魔药瓶的指南。点击屏幕上方其中的一个图标来选择你

想使用的句子类型。然后

再从下拉菜单中选出适合的单词来完成该句子。点击绿色按钮来听句子发音

,并给Grace发出指令。 你

尽量避开爬行者,否则将会失去你刚刚搞到手的魔药瓶。当你拥有五个魔

药瓶后你就可以帮助Grace找 到线索,

或者可以收集更多的魔药瓶来获取加分。找到最终的线索就可以闯过此关

。

San Francisco, California Map Game (Episode 4)
Appendix B – The Forgotten World Quick Reference Guide

A Study of a Gaming Approach to English Language Learning for Middle School Students B-21

Roswell, New Mexico Map Game (Episode 6)

New Orleans, LA Map Game (Episode 8)
Appendix B – The Forgotten World Quick Reference Guide

B-22 A Study of a Gaming Approach to English Language Learning for Middle School Students

Phoneme Game: In the phoneme game, Zuka’s memory banks have become scrambled. As 

a result, Zuka has forgotten how to pronounce certain English words. The player must 

restore Zuka’s memory banks and its ability to communicate by unscrambling its memory 

and teaching it how to pronounce a series of words.

Episodes: 9, 12

Directions: Arrange the sounds correctly to teach Zuka how each word is pronounced. 

Click on the Audio button on the left to hear the sounds. Drag the sounds to place them in 

the correct order for each word. Complete the sound arrangement correctly for all five 

words to win the game.

将语音正确排序来教Zuka每一个词的发音。点击左侧的声音(Audio)

按钮听音,然后将不同音拖到正

确的位置,将它们组合成单词的发音。若能正确完成五个单词的语音

排列即可赢得此节
Appendix B – The Forgotten World Quick Reference Guide

A Study of a Gaming Approach to English Language Learning for Middle School Students B-23

Shopping Spree Game: The game is designed to test listening comprehension in a 

simulated, real-world environment. The player’s challenge is to locate a variety of items in 

a store by listening to audio clues, which are presented as snippets of conversations from 

different areas of the store.

Episodes: 3, 8

Directions: Use the arrow keys on the keyboard to locate the items on your shopping list. 

Listen to the clues to help figure out where each item is located. When you find an item on 

your list, click on it and then check off the item on your list to add it to your shopping cart. 

To win the game, find all five items on your list before the time runs out.

用键盘上的方向键来定位你购物清单上的所有物品,请仔细听语音提示,以

便确定每个物品所处的正

确位置。请仔细收听语音提示信息,在找到任何购物单上的商品后,点击鼠

标左键将其从购物单清除, 

放入购物车。要在规定时间内找全所有五件商品才能赢哦!

Shopping Spree (Episode 3)
Appendix B – The Forgotten World Quick Reference Guide

B-24 A Study of a Gaming Approach to English Language Learning for Middle School Students

Shopping Spree (Episode 8)
Appendix B – The Forgotten World Quick Reference Guide

A Study of a Gaming Approach to English Language Learning for Middle School Students B-25

Story Recap Game: Designed to test reading and writing comprehension by asking players 

to briefly summarize key events they have encountered throughout the story. The player 

listens to the dialog for each event and then places the series of events in the sequence in 

which they occurred in the story.

Episodes: 3, 6, 9, 12

Directions: Click on the pictures to hear what each person said during the story. Then, put 

the words in the correct order to match what the person said and click the Done button to 

check your work. When all five people have the correct words, drag the pictures to place 

them in the order that these things happened in the story.

点击图片来听故事中每个人物的谈话。然后将单词其按正确的顺序排

列组合,使之同故事中每个人物 的谈话相吻合。最后点击完成
Appendix B – The Forgotten World Quick Reference Guide

B-26 A Study of a Gaming Approach to English Language Learning for Middle School Students

Tri Words Game is used as the pre-assessment. The game allows players to test their 

understanding of a wide variety of vocabulary by matching spoken audio to written words 

and phrases, as well as to symbols and images.

Episodes: 1, 7, 11

Directions for Pre-Assessment: Your goal is to clear the board of all triangles by matching 

sets of three or more triangles that are next to each other. To create a match, click on a 

triangle to see its Mandarin word. Then, find the word in English as it moves across the 

screen and drag it to the correct triangle to create a match. Match sets of three or more 

adjacent triangles to clear them from the board. Clear the entire board to win the game.

本节的目标是通过把三个或三个以上相邻的三角形连在一起来清除屏幕上所

有的三角形。首先你要点 击一个三角形,就能看见一下其中的汉语词语。

然后从在屏幕上滚动的英语单词中寻找与汉语词语匹

配的英语单词,并将其拖到正确的三角形里,与相应的汉语词语匹配。,完

成三个或三个以上相邻的

三角形的匹配即可将它们从屏幕上清除,清除屏幕上所有三角形后即可赢

得本节比赛。

Directions: Your goal is to clear the board of all triangles by matching sets of three or more 

triangles that are next to each other. To create a match, click on a triangle to hear its word. 

Then, find a matching word or picture as it moves across the screen and drag it to the 

triangle. Match sets of three or more adjacent triangles to clear them from the board. Clear 

the entire board to win the game.

本节的目标是通过把三个或三个以上相邻的三角形连在一起来清除屏幕上所

有的三角形。首先点击一

个三角形来听单词读音,然后屏幕上会出现该单词或者图片,把对应的词语

或者图片拖进三角形里。

完成三个或三个以上相邻的三角形的匹配,即可将它们从屏幕上清除,清除

屏幕上所有三角形后即可 赢得本节比赛。
Appendix B – The Forgotten World Quick Reference Guide

A Study of a Gaming Approach to English Language Learning for Middle School Students B-27

Zuka Symbol Puzzle Game: The player must communicate with Zuka using 

symbolic icons that represent words and concepts to construct a series of English 

phrases. It encourages comprehension of English by “constructing” communication Ǥ

Episodes: 2, 4, 10
Appendix B – The Forgotten World Quick Reference Guide

B-28 A Study of a Gaming Approach to English Language Learning for Middle School Students

Directions: Click on the buttons at the left of the screen to hear each sentence. Then, 

drag the tiles from the top of the screen into the empty spaces to create each sentence in 

symbols and words. When all of the spaces are filled, click the Submit button to check 

your work. If any sentences are not correct, move those tiles to the correct positions 

and click Submit again. To win the game, use all of the tiles to complete all five 

sentences correctly.

点击屏幕左侧的按钮来听每一个句子。然后将屏幕上方的方砖拖至空白处,用符号和单词

来造句。当 所有空白处被填满后,点击提交按钮

(Submit)检查句子是否正确。如句子错误,重新将方砖移到正

确的位置再次提交。在贴完所有的方砖,正确完成五个句子之后,你将赢得这场游戏

!
Appendix B – The Forgotten World Quick Reference Guide

A Study of a Gaming Approach to English Language Learning for Middle School Students B-29

HOW TO ACCESS FILES ON THE TEACHER’S HUB

tfwteachers.com

Teachers will find the following information on the Teachers’ Hub:

Class Progress Reports

Individual Student Progress Reports

The Teachers Guide

Simplified Scripts of The Forgotten World Episodes

Sound Clips from Episodes

To go to The Forgotten World Teachers’ Hub:

Step 1: Open a web browser and type in http://www.tfwteachers.com

Step 2: Log in with your user name and password.

Step 3: Click on “My Class”, and view your students’ progress.
Appendix B – The Forgotten World Quick Reference Guide

A Study of a Gaming Approach to English Language Learning for Middle School Students B-30

Step 4: To see only one student’s progress, click on that student name in your class list.
Appendix B – The Forgotten World Quick Reference Guide

A Study of a Gaming Approach to English Language Learning for Middle School Students B-31

Step 5: To download copies of the Teachers Guide, scripts, sound 

files or other teacher content, click on Downloads in the menu.

Step 6: When finished, please click “Log OutǤ”

Distributed by:

Coastline Community College

11460 Warner Avenue

Fountain Valley, CA 92708

Telephone: (714) 241-6109

Fax: (714) 241-6286

E-mail: CoastLearning@coastline.edu

Copyright 2009 by the Coast Community College District

Permission to use The Forgotten World Teachers Guide is granted under a Creative Commons 

Attribution Share Alike 3.0 United States License.

If there are any questions regarding use and/or additional teacher resources, contact Coastline 

Community College (714) 241-6109.

Funded in part by a grant from the William and Flora Hewlett Foundation.

Coastline Community College District

Trustees 

Mary L. Hornbuckle Walter G. Howald Jim Moreno

Jerry Patterson

Lorraine Prinsky

Interim Chancellor Ding-Jo Currie, Ph.D.
Appendix B – The Forgotten World Quick Reference Guide

A Study of a Gaming Approach to English Language Learning for Middle School Students B-32

Coastline Community College

Evangeline N. Meneses, Ed.D., Acting President

Dan C. Jones, Executive Dean, Center for Instructional Systems Development Ted Boehler, 

Ed.D., Dean, Learning Technology & Innovation

Laurie R. Melby, Director of Video & Media Production

Lynn M. Dahnke, Director, Marketing & Partnerships

Robert D. Nash, Director, Instructional Design & Faculty Support Judy Garvey, Director, 

eMedia & Publishing

Dave Thompson, Director, eLearning Research & Development

OLLI Team

Coastline Community College Chris Mefford

Michelle Nieman

Jingfang Satow

Meg Yanalunas

Learning Games Network Alex Chisholm

Sherri Wright
C-1

Appendix C.

English Proficiency Tests Under Consideration, 

October 2008 

Appendix C – English Proficiency Tests Under Consideration, October 2008

A Study of a Gaming Approach to English Language Learning for Middle School Students C-3

Continued on next page...

Categories of 

Information

ACTFL Oral Proficiency 

Interview

OPI; OPIc; OPIc Jr.

IPT 2004 

(IDEA Proficiency Test)

The MAC II 

(Maculaitis Assessment of 

Competencies II Test of 

English Language 

Proficiency) 

Maker and Web 

site

ACTFL

http://www.actfl.org/i4a/pages

/index.cfm?pageid=3642

Ballard Tighe

http://www.ballardighe.com/products/la/

Questar Associates, Inc.

http://www.questarai.com/catal

og_products/mac2_main.htm

Level of English 

assessed

 OPI: 10 proficiency 

ratings from Novice Low 

to Superior

 OPIc: 9 ratings from 

Novice Low to 

Advanced.

 OPIc Jr.: 6 ratings from 

Novice Low to 

Intermediate High. 

(According to the ACTFL 

Testing Office, OPIc Jr. 

is designed because of 

the demand from the 

Asian market. It is 

currently used in Korea 

and the self-administered 

background survey that 

precedes the test is in 

Korean. A US version is 

still in development and 

its expected release is in 

November 2008)

For Oral Tests:

 Pre-IPT (ages 3, 4, and 5)

 IPT I (grades K-6)

 IPT II (grades 7-12)

For Reading and Writing 

Tests:

 Pre-IPT (ages 3, 4, and 5)

 IPT 1 (grades 2-3)

 IPT 2 (grades 4-6)

 IPT 3 (grades 7-12)

For OLLI, IPT I seems 

appropriate for the oral test. 

For the reading and writing 

tests, IPT 1 or IPT 2 seem 

appropriate.

For testing:

 Red (grades K-1)

 Blue (grades 2-3)

 Orange (grades 4-5)

 Ivory (grades 6-8)

 Tan (grades 9-12)

For OLLI, Blue or Orange 

levels seem appropriate.

English Competency Levels 

and standard score are 

reported. E.g. For the speaking 

test:

 Basic beginner (score 

below 143 for orange; 

below 153 for blue)

 Beginner (143-162 for 

orange; 153-173 for blue)

 Low intermediate (163-

185 for orange; 174-194 

for blue)

 High intermediate (186-

215 for orange; 195-220 

for red)

 Advanced (above 215 for 

orange; above 220 for 

blue)

Skill areas 

assessed1

Speaking and oral proficiency 1. Oral: both speaking and 

Listening (individual); prerecorded examiner is not 

recommended because 

the test requires 

interactions.2

2. Reading (group)

3. Writing (group)

1. Speaking (individual); 

examiner to rate 

responses immediately

2. Listening (group); 

audiotapes can be 

purchased for the Blue, 

Orange, Ivory, and Tan 

levels

3. Reading (group)

4. Writing (group)
Appendix C – English Proficiency Tests Under Consideration, October 2008

C-4 A Study of a Gaming Approach to English Language Learning for Middle School Students

Continued on next page...

Continued from previous page.

Categories of 

Information

ACTFL Oral Proficiency 

Interview

OPI; OPIc; OPIc Jr.

IPT 2004 

(IDEA Proficiency Test)

The MAC II 

(Maculaitis Assessment of 

Competencies II Test of 

English Language 

Proficiency) 

Potential to 

measure growth

 Only proficiency ratings 

are offered but the 

ACTFL Testing Office 

claims that OPIc is used 

regularly to measure pre 

and post semester 

testings

 Alternate test forms to 

ensure reliability of scores 

between testing (Forms E 

and F for oral tests and 

Forms C and D for 

reading/writing tests)

 The IPT oral tests 

provides score levels (raw 

scores can be derived)

 The IPT reading tests 

generate percentiles, 

Normal Curve Equivalent 

scores, and Standard 

scores (raw scores can be 

calculated)

 The IPT writing tests use 

a four-point rubric for 

scoring

 Alternate test forms are 

available at the testing 

level for pre- and posttesting

 Raw score and converted 

standard score can be 

tallied for each student 

and for each skill area

Time required 

for 

administration

30-40 minutes (20-30 

minutes for the test and 10 

minutes for background 

survey and self-assessment)

 Oral (individual): untimed, 

but 15-20 minutes, scored 

as students are being 

tested.

 Reading (group): untimed, 

but usually 60 minutes

 Writing (group): untimed, 

but usually 35 minutes

 Speaking (individual): 6 

min

 Listening (group): 20-30 

min

 Reading (group): 35-50 

min

 Writing (group): 15 min for 

blue; 40-45 min for higher 

levels

Special 

requirements for 

administration

OPIc and OPIc Jr.: Web 

accessibility and high speed 

connection, headsets with 

microphones

The oral test requires a live 

examiner.

Speaking test requires a live 

examiner.

Appropriateness 

for age group

OPI and OPIc: General 

audience

OPIc Jr.: K-8 students

Pre-K to 12 Pre-K to 12
Appendix C – English Proficiency Tests Under Consideration, October 2008

A Study of a Gaming Approach to English Language Learning for Middle School Students C-5

Continued on next page...

Continued from previous page.

Categories of 

Information

ACTFL Oral Proficiency 

Interview

OPI; OPIc; OPIc Jr.

IPT 2004 

(IDEA Proficiency Test)

The MAC II 

(Maculaitis Assessment of 

Competencies II Test of 

English Language 

Proficiency) 

Cultural 

appropriateness

 The Asian version of the 

OPIc Jr. lets the testers 

identify themselves in 

four school levels: 

Preschool/Kindergarten, 

Grades 1-3, Grades 4-6, 

and Middle School. The 

U.S. version of the OPIc 

Jr. is being released at 

the end of this year, but

the school levels are 

Elementary, Middle, and 

High schools.

 Based on the (demo) 

Asian version of the 

OPIc Jr., the background 

survey is geared toward 

the Korean audience 

(e.g. mention of afterschool academy; 

activities), but the ACTFL 

Testing Office claims that 

they can make 

adjustments and also 

make the Chinese 

instructions available.

 Some test questions are 

asked in a row. The 

ACTFL Testing Office 

explains that the test 

does not intend for the 

test-taker to be able to 

answer all the questions. 

Their proficiency is 

measured by the details 

that they can provide at 

any of the questions 

read, even when they 

say, ―I can’t answer the 

rest of the questions 

because I did not 

remember what they 

were.‖ Such a statement 

may demonstrate their 

oral proficiency.

Based on the sample tests, 

two issues were identified:

 Store varieties: Pet store

 Children’s activities: boy 

goes to the post office, 

eating popcorn and lying 

on the floor in front of TV

Based on the sample test, 

three issues were identified:

 Food items: bread and 

milk

 Pictures: few Asians and 

some everyday objects 

may look different in China 

(e.g. a school desk)

 Person names: some 

Spanish names
Appendix C – English Proficiency Tests Under Consideration, October 2008

C-6 A Study of a Gaming Approach to English Language Learning for Middle School Students

Continued from previous page.

Continued on next page...

Categories of 

Information

ACTFL Oral Proficiency 

Interview

OPI; OPIc; OPIc Jr.

IPT 2004 

(IDEA Proficiency Test)

The MAC II 

(Maculaitis Assessment of 

Competencies II Test of 

English Language 

Proficiency) 

Cost Testing, scoring, and 

customer support:

 $25 per test for OPIc Jr.; 

volume discount is 

available

 $50 per test for OPI

Testing:

 In-service training kit for 

IPT: $135

 IPT I oral test set (50 

booklets): $185

 IPT I oral test alternate 

forms (50): $51

 IPT I oral test diagnostic 

answer sheets (50): $51

 IPT I oral test scannable 

answer sheets (50): $51

 IPT 1 reading/writing test 

set (50): $228

 IPT 1 reading/writing 

alternate test booklets 

(50): $81 for reading and 

$60 for writing

Scoring:

 Hand score

 Use own scanning 

equipment

 Contract with a scanning 

service

Testing:

 Classroom set for a class 

of 25: $180 for red/blue; 

$239 for higher levels

 Administration manual: 

$45

 Test booklets: $105 per 25 

for red/blue; $138 per 25 

for higher levels

 Pictures booklet (red/blue 

only): $18

 Answer sheets (orange, 

ivory, tan): $21 per 25

 Writing forms (orange, 

ivory, tan): $29 per 25

 Listening test audiotapes: 

$27

 Student record forms: $32 

per 25

Scoring:

 Scoring services and 

rating of open-ended 

writing items

o Minimum charge: 

$250

o Per student: $3.62 for 

red/blue; $2.57 for 

orange, ivory, or tan

 Hand scoring 

o EZ Converter: $65 per 

form

o Scoring keys: $19

o Handbook with norms 

tables: $49

Option to adapt 

instrument

OPIC Jr.

 The background survey 

determines the test 

questions. For the OLLI 

audience, a uniform 

background survey 

needs to be used. 

 A button is available for 

the test takers to repeat 

the question once, but 

they cannot go back to a 

previous question.

According to psychometrician 

George Seretis of Ballard 

Tighe (Maker of the IPT), 

minor adaptation is possible as 

long as the test item remains 

in context. However, if the 

change is likely to change the 

item, it is better to simply skip 

it and give the student the 

credit.

TBD

Vocabulary 

overlap with 

OLLI

OLLI is developed based on 

ACTFEL Standards and 

ELLS E-Learning Standard 

(to raise speech acquisition 

from Novice Low and Novice 

High)

To be assessed To be assessed
Appendix C – English Proficiency Tests Under Consideration, October 2008

A Study of a Gaming Approach to English Language Learning for Middle School Students C-7

Continued from previous page.

Categories of 

Information

ACTFL Oral Proficiency 

Interview

OPI; OPIc; OPIc Jr.

IPT 2004 

(IDEA Proficiency Test)

The MAC II 

(Maculaitis Assessment of 

Competencies II Test of 

English Language 

Proficiency) 

Vocabulary 

overlap with 

curriculum

? Need curriculum materials To be assessed with 

curriculum materials

To be assessed with 

curriculum materials

Estimated effect 

size over one 

year

Information unavailable According to psychometrician 

George Seretis, they did a 

study five years ago but no 

information is available on 

standard deviation. For the 

oral test, the IPT 1 (level 

recommended for OLLI based 

on preliminary assessment) 

has one score level gain 

(about 14 points in raw score). 

For the reading and writing 

tests, there is a mean of 11-

point gain and three-point gain, 

respectively.

TBD

1 ACTFL does offer a test to assess writing proficiency, but it is intended for the general audience.

2

It’s possible to derive speaking and listening scores separately, but the norms have not been updated

D-1

Appendix D. 

Plan for Pilot Test of The Forgotten World in Gansu 

Province, April–June 2009

Appendix D – Plan for Pilot Test of The Forgotten World in Gansu Province, April – June, 2009

A Study of a Gaming Approach to English Language Learning for Middle School Students D-3

Contents

I. Background and Goals for the Pilot .................................................................................... D-5

1. Pilot test assessment instruments................................................................................. D-7

Pretest results............................................................................................................... D-7

Suggestions for spring pilot ......................................................................................... D-9

2. Measure implementation............................................................................................ D-10

3. Evaluate teacher training............................................................................................ D-11

4. Field test data collection procedures.......................................................................... D-11

5. Gather feedback on experiences using The Forgotten World.................................... D-11

6. Pilot questionnaires.................................................................................................... D-12

Teacher questionnaires............................................................................................... D-12

Student questionnaires............................................................................................... D-12

II. Plan for Full-Scale Evaluation.......................................................................................... D-12

III. Summary and next steps.................................................................................................. D-14

Appendix D – Plan for Pilot Test of The Forgotten World in Gansu Province, April – June, 2009

A Study of a Gaming Approach to English Language Learning for Middle School Students D-5

I. Background and Goals for the Pilot

In the fall of 2008, the Chinese Ministry of Education informed the U.S. that schools in Gansu 

Province would be available to participate in the evaluation of The Forgotten World (TFW), a 

computer-based game designed to help middle-school students learn English. A brief trip was 

made to Gansu Province in December, 2008, to visit schools and assess whether these schools 

would provide a good base for the evaluation. Among our questions were the following:

 Are schools and teachers willing to implement the program as a supplement to regular 

instruction, providing access to TFW in the computer lab at least one hour per week?

 Do the students in seventh or eighth grade exhibit the expected level of English that is

targeted by TFW?

 Do the schools have the requisite computing configurations to implement the program?

We found that principals and teachers were eager to implement the program, and teachers in the 

two pilot schools that were visited promised that students would be given access to TFW. They 

also asked for a teacher‘s guide and some ideas about how they might use TFW in class. 

Students in all grades (7 – 9) were eager to participate in the pilot sessions and, based on the 

preliminary tests and examination to test booklets, had sufficient background in English to 

benefit from TFW. Unfortunately, the computer labs did not have the suggested hardware and 

software. Staff from the Provincial Center for Educational Technology (PCET) were able to 

develop a ―work-around‖ for our visit that supported most program features; however, upgraded 

equipment will be needed for schools to fully use the program. Needed upgrades to the 

configurations will be made for the pilot test (two schools) but are uncertain for the full-scale (10 

schools). 

We are proceeding with the pilot test and planning for the full-scale evaluation. The purpose of 

this document is to outline our goals for the pilot test. The aim of the pilot is to ensure that all 

instruments and procedures are tested in advance of the full-scale evaluation. We have five 

primary goals for the pilot:

1. Confirm that all instruments are culturally appropriate for Chinese middle-school 

students and the proposed test battery can measure change over an academic year.

2. Develop and collect data to measure program implementation.

3. Assess whether teacher training is sufficient.

4. Test all data collection protocols and procedures.

5. Gather feedback from teachers and students concerning their experiences using The 

Forgotten World.

We have developed specific evaluation questions to be answered under each of these goals. 

Exhibit D-1 presents the evaluation questions, the method of data collection, the organization or 

person responsible, the timing of the data collection (April or June), and the organization that 

will enter the data and conduct the analysis. This document includes a section discussing each 

goal and related evaluation questions and plans.
Appendix D – Plan for Pilot Test of The Forgotten World in Gansu Province, April – June, 2009

D-6 A Study of a Gaming Approach to English Language Learning for Middle School Students

Exhibit D-1. Evaluation Questions for Gansu Pilot

Evaluation Questions Method/Source

Responsible 

Person

April Data 

Collection

June Data 

Collection Analysis

Gather information on assessments for design of full-scale evaluation

Is proposed test battery 

able to measure change 

over one year? Proposed 

battery to include IPT 

reading, writing; Mac II 

listening (adapted); use 

pre- and post-test 

versions

Assessments 

administered at two

schools, all 7th- and 8th-

grade students RTI Zhu

Additional 

testing if 

needed RTI

Pilot speaking test using 

Skype Sample of students RTI Green Green RTI

What are students 

beginning and end-ofyear scores on provincial 

tests? School or province ECNU Zhu ECNU

Questionnaires

Student and teacher 

questionnaires to gather 

information on prior 

experience with English, 

current exposure and 

demographic information RTI/ECNU Zhu Zhu ECNU

Measure implementation of program

Can we track student 

use? System data LGN LGN RTI

How do teachers use 

TFW? Teacher logs RTI Green Green RTI

Pace of learning - how 

much is covered?

Teacher logs and system 

data RTI and LGN Green Green RTI

Do teachers use Teacher 

Reports?

System data and postpilot quex RTI and LGN Green RTI

What type of technical 

support was needed?

Structured interviews 

with PCET and school 

tech personnel RTI/ECNU Zhu ECNU

What problems did the 

school face in 

implementing TFW? What 

were the rewards?

Structured interviews 

with principals RTI/ECNU Zhu ECNU

Confirm hardware and 

software configurations 

used at schools

Checklist completed by 

school tech personnel RTI/LGN Zhu ECNU

Evaluation Questions Method/Source

Responsible 

Person

April Data 

Collection

June Data 

Collection Analysis

Evaluate teacher training

Materials sufficient for 

startup?

End-of-training 

questionnaire RTI Green RTI

How much teacher 

training is needed?

End-of-training 

questionnaire RTI Green RTI

Feedback after pilot 

complete—what training 

would have helped?

Post pilot questionnaires 

and focus groups RTI Sha RTI

(continued)
Appendix D – Plan for Pilot Test of The Forgotten World in Gansu Province, April – June, 2009

A Study of a Gaming Approach to English Language Learning for Middle School Students D-7

Exhibit D-1. Evaluation Questions for Gansu Pilot (continued)

Evaluation Questions Method/Source

Responsible 

Person

April Data 

Collection

June Data 

Collection Analysis

Test procedures for data collection

How are test 

administrators identified? 

Can teachers be used?

Protocols to be 

developed by ECNU with 

RTI approval ECNU Zhu RTI

What is the best method 

for training test 

administrators? How long 

is needed?

Protocols to be 

developed by ECNU with 

RTI approval ECNU Zhu RTI

Do test administrators 

follow protocols?

Observations of test 

sessions RTI/ECNU Zhu ECNU

What is the best way to 

handle informed consent?

Protocols to be 

developed by ECNU with 

RTI approval RTI/ECNU Zhu Zhu RTI

Gather feedback on using The Forgotten World

How do students use the 

Forgotten World? What 

do they like and dislike?

How does it help them 

learn English?

Questionnaires and cog 

interviews RTI Sha RTI

What do teachers like and 

dislike about The 

Forgotten World?

Post-pilot questionnaires 

and cog interviews RTI Sha RTI

1. Pilot test assessment instruments

Pretest results

In December, we conducted a small pretest of two instruments that were identified as promising, 

the Mac II (grades 2 – 3) and the IPT Level 2.8The Mac II was administered to 20 seventh-grade 

students. We administered the IPT to 20 eighth-grade students. A summary of results are 

presented in exhibit D-2.

Exhibit D-2. Test scores for IPT and MAC from Gansu December Pretest

IPT MAC

Total items Range Median Total items Range Median

Reading 53 30 – 45 38.5 35 4 - 18 11

Writing 6

1

2 – 6 5 35 4 - 30 12.5

Listening 24 4 - 16 9

Speaking

1. Only one part of IPT Writing test was administered because of time constraints.

The MAC appears to capture the range of English competency for seventh-grade students and the 

IPT captures the range of competency for the eighth-grade students tested at the two schools. The 

 

8 Other tests were reviewed but removed from consideration. The ACTFL proficiency tests and the TOEIC 

are geared for adult populations; the TOEIC Bridge was age-inappropriate; and the Cambridge Young Learners 

Exam does not provide scale scores (only proficiency levels). The distributor of the Stanford English Language 

Proficiency test was unwilling to work with the project 
Appendix D – Plan for Pilot Test of The Forgotten World in Gansu Province, April – June, 2009

D-8 A Study of a Gaming Approach to English Language Learning for Middle School Students

only subtest that appears to have a ceiling or floor effect is the IPT writing test; however, we 

only administered one subtest. Since we did not administer the most difficult section, we do not 

believe that this would be an issue in the full administration. 

We also matched the vocabulary used in TFW, the English textbooks used in the schools, and 

assessment items. For the assessments, we identified items that three or fewer students answered 

correctly. Only one item in the IPT fell into this category, and the key word in the item was 

―pain,‖ a word that is not covered until the ninth-grade textbook. The MAC contained 11 items

that fell into this category: Seven of those key words could not be found in the textbooks, one

was in the seventh-grade text, two in the eighth-grade text, and one in the ninth-grade text. The 

MAC would need substantial modification to be used in the evaluation.

We also compared the vocabulary in the Chinese texts for seventh and eighth grades with TFW 

vocabulary to try to determine which grade would be better ―match‖ for TFW. The TFW was 

designed for students with some prior exposure to English and assumes that students have a 500-

word vocabulary. We matched the list of 500 words to TFW and present the results in 

Exhibit D-3. 

Exhibit D-3. Prerequisite vocabulary words presented in textbooks by grade level 

Grade Level Count

7th grade 269

8th grade 71

9th grade 23

Of the 500 words, 363 could be found in the middle school textbooks, with over half (269) 

appearing in the seventh-grade text book. Thus, it seems unwise to assume that seventh-grade 

students will have the working vocabulary expected by the game designers.

Exhibit D-4 presents information on matches between the TFW vocabulary and the middle 

school textbooks. In addition to the 500 prerequisite words, TFW presents over 1,000 new words 

across the 12 episodes; about half of the words (508) are also found in the Chinese middle school 

textbooks, and most of these (267) are found in the seventh-grade text. 

The seventh- and eighth-grade textbooks contain about 492 words each or 985 words total. This 

means that about 40 percent of the vocabulary ((267+145)/1000) normally taught in these two 

grades will be reinforced by TFW. 
Appendix D – Plan for Pilot Test of The Forgotten World in Gansu Province, April – June, 2009

A Study of a Gaming Approach to English Language Learning for Middle School Students D-9

Exhibit D-4. Matched Vocabulary Count by TFW Episode and Textbook Grade 

Levels

TFW Episode 7th 8th 9th Total

1 65 25 11 101

 2 28 5 10 43

 3 9 15 18 42

 4 20 30 14 64

 5 11 3 2 16

 6 19 14 10 43

 7 16 13 5 34

 8 18 14 13 45

 9 31 6 6 43

10 10 4 0 14

11 27 8 1 36

12 13 8 6 27

Total 267 145 96 508

Suggestions for spring pilot

Based on our experience in Gansu with the pretest, we suggest the following:

1. We will ask eighth-grade students to participate in the evaluation. This group will have 

much of the prerequisite vocabulary and will have learned about one-quarter (267) of 

TFW new words in seventh grade. There will still be about three-quarters of the TFW 

vocabulary that they will have to learn through the episodes and games. Given that 

vocabulary is antecedent to more complex language competencies (sentence structure, 

grammar, etc.) we think that eighth-grade students are in the best position to benefit from 

TFW. 

2. For reading, we will use the IPT. In the pretest the reading test seemed well-matched to 

student abilities—there were no students who were at the top or bottom of the test. This 

test also requires very little modification, if any. We will need to examine the alternate 

forms of this test (we need one form for the pre-test and one for the post-test) to make 

sure that there are no culturally inappropriate items.

3. For writing, we will use the IPT, including a second writing subtest that requires students 

to write a story. We will not use the subtest that measures ―writing conventions‖ such as 

punctuation, since TFW does not aim to improve student competency in this area. The 

range of total possible scores on the writing subtest will be 0 to 9 points, although we 

could consider expanding the grading rubric.

4. For listening, we propose to revise the MAC listening test. For the MAC test, a tape is 

supplied and a native U.S. English-speaking person provides instructions and reads all the 

questions to students. In administering the pretest, we found the instructions to be 

somewhat cumbersome and the layout of the answer sheets was confusing to students. 

We will simplify the directions and the answer forms. Two forms will be needed; one for 

administration at the beginning of the year and one for administration at the end of the 

year.
Appendix D – Plan for Pilot Test of The Forgotten World in Gansu Province, April – June, 2009

D-10 A Study of a Gaming Approach to English Language Learning for Middle School Students

5. RTI will explore devising a new test, assessing reading and listening skills based on items 

that appear in TFW that do not appear in the school textbooks. This test will be 

administered at the end of the school year. The structure of the listening test and items 

will mirror the MAC listening test and will include: eight items that ask students to select 

a picture that matches the word, eight items that are based on two short stories that are 

read to students; and eight items that require students to select an appropriate response to 

a conversational question. An audiotape will be professionally recorded that will be used 

in all classrooms. There will be 26 reading items based on TFW. The total test will have 

50 items.

6. The reading, writing, and listening tests should provide us with a scale of at least 86 

points (53 reading, 10 writing, and 24 listening), although we may want to consider 

weighting the subtests. 

7. For the pretest, we propose administering a speaking test to a subsample of students—IF 

the voice recognition applications can be made to work on the school machines. The 

speaking test will be patterned on the MAC and will have three parts: students view 

pictures and say the word in English; students are asked personal or conversational 

questions about themselves; and students view a picture and are asked to tell a story 

about the picture. The sessions will be tape-recorded. We had hoped to administer these

tests via computer but do not believe that the Internet connections will be sufficient.

8. We will ask to administer the assessments to ALL seventh and eighth grade students in 

the two schools participating in the pilot. This will provide us with about 1,000 students 

per grade or about 500 per test form. It will also allow us to estimate the growth in 

language during an academic year by comparing end-of-year seventh-grade students 

(equivalent to beginning eighth-grade students) with end-of year eighth-grade students. 

Note that this will only be an approximation since we won‘t have beginning and end-of 

year scores for the same students. (We were unable to launch the pilot in the fall because 

of delays in software development).

9. The proposed test battery will require approximately two hours of testing time. Reading 

typically requires 60 minutes; writing 30 minutes and listening 30 minutes.

10. The IPT reading/writing test costs approximately $4 per booklet ($8,000 for the pilot).

11. Schools will be asked to provide beginning and end-of-year test scores on the district 

assessments for eighth-grade students participating in the pilot. Officials in Gansu agreed 

to this in December. We will need to collect identifying information from students (name 

and school, perhaps birth date or school ID) to make the match. 

2. Measure implementation

One of the goals of the pilot is to make sure that conditions are such that schools and teachers 

can implement the program. Thus we want to confirm that TFW is fully functional, the 

equipment is able to support student use, any technical support that is required is given to 

schools, and teachers refer to TFW in the classroom and give students the opportunity to use the 

program. At the December meeting in Gansu, the principals stated that they would make sure 

that students have access to the computer lab for 40 minutes at least once a week, and the 

principal of school committed to a higher level, promising to incorporate TFW in English classes 

once each week. We will want to confirm these agreements and monitor the implementation. We 

also want to learn how quickly or slowly students will move through TFW so that we can time 

the full-scale data collection to closely follow the completion of TFW.
Appendix D – Plan for Pilot Test of The Forgotten World in Gansu Province, April – June, 2009

A Study of a Gaming Approach to English Language Learning for Middle School Students D-11

Exhibit D-1 provides information on the questions we want to answer in the pilot. We will need 

to collect data from several sources:

 We have asked Learning Games Network (LGN) to collect information on students‘ use 

of TFW (total time and highest-level episode), teacher use of TFW (total time or number 

of logins and a count of times teacher reports are accessed), and ―down periods‖ when 

the system is not be used at all by the school (to uncover system issues or other events 

preventing the use of the program). PCET will be asked to contact schools that have 

down periods of more than one week to see if technical assistance is needed.

 Teachers will be asked to keep brief logs of their use of TFW, both in class and when 

students are given access to the lab. We will develop a Chinese-language template for 

the log.

 Structured interviews will be conducted at the end of the pilot with PCET and school 

tech personnel to document the types of system support that were needed.

 Structured interviews will be conducted with principals at the end of the pilot to discuss 

any issues in implementation and also the perceived benefits of using TFW.

 A checklist documenting the elements of the computing infrastructure will be collected 

from each school.

3. Evaluate teacher training

In the pilot, we want to find out from teachers if the training they received was sufficient for 

their subsequent use of TFW, and to gather their suggestions for ways it might be improved. We 

will do this at two points in time. An initial questionnaire will be developed to be administered at 

the end of teacher training to assess whether teachers feel they have the knowledge they need to 

begin using TFW. At the end of the pilot, teachers will be asked to complete another 

questionnaire assessing, in retrospect, if the training was adequate. These data will be supplied to 

Coastline to help in creating the final teaching materials for the full-scale evaluation.

4. Field test data collection procedures

Professor Zhu, our data collection partner in China at East China Normal University (ECNU) has 

arranged to bring graduate students from Shanghai to administer the tests and will also hire 

additional administrators in Gansu province (probably from Northwest Normal University). We 

will arrange a training session with Professor Zhu on the administration of the tests and will help 

to prepare training manuals. RTI will observe the test administrator training and some test 

administrations. RTI will also prepare a short information form that test administrators will be 

asked to complete that describes salient features of the test session as well as any issues that 

arose during the test session.

5. Gather feedback on experiences using The Forgotten World

At the completion of the pilot in June, we will collect some information concerning students‘ 

experience using TFW on the student questionnaire. We will also collect information from 

teachers on a teacher questionnaire, as well as convene a focus group of teachers to discuss the 

experience.
Appendix D – Plan for Pilot Test of The Forgotten World in Gansu Province, April – June, 2009

D-12 A Study of a Gaming Approach to English Language Learning for Middle School Students

We may wish to conduct some number of cognitive interviews with students as they use the 

software. This needs to be discussed with LGN and Coastline, to determine whether this 

information will be useful to future development.

6. Pilot questionnaires

Teacher questionnaires

Teachers will be asked to complete a training evaluation, a teaching log, and a questionnaire, all 

of which will be prepared in Chinese. The questionnaire to be in administered in June will 

capture information on:

 The quality of training

 Experiences using TFW

 Experience teaching English

 Education and credentials for teaching English

 Informal exposure to English, including travel

 Likelihood that they will continue using TFW

 Limited demographic items

Student questionnaires

Questionnaires will be administered to students in April and June. The questionnaires will be 

designed to capture:

 Experience using TFW

 Years of formal English instruction

 Informal exposure to English (TV, videos, friends or relatives, etc)

 Importance of learning English

 Expectations for future

 Limited demographic items

RTI will develop a list of items and ECNU will develop the Chinese-language questionnaires. 

The questionnaires will be reviewed by a native Chinese-speaking survey methodologist at RTI. 

II. Plan for Full-Scale Evaluation

There are three major questions to be answered by the full-scale evaluation:

1. Do students exposed to TFW demonstrate improved English-language skills in reading, 

writing, and listening, compared to students who have not been exposed to TFW?

Primary measure: Score on English-language tests at beginning and end of year that are 

developed by the province.

Secondary measure: IPT reading and writing tests; MAC listening tests administered to treatment 

and comparison schools at beginning and end of the year.
Appendix D – Plan for Pilot Test of The Forgotten World in Gansu Province, April – June, 2009

A Study of a Gaming Approach to English Language Learning for Middle School Students D-13

2. Do students exposed to OLLI master the vocabulary and language presented by the 

program that is not covered by classroom texts?

Primary measure: Reading and listening test developed for end-of-year based on words unique to 

TFW that are not included in eighth-grade textbook. This test would need to be developed by 

RTI and administered to treatment schools only.

3. What implementation factors make a difference in the degree to which students benefit 

from TFW?

Primary measure: Primary outcome measure will be TFW-based test; may also used provincial 

tests or IPT or MAC II. Implementation factors to be measured include: time spent on TFW, 

teacher‘s use of TFW in the classroom, level of English at introduction of TFW, etc.

RTI is currently planning the full-scale evaluation. On approval from ED, we will ask PCET to 

identify 20 volunteer schools. These schools must be willing to participate in the evaluation, use 

the Go For It! English textbook series, be located in reasonably accessible area, and have the 

requisite hardware. We will then randomly assign 10 as comparison schools and 10 as treatment 

schools. We will request that we randomly select four classrooms of eighth-grade English, taught 

by two different teachers. We were told that there are approximately 60 students per class, so this 

will provide over 200 students per school, or 2,000+ participants in the treatment and comparison 

groups. 

With 20 schools, assuming an intraclass correlation coefficient of .22, we will be able to detect 

an effect size of approximately .65.9This means that the treatment group will need to have a 

mean score that is two-thirds of a standard deviation higher than the comparison group to 

conclude that TFW is effective. Since we have no previous data on the effectiveness of TFW, it 

is difficult to know whether this assumption is warranted—it is an extremely high bar.

Twenty teachers will need to be trained in the use of TFW. If teachers are asked to administer the 

tests, we will need to provide training on test administration to 40 teachers. At this point, we do 

not envision providing an alternative intervention to comparison schools.

We may want to consider testing additional eighth-grade students who are not exposed to TFW 

at the treatment schools. If there is no contamination, these students would provide strong 

statistical controls. If there is contamination, it might provide documentation for the ―viral‖ 

effect of gaming applications and provide us with data that we can use to examine dose-related 

response.

Additional power calculations will be prepared once we have the scale for the assessments. 

However, final power calculations cannot be estimated until we have sample assessment data. If 

it is clear that we cannot meet the power requirements, we may wish to reconsider the first 

question posed above. We may also want to increase the prominence of the qualitative data 

 

9 This was estimated earlier assuming 70 students per school. We expect that increasing the sample size in 

each school will allow us to detect a slightly lower effect size.
Appendix D – Plan for Pilot Test of The Forgotten World in Gansu Province, April – June, 2009

D-14 A Study of a Gaming Approach to English Language Learning for Middle School Students

(teacher focus groups and student cognitive interviews) in the final design for the full-scale 

evaluation.

III. Summary and next steps

Once this plan is approved, a subcontract will be issued to ECNU to conduct the tasks assigned 

in Exhibit D-1. We will also ask outside consultants to review the plan, and proposed 

instruments, and provide feedback by March 22. Instruments will be submitted to ED for 

approval on March 30.
E-1

Appendix E. 

Evaluation Plan Update, July 2009 (Revised)

E-3

Appendix E:

Evaluation Plan Update, July 2009 (Revised)

1. Purpose of the evaluation

The purpose of this project is to evaluate the implementation and efficacy of English language 

learning software developed especially for middle school students in western China. Following a 

trip to China by the U.S. president in 2001, the U.S. and China signed a bilateral memorandum 

of understanding for a joint research and development project, known as the U.S.-China 

E-Language Project. The E-Language software—named The Forgotten World (TFW)—has been 

developed by Coastline Community College and Learning Games Network, with funding from 

the Hewlett Foundation.

The focus of the evaluation is to compare the proficiency of the students exposed to the TFW

program with the proficiency of students who do not have access to TFW. We will also identify 

implementation factors that facilitate or impede program effectiveness. A secondary purpose will 

be to obtain user reaction to instructional features of the program. The major research questions 

to be addressed are as follows:

 Do students exposed to TFW English demonstrate increases in proficiency when 

compared to students without access to TFW English, given the same curriculum and 

under conditions that allow reasonable access to the software? Proficiency in two main 

areas and two cross-cutting areas will be measured:

o Listening skills

o Reading skills

 Does TFW increase student motivation to learn English?

 Does the program work for students at all levels of beginning English?

 What implementation factors make a difference in promoting student proficiency?

An experimental study, conducted by RTI in partnership with East China Normal University 

(ECNU), is being conducted to answer this question. Although the circumstances do not allow 

for a fully powered hypothesis-testing design, the current design will allow us to examine the 

impact of TFW on a large group of students and will allow us to generate hypotheses about 

whether the program is effective. 

The design will compare eighth-grade students in five comparison and five treatment schools in 

Gansu province, China. The selected schools are large middle schools with approximately 2,800 

eighth-grade students enrolled in the treatment schools and another 2,800 enrolled in the 

comparison schools. While we initially requested 20 schools (10 treatment and 10 comparison), 

the provincial authorities determined that there were not enough schools that met the 

technological and participation requirements for the project. A decision was made that we would 

use the 10 schools that met the requirements and divide them into two groups, a treatment group 

that would implement TFW and a comparison group. 
Appendix E – Evaluation Plan Update, July 2009 (Revised)

E-4 A Study of a Gaming Approach to English Language Learning for Middle School Students

Students in the second year of middle school (eighth-grade students) will participate in the 

evaluation. Proficiency gains will be measured over the course of a year (two semesters), and the 

evaluation will be conducted in the 2009-2010 academic year. A schedule is provided in the 

attachment.

2. Instrumentation (Task 3)

Assessment instruments and questionnaires were tested in two pilot schools in the province in the 

spring of 2009. The IPT Reading and IPT Writing assessments were used, along with the MAC

II Listening assessment. The IPT Reading assessment was used without modification; the other 

two assessments were adapted based on preliminary work in China. For the writing test, some of 

the picture prompts were substituted using pictures from other IPT tests that were more culturally 

appropriate for Chinese middle school students. The listening test directions were modified so 

that the directions were delivered to students in Chinese, and the answer form was modified to 

make it more understandable to students. All of the listening instructions were professionally 

recorded at RTI to ensure comparability across sites.

The purpose of the pilot was to make sure that the tests were able to measure growth in English 

proficiency among this population and that the modifications made to the tests did not alter the 

equivalency of the pre- and post-test forms. The tests were administered at two pilot schools to 

seventh- and eighth-grade students. Since we did not have a full year for the pilot, end-of-year 

seventh-grade students represent scores that can be expected from beginning eighth-grade 

students.

Results are presented in the tables below. Exhibits E-1, E-2, and E-3 present information on each 

of the major tests that were used. As can be seen looking across rows, the two forms (pre-test and 

post-test) were roughly equivalent across the tests; the largest difference is on the writing test. 

Also, comparing results for seventh- and eighth-grade students, eighth-grade students 

consistently performed better than seventh-grade students. Again, the writing test showed the 

weakest result and may require some additional modification. A large number of students left the 

writing test blank; in focus groups, students reported that they were unfamiliar with the test 

format or had little or no experience writing English or both.

Exhibit E-1. IPT 2004 Reading Test Average Raw Score (Total Items = 53)

Grade

Form 1C

(n = 940)

Form 1D

(n = 907)

Difference between 

Form 1C and 1D

7th grade

(n = 1,017)

20.07

SD = 6.08

(n = 513)

20.58

SD = 6.68

(n = 504)

-.51

t=-1.27

not significant (p>.05)

8th grade

(n = 830)

29.71

SD = 9.91

(n = 427)

30.41

SD = 9.14

(n = 403)

.70

t=1.06

not significant (p>.05)

Difference between 7th and 

8th grade

-9.64

t=-18.28

p<.05

-9.83

t=-18.07

p<.05

Average Total

(N = 1,847)

24.45

SD = 9.37

24.95

SD = 9.25
Appendix E – Evaluation Plan Update, July 2009 (Revised)

A Study of a Gaming Approach to English Language Learning for Middle School Students E-5

Exhibit E-2. MAC II Listening Test Average Raw Score (Total Items = 24)

Grade Form A2

(n = 940)

Form B2

(n = 899)

Difference between Form

A2 and Form B2

7th grade

(n = 1018)

12.80

SD = 2.96

(n = 513)

11.76

SD = 3.08

(n = 505)

1.04

t=5.49

p<.05

8th grade

(n = 821)

14.43

SD = 3.62

(n = 427)

13.60

SD = 3.36

(n = 394)

.83

t=3.4

p<.05

Difference 

between 7th 

and 8th grade

-1.63

t =-7.59

p<.05

-1.84

t=-8.54

p<.05

Average Total

(N = 1,839)

13.54

SD = 3.37

12.57

SD = 3.33

Exhibit E-3. IPT 2004 Writing Test Average Raw Score 

(Total Items = 3; Highest Possible Score = 9)

Grade

Form 1C

(n = 922)

Form 1D

(n = 894)

7th grade

(n = 996; 354 received a 

score of zero)

1.80

SD = 1.94

(n = 498; 189 received a score of 

zero)

2.20

SD = 2.37

(n = 498; 165 received a score of 

zero)

8th grade

(n = 820; 202 received a 

score of zero)

2.44

SD = 2.68

(n = 424; 154 received a score of 

zero)

3.84

SD = 2.40

(n = 396; 48 received a score of 

zero)

Average Total

(N = 1,816)

2.09

SD = 2.33

2.93

SD = 2.52

Exhibit E-4 presents information on the combined score from the reading and listening tests. 

Again, the tests are able to quantify a difference between seventh- and eighth-grade students.

Exhibit E-4. Reading and Listening Tests Combined Average Raw Score

(Total Items = 77)

Grade/Form

Regular Form

(n = 932)

Alternate Form

(n = 873)

7th grade

(n = 1,002)

32.88

SD = 7.81

(n = 509)

32.40

SD = 8.62

(n = 493)

8th grade

(n = 803)

44.09

SD = 12.43

(n = 423)

43.99

SD = 11.32

(n = 380)

Average Total

(N = 1,805)

37.97

SD = 11.60

37.44

SD = 11.43

Exhibit E-5 presents the combined raw score on the reading and writing test by the students‘ 

scores on an eighth-grade English test administered by the schools. As expected, students who 

score well on the eighth-grade school test tend to score well on the tests administered by the 
Appendix E – Evaluation Plan Update, July 2009 (Revised)

E-6 A Study of a Gaming Approach to English Language Learning for Middle School Students

project, and students who perform poorly on the school tests tend to perform poorly on the tests 

we administered.

Exhibit E-5. Reading and Listening Combined Raw Score by School Score of 

Eighth-Graders

Pilot Reading and Listening Combined Score

Percent Correct

Score on 8th-Grade 

School Test

76-100% 51-75% 26-50% 0-25% Total

Percent Correct

76-100% 32.2% 55.3% 10.5% 2.0% 152

51-75% 11.6% 69.5% 18.1% 0.8% 249

26-50% 3.8% 47.3% 45.8% 3.1% 260

0-25% 4.2% 26.0% 65.6% 4.2% 96

Total 92 405 243 17 757

Finally, the correlation between the subtests appears to be reasonable, as presented in Exhibit 

E-6. The school score is not highly correlated with the listening test. This is consistent with the 

rationale for the study; teachers and students in rural areas have little opportunity to interact with 

native English speakers and consequently their listening (and speaking skills) are frequently 

weaker than their reading skills.

Exhibit E-6. Correlations Between Tests

Reading Listening Writing School Score

Reading 1

Listening 0.56216653 1

Writing 0.64582232 0.37361175 1

School Score 0.47543256 0.30729068 0.498881887 1

Finally, RTI may develop one additional short test to be administered at the end of the year. This 

test will be focused on the vocabulary that is unique to TFW and the vocabulary that is shared by 

the school textbook and TFW.
Appendix E – Evaluation Plan Update, July 2009 (Revised)

E-7

Attachment: Schedule of Major Activities

Schedule of Major Activities for Full-Scale Evaluation

Activity Responsible Date

Identification of sample schools PCET June 30, 2009

Random assignment RTI July 10, 2009

Invitations to teacher training PCET July 15, 2009

LGN/Coastline visit selected schools LGN July 30 – August 3, 2009

Teacher training LGN/NWN August 3 – 6, 2009

Final preparation and shipping of 

materials for pre-test

RTI August 15, 2009

School visits to test equipment and meet 

with teachers

LGN/NWN September 14 – 18, 2009

Test administrator training RTI/ECNU September 28, 2009

Pre-test administration in schools ECNU September 29 – October 9, 2009

Quality control observations RTI/ECNU November 23 – 27, 2009

Final preparation and shipping of 

materials for post-test

RTI April 30, 2010

Post-test administration ECNU June, 2010

Data entry ECNU June 30, 2010

Analysis and reporting RTI July – September 2010

F-1

Appendix F.

Data Collection Procedures and Materials: 2009 PreTest Test Administrator Manual, Attached Informed 

Consent Forms, and Questionnaires 

F-3

RTI Project 0209234.002.008

September, 2009

Test Administrator’s Manual

Evaluation of the E-Learning System: OpenLanguage Learning Initiatives for English

Main Study

Lanzhou, Gansu

China

Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students F-5

1. Background and Purpose of the Evaluation

The U.S. Department of Education [Policy and Program Studies Service (PPSS), within 

the Office of Planning, Evaluation, and Policy Development (OPEPD)], is sponsoring an 

evaluation of the E-language learning system named Open Language Learning Initiative for 

English. (OLLI English). The computer-based game developed to help students learn English is 

called The Forgotten Word (TFW).

RTI International (RTI) is carrying out the evaluation which is led by Dr. Patricia 

Green. RTI is collaborating with East China Normal University (ECNU) for the work in 

China. The project director at ECNU is Professor Zhu Yiming, a professor of the ECNU 

Department of Education.

This evaluation has been scheduled to take place in Chinese middle schools for two 

semesters from September 2009 to June 2010. The Chinese Ministry of Education identified 

Gansu Province as a site that fit the requirements for the evaluation. The provincial authorities 

selected the sampled schools. The evaluation aims to answer the following key evaluation 

question:

Do students exposed to TFW demonstrate increases in proficiency in reading, writing, 

listening, and speaking skills when compared to students without access to TFW, given the same 

curriculum and under conditions that allow reasonable access to the software?

The role of the test administrators is to:

1. Confirm (and update if needed) the current student roster with the student tracking form.

2. Distribute consent forms to all students and read aloud the form.

3. Administer reading and listening tests. 

4. Administer a questionnaire to all students participating in the survey about their thoughts 

and experience about learning English (attached to the listening test).

5. Administer a questionnaire to all teachers participating in the teachers‘ training for the 

survey. The questionnaire asks about the teachers‘ thoughts and experience about teaching 

English.

2. Verifying the Student Roster with the Student Tracking 

Form

2.1 Prior to Test Administration - Student Roster vs. Student 

Tracking Form

The first important step is to obtain a current student roster for each selected classroom at 

the school PRIOR to the test day (or at least 2 hours before testing is scheduled to begin). You 

will need to review each student name listed on the current student roster against the tracking 

form. A sample student tracking form is included in this manual as Attachment 1.

■ If a student is missing from the student tracking form, you must add this student to the bottom 

of the tracking form on a line that has been assigned an ID but has no name. Please put the 

student‘s name on the list. It is very important that we know who has been added to which 
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

F-6 A Study of a Gaming Approach to English Language Learning for Middle School Students

classroom and school. Please check with these students and make sure they include their 

TFW User ID on the questionnaire, or make a note that they do not have one assigned.

■ If you notice a spelling change, please update the student tracking form with the correct 

spelling.

■ If you have more than 3 additional students on the student list in the class NOT listed on the 

tracking form, you should assign new student IDs, using the same school and classroom IDs, 

but adding student IDs beginning with 501.

2.2 On Test Day

All test materials have been bundled by classroom. Test materials for each student will be 

inside individual envelopes. The outside of each envelope will list the student‘s name and ID –

YOU MUST DISTRIBUTE THE ENVELOPES TO STUDENTS INDIVIDUALLY, 

MATCHING THE NAME ON THE OUTSIDE OF THE ENVELOPE TO THE CORRECT 

STUDENT. If you have added students to the student tracking form, you must give the newly 

added student the corresponding student packet ID.

The students should confirm that they have received the correct envelope by verifying the 

name on the outside of the envelope.

During the test, or after the test, please use the student tracking form to check off which 

students take the test. Please make a check after the student ID on the tracking form.

3. Consent Forms

3.1 Parent Letter

We will give the schools a letter addressed to parents of the 8th grade students who are 

participating in the study, and ask the schools to distribute the letters to the parents. The letter 

describes the purpose of the study, students‘ participation, and confidentiality. Parents who do 

not wish to have their child participate in the study can tear off the bottom portion of the letter 

that says, ―I do not wish to have my child participate in this research‖ and return it to the test 

administrator. This student shall then be excluded from the study. Samples of the letter are 

included in English as Attachments 4 and 5 (and 4a and 5a in Chinese).

3.2 Student Assent Form 

Before the start of the English language assessment tests and student questionnaire, the 

test administrator should give each student a copy of the student assent form and read it aloud to 

the students. The form will explain the purpose of the study, the voluntary nature of their 

participation, and that they are asked to complete assessment tests for reading and listening, as 

well as a questionnaire about their thoughts and experience with learning English. The form also 

assures the students that their individual test scores or responses to the questionnaire will not be 

shared with their teachers, their parents, China‘s Ministry of Education, or the U.S. Department 

of Education. To ensure confidentiality, the students‘ name will not be recorded on the tests but 

will be represented by an ID number. If a student does not wish to participate, he or she shall be 

excluded from the study. Copies of the forms are provided in English as Attachments 2 and 3 

(and 2a and 3a in Chinese).
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students F-7

3.3 Teacher Consent Form for Participation

Before administering the teacher‘s questionnaire, the administrator should give a consent 

form to each teacher. The form will explain the purpose of the study, the voluntary nature of 

their participation, and that they are asked to complete a questionnaire about their thoughts and 

experience surrounding teaching English. The form also assures the teachers that individual 

responses to the questionnaire will not be shared with the school, other teachers, China‘s 

Ministry of Education, or the U.S. Department of Education. To ensure confidentiality, their 

name will not be recorded on the questionnaire but will be represented by an ID number. If a 

teacher does not wish to participate, he or she shall be excluded from the study. Copies of the 

forms are provided in English as Attachments 6 and 7 (and 6a and 7a-Chinese). A copy of the 

Teacher Questionnaire is provided as Attachment 8 and 8a.

4. Assessment Tests and Questionnaires

As mentioned previously, you will be administering the reading test, the listening test and 

the student questionnaire to all students sampled for this study. It is very important that you 

follow the scripts for each test – each student must hear the same information and you must 

read/play the entire script for each test.

4.1 Reading Test

The reading test is also called the  ̳IPT‘ test and all items are in one booklet. Test 

administrators distribute the envelopes (that contain both tests and questionnaires) before 

beginning to read the directions for the reading test. After the envelopes are distributed, instruct 

students to take out the reading test booklet.

When all students have the reading test on their desk, instruct them to open the booklet 

and begin reading the script with the instructions for the first section. 

The reading test has five sections and you will need to provide instructions (from the 

script) before each section. 

A copy of the script (in Chinese and English) is provided in Attachment 9. This script 

must be practiced in advance of administering the test.

A copy of the reading test is provided as Appendix A.

4.2 Listening Test

Directions for the listening test must be played for the students using a cassette tape 

recorder. Before beginning the tape, ask students to take the listening test from the envelope and 

instruct them to open the booklet to the first page. Test administrators should explain to students 

how the bubbles should be marked and then play the standardized recording for the test. Make 

sure that all students understand what to do before the recording is played. As part of the test 

instructions, the recording will cover the function of the bubbles again. The test instructions can 

be replayed, but the actual English prompts should only be played once.
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

F-8 A Study of a Gaming Approach to English Language Learning for Middle School Students

A copy of the booklet containing the listening test and questionnaire is in Appendix B.

4.3 Student Questionnaire

The student questionnaire is the same booklet as the listening test. This part of the 

assessment asks the student how they feel about learning English. Some students may be getting 

tired of the test at this point, so you should encourage students to complete it and do their best. A 

copy of the student questionnaire is provided in Attachment 9.

4.4 Teacher Questionnaire

The teacher questionnaires will be distributed to eighth-grade English teachers on arrival 

to school and collected before departing from the school. A copy of the teacher questionnaire is 

provided in Attachment 8.

5. Collecting the Test Booklets

When students have completed the test and questionnaires, collect all booklets from 

students. The tests SHOULD NOT be put back into the envelopes. Please leave the envelopes in 

the classroom or in a trash container at the school.

Thank the students and teacher for their participation before leaving the classroom.

Before leaving the school, please check off each booklet, including the teacher 

questionnaire, on the Student Tracking Form.

Please bundle books from each class together and place in mailing box (do not use boxes 

that were sent from U.S. as they may be damaged).

DO NOT include the student tracking form in the box with the tests. If for any reason a 

box would be lost or stolen, it is important that a person looking at the tests would be unable to 

link a test with a student name. Please give the tracking forms to Dr. Zhu or Liu Lu.

Thank you for your time and effort in this important China and U.S. evaluation!

6. CHECKLIST

Before going to school, check that you have:

 Copy of the Student Tracking Form for each class that you will be testing.

 Envelopes with booklets for all students on Tracking Form(s).

 Copies of Student Assent Forms.

 Script with instructions for reading test.

 Tape and tape recorder for listening test.
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students F-9

In the classroom:

1. Introduce yourself to the class and explain purpose of the study

2. Give the consent forms to all students.

3. Distribute the envelopes with test booklets by calling student names.

4. Read aloud the consent form.

5. Instruct students to take the reading booklet from the envelope and read instructions for 

the first section of the test. When students have finished first section, read instructions for 

next section. There are 5 sections in the test.

6. Instruct students to take the listening test booklet from the envelope. Give instructions for 

filling out ―bubbles‖ and begin playing tape recorded instructions.

7. When students have finished listening test, tell them to turn the page and complete the 

questionnaire.

8. When students have finished, collect all booklets. Leave envelopes at school.

After the test session:

 Complete the Student Tracking Form by checking off the IDs of all completed 

booklets.

 If any unusual circumstances arose during the test, attach an explanatory note to 

the Student Tracking Form.
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached Informed Consent Forms, and Questionnaires

F-10 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 1. Student Tracking Form

School in English School in Chinese School ID Class# Serial Number Student Name School test ID Class test ID Student test ID

10 1 1 910 201 301

1 2 910 201 302

1 3 910 201 303

1 4 910 201 304

1 5 910 201 305

1 6 910 201 306

1 7 910 201 307

1 8 910 201 308

1 9 910 201 309

1 10 910 201 310

1 11 910 201 311

1 12 910 201 312

1 13 910 201 313

1 14 910 201 314

1 15 910 201 315

1 16 910 201 316

1 17 910 201 317

1 18 910 201 318

1 19 910 201 319

1 20 910 201 320

1 21 910 201 321

1 22 910 201 322

1 23 910 201 323

1 24 910 201 324

1 25 910 201 325

1 26 910 201 326

1 27 910 201 327

1 28 910 201 328
F-11

Attachment 2. Assent Form – Student - TFW

Dear Student:

Thank you for volunteering to participate in this study. This study is an Evaluation of the Open-Language Learning 

Initiative for English (OLLI English). It is sponsored by the U.S. Department of Education, as part of a collaboration 

project with China‘s Ministry of Education. RTI International, a US-based non-profit research organization, is 

conducting this study. We have also asked East China Normal University (ECNU) to help us.

The purpose of the study is to examine the effectiveness of English language learning software on students‘ English 

proficiency. We are interested in how students listen and read in English. To do this, we will give the students some 

tests that will take about 1.5 hours to complete. In addition, we will ask you to fill out a questionnaire about your 

experience and thoughts with learning English. You are one of about 6,000 middle school students participating in 

this study.

Your participation is voluntary. You may change your mind and not participate at any point. If you decide not to 

participate, there is no penalty. You may choose not to answer any question. There are no specific risks to you from 

taking part in the study. 

There are no other direct benefits to you from participating, but you will be helping us understand the effectiveness 

of English language learning software on students‘ English proficiency. Your name will not be recorded on the tests 

or the questionnaire and you are only represented by an ID. We are required to keep all information confidential and 

what you say or do on these tests will not affect your grades at school. Only authorized researchers at RTI and 

ECNU will know your test scores and your responses to the questionnaire. No one else will know, not your school, 

your teachers, the U.S. Department of Education, China‘s Ministry of Education, or your parents.

If you have any questions about this study or your participation, please feel free to ask the test administrator.

We thank you in advance for your help!

Patricia J. Green

Project Director

RTI International
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

F-12 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 3. Assent Form – Comparison Students – no TFW

Dear Student:

Thank you for volunteering to participate in this study. This is a study of the Evaluation of the Open-Language 

Learning Initiative for English (OLLI English). It is sponsored by the U.S. Department of Education, as part of a 

collaboration project with China‘s Ministry of Education. RTI International, a US-based non-profit research 

organization, is conducting this study. We have also asked East China Normal University (ECNU) to help us.

The purpose of the study is to examine the effectiveness of English language learning. We are interested in how 

students listen and read in English. To do this, we will give the students some tests that will take about 1.5 hours to 

complete. In addition, we will ask you to fill out a questionnaire about your experience and thoughts with learning 

English. You are one of about 6,000 middle school students participating in this study.

Your participation is voluntary. You may change your mind and not participate at any point. If you decide not to 

participate, there is no penalty. You may choose not to answer any question. There are no specific risks to you from 

taking part in the study. There are no other direct benefits to you from participating, but you will be helping us 

understand the effectiveness of English language learning. Your name will not be recorded on the tests or the 

questionnaire and you are only represented by an ID. We are required to keep all information confidential and what 

you say or do on these tests will not affect your grades at school. Only authorized researchers at RTI and ECNU will 

know your test scores and your responses to the questionnaire. No one else will know, not your school, your 

teachers, the U.S. Department of Education, China‘s Ministry of Education, or your parents.

If you have any questions about this study or your participation, please feel free to ask the test administrator.

We thank you in advance for your help!

Patricia J. Green

Project Director

RTI International
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaire

A Study of a Gaming Approach to English Language Learning for Middle School Students F-13

Attachment 4. Parent Letter – Treatment Schools -TFW Parents

Dear Parent or Guardian:

We are pleased to inform you that your child‘s school has been selected to participate in an important study called 

the Evaluation of the Open-Language Learning Initiative for English (OLLI English). It is sponsored by the U.S. 

Department of Education, as part of a collaboration project with China‘s Ministry of Education. RTI International, a 

US-based non-profit research organization, is conducting this study. We have also asked East China Normal 

University (ECNU) to help us.

The purpose of the study is to examine the effectiveness of English language learning software on students‘ English 

proficiency. We would like to measure students‘ English language proficiency in listening and reading. To do this, 

we will give the students some tests that will take about 1.5 hours to complete. In addition, we will ask your child to 

fill out a questionnaire about his or her experience and thoughts with learning English. Your child will be one of 

about 6,000 middle school students participating in this study. Your child‘s participation is voluntary. You or your 

child may withdraw from the study at any point. 

There is no penalty if you or your child decides not to participate. Your child may choose not to answer any 

question. There are no specific risks to your child from taking part in the study but he or she might feel slightly 

uncomfortable with the test administrator because your child might not have met him or her before. There are no 

other direct benefits to your child from participating, but he or she will be helping us understand the effectiveness of 

English language learning software on students‘ English proficiency. Your child‘s name will not be recorded on the 

tests or the questionnaire but will only be represented by an ID. We are required to keep all information confidential 

and your child‘s performance on these tests will not affect his or her grades at school. Only authorized researchers at 

RTI and ECNU will know your child‘s test scores and responses to the questionnaire. They will not be shared with 

the school, the teachers, the U.S. Department of Education, China‘s Ministry of Education, or with you. 

If you do not wish to have your child participate, please tear off the bottom portion of this letter and have your child 

give it to the test administrator. If you have any questions about this study or your child‘s participation, please feel 

free to ask the test administrator that will be at your child‘s school. We thank you in advance for your cooperation in 

this important study.

Sincerely,

Patricia J. Green

Project Director

RTI International

--------------------------------------------------------------------------------------------------------------------

I do not wish to have my child participate in this research.
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

F-14 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 5. Parent Letter – Comparison Schools - Non TFW Parents

Dear Parent or Guardian:

We are pleased to inform you that your child‘s school has been selected to participate in an important study called 

the Evaluation of the Open-Language Learning Initiative for English (OLLI English). It is sponsored by the U.S. 

Department of Education, as part of a collaboration project with China‘s Ministry of Education. RTI International, a 

US-based non-profit research organization, is conducting this study. We have also asked East China Normal 

University (ECNU) to help us.

The purpose of the study is to examine the effectiveness of English language learning. We would like to measure 

students‘ English language proficiency in listening and reading. To do this, we will give the students some tests that 

will take about 1.5 hours to complete. In addition, we will ask your child to fill out a questionnaire about his or her 

experience and thoughts with learning English. Your child will be one of about 6,000 middle school students 

participating in this study. Your child‘s participation is voluntary. You or your child may withdraw from the study at 

any point. 

There is no penalty if you or your child decides not to participate. Your child may choose not to answer any 

question. There are no specific risks to your child from taking part in the study. There are no other direct benefits to 

your child from participating, but he or she will be helping us understand the effectiveness of English language 

learning in China. Your child‘s name will not be recorded on the tests or the questionnaire but will only be 

represented by an ID. We are required to keep all information confidential and your child‘s performance on these 

tests will not affect his or her grades at school. Only authorized researchers at RTI and ECNU will know your 

child‘s test scores and responses to the questionnaire. They will not be shared with the school, the teachers, the U.S. 

Department of Education, China‘s Ministry of Education, or with you. 

If you do not wish to have your child participate, please tear off the bottom portion of this letter and have your child 

give it to the test administrator. If you have any questions about this study or your child‘s participation, please feel 

free to ask the test administrator that will be at your child‘s school. We thank you in advance for your cooperation in 

this important study.

Sincerely,

Patricia J. Green

Project Director

RTI International

--------------------------------------------------------------------------------------------------------------------

I do not wish to have my child participate in this research.
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaire

A Study of a Gaming Approach to English Language Learning for Middle School Students F-15

Attachment 6. Consent Form for Participation in the Teacher Questionnaire - TFW

Dear Teacher:

Thank you for volunteering to participate in this study. This study is an Evaluation of the Open-Language Learning 

Initiative for English (OLLI English). It is sponsored by the U.S. Department of Education, as part of a collaboration 

project with China‘s Ministry of Education. RTI International, a US-based non-profit research organization, is 

conducting this study. We have also asked East China Normal University (ECNU) to help us.

The purpose of the study is to examine the effectiveness of English-language learning software on students' English 

proficiency. We would like to measure students‘ English language proficiency in listening and reading. At this time, 

we are asking you to fill out a questionnaire about your experience and thoughts with teaching English. It will take 

about 10 minutes to complete. You will be one of about 90 middle school teachers participating in this study.

Your participation is voluntary. You may withdraw from the study at any point. There is no penalty if you decide 

not to participate. You may choose not to answer any question. There are no specific risks to you from taking part in 

the study. There are no other direct benefits to you from participating, but you will be helping us understand 

teachers‘ experiences in teaching English . Your name will not be recorded on the questionnaire, but will be 

represented by an ID. We are required to keep all information confidential and your responses in the questionnaire 

will not affect your status at school. Only authorized researchers at RTI and ECNU will know your responses to the 

questionnaire. They will not be shared with the school, others teachers, the U.S. Department of Education, China‘s 

Ministry of Education, or with your students.

If you have any questions about this study or your participation, please feel free to ask the questionnaire 

administrator. 

We thank you in advance for your help!

Sincerely,

Patricia J. Green

Project Director

RTI International
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

F-16 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 7. Consent Form for Participation in the Teacher Questionnaire – NO 

TFW

Dear Teacher:

Thank you for volunteering to participate in this study. This study is an Evaluation of the Open-Language Learning 

Initiative for English (OLLI English). It is sponsored by the U.S. Department of Education, as part of a collaboration 

project with China‘s Ministry of Education. RTI International, a US-based non-profit research organization, is 

conducting this study. We have also asked East China Normal University (ECNU) to help us.

The purpose of the study is to examine the effectiveness of English-language learning. We would like to measure 

students‘ English language proficiency in listening and reading. At this time, we are asking you to fill out a 

questionnaire about your experience and thoughts with teaching English. It will take about 10 minutes to complete. 

You will be one of about 90 middle school teachers participating in this study.

Your participation is voluntary. You may withdraw from the study at any point. There is no penalty if you decide 

not to participate. You may choose not to answer any question. There are no specific risks to you from taking part in 

the study. There are no other direct benefits to you from participating, but you will be helping us understand 

experiences and thoughts about teaching English. Your name will not be recorded on the questionnaire, but will be 

represented by an ID. We are required to keep all information confidential and your responses in the questionnaire 

will not affect your status at school. Only authorized researchers at RTI and ECNU will know your responses to the 

questionnaire. They will not be shared with the school, others teachers, the U.S. Department of Education, China‘s 

Ministry of Education, or with your students.

If you have any questions about this study or your participation, please feel free to ask the questionnaire 

administrator. 

We thank you in advance for your help!

Sincerely,

Patricia J. Green

Project Director

RTI International
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaire

A Study of a Gaming Approach to English Language Learning for Middle School Students F-17

Atachment 2a. Assent Form- Student TFW 

亲爱的同学:

谢谢你自愿参与本项研究。这是对Open-Language Learning Initiative for English (OLLI 

English) 

所做的一项评估。这项研究是由美国教育部及中国教育部共同主办,而由在美国的非营利

组织RTI International进行实施。我们也请了华东师范大学一起合作。

这项研究是为了要了解英语学习软件是否有助于学生的英文程度。我们希望了解学生们在

英文上的听、读的能力。研究办法是通过让学生完成一些大约需要1.5小时完成的测试。

我们也会请你填写一份问卷以便了解你在学习英文的经验和想法。你是约6000个参加研

究的中学学生之一。

你的参与是属于完全自愿的。你在任何时候都可以改变主意并决定停止参与,如果你决定

不要参与并不会受任何处分。你也可以决定不回答任何一个测验中的问题。你参与本项研

究不会遭受风险。

你参与本项研究也不会享受到任何直接利益,但是会帮助我们了解英语学习软件在提高学

生英语熟练程度上的有效性。你的名字并不会被纪录在测验卷或问卷上,你的名字会用代

码代替。我们被规定必须将所有资料加以保密,你在进行测验时所说的或做的一切,都不

会影响你在学校的成绩。唯有被授权的RTI和华东师范大学的研究员才会知道你的测试结

果和问卷回答。其它任何人,包括你的学校、老师、美国教育部、中国教育部或你的父母

都不会知道。

管理这次英文测验的长辈可以回答你有的任何问题,不管是关于此项研究或你的参与,如

果你有疑问请尽量发问。

在此先谢谢你的参与。

RTI International研究项目主管

Patricia Green
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

F-18 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 3a. Assent Form- Comparison Students – No TFW

亲爱的同学:

谢谢你自愿参与本项研究。这是对Open-Language Learning Initiative for English (OLLI 

English) 

所做的一项评估。这项研究是由美国教育部及中国教育部共同主办,而由在美国的非营利

组织RTI International进行实施。我们也请了华东师范大学一起合作。

这项研究是为了要了解英语语言学习的有效性。我们希望了解学生们在英文上的听、读的

能力。研究办法是通过让学生完成一些大约需要1.5小时完成的测试。我们也会请你填写

一份问卷以便了解你在学习英文的经验和想法。你是约6000个参加研究的中学学生之一

。

你的参与是属于完全自愿的。你在任何时候都可以改变主意并决定停止参与,如果你决定

不要参与并不会受任何处分。你也可以决定不回答任何一个测验中的问题。你参与本项研

究不会遭受风险。

你参与本项研究也不会享受到任何直接利益,但是会帮助我们了解英语语言学习的有效性

。你的名字并不会被纪录在测验卷或问卷上,你的名字会用代码代替。我们被规定必须将

所有资料加以保密,你在进行测验时所说的或做的一切,都不会影响你在学校的成绩。唯

有被授权的RTI和华东师范大学的研究员才会知道你的测试结果和问卷回答。其它任何人

,包括你的学校、老师、美国教育部、中国教育部或你的父母都不会知道。

管理这次英文测验的长辈可以回答你有的任何问题,不管是关于此项研究或你的参与,如

果你有疑问请尽量发问。

在此先谢谢你的参与。

RTI International研究项目主管

Patricia Green
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaire

A Study of a Gaming Approach to English Language Learning for Middle School Students F-19

Attachment 4a. Parent Letter – TFW Parents

敬爱的父母或监护人:

我们很高兴通知您,您的孩子的学校已被选择参与一个非常重要的研究项目,叫 OpenLanguage Learning Initiative for English (OLLI 

English)。这项研究是由美国教育部及中国教育部共同主办,而由在美国的非营利组织

RTI International进行实施。我们也请了华东师范大学 一起合作。

此项研究是要了解英语学习软件是否有助于学生的英文程度。我们会测量学生在英文上的

听、读的能力。研究办法是通过让学生完成一些大约需要1.5小时完成的测试。此外,我

们也会请你的孩子填写一份问卷以便了解他在学习英文的经验和想法。您的孩子是大约6

000个自愿参加此项研究的中学生之一。您的孩子的参与是纯属自愿的,您或您的孩子在

任何时候都可以改变主意并决定停止参与。

如果您或您的孩子决定不要参与,将不会受到任何处分。您的孩子可以决定不回答任何测

验中的问题。你的孩子参与本项研究不会遭受风险,但是他可能会觉得有些不习惯因为他

不一定认识管理这次英文测验和问卷的研究员。你的孩子参与本项研究也不会享受到任何

直接利益,但是会帮助我们了解英语学习软件在提高学生英语熟练程度上的有效性。您的

孩子的名字并不会被纪录在测验卷或问卷上,他的名字会用代码代替。我们被规定必须将

所有资料加以保密,您的孩子在测验上的表现不会影响他在学校的成绩。唯有被授权的R

TI和华东师范大学的研究员才会知道您的孩子的测试结果和问卷回答。其它任何人,包括

学校、老师、美国教育部、中国教育部或您都不会被告知。

如果您不希望您的孩子参与这次的研究项目,请撕下这封信下角,让您的孩子带回给管理

这次英文测验的研究员。如果您对此项研究或您的孩子的参与有任何问题,请洽询在您的

孩子的学校管理这次英文测验的研究员。感谢您配合这项重要的研究。

RTI International研究项目主管

Patricia Green 敬上

--------------------------------------------------------------

我不希望我的孩子参与这次的研究项目。
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

F-20 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 5a. Parent Letter – Non TFW Parents

敬爱的父母或监护人:

我们很高兴通知您,您的孩子的学校已被选择参与一个非常重要的研究项目,叫 OpenLanguage Learning Initiative for English (OLLI 

English)。这项研究是由美国教育部及中国教育部共同主办,而由在美国的非营利组织

RTI International进行实施。我们也请了华东师范大学 一起合作。

此项研究是要了解英语语言学习的有效性。我们会测量学生在英文上的听、读的能力。研

究办法是通过让学生完成一些大约需要1.5小时完成的测试。此外,我们也会请你的孩子

填写一份问卷以便了解他在学习英文的经验和想法。您的孩子是大约6000个自愿参加此

项研究的中学生之一。您的孩子的参与是纯属自愿的,您或您的孩子在任何时候都可以改

变主意并决定停止参与。

如果您或您的孩子决定不要参与,将不会受到任何处分。您的孩子可以决定不回答任何测

验中的问题。你的孩子参与本项研究不会遭受风险,但是他可能会觉得有些不习惯因为他

不一定认识管理这次英文测验和问卷的研究员。你的孩子参与本项研究也不会享受到任何

直接利益,但是会帮助我们了解中国英语语言学习的有效性。您的孩子的名字并不会被纪

录在测验卷或问卷上,他的名字会用代码代替。我们被规定必须将所有资料加以保密,您

的孩子在测验上的表现不会影响他在学校的成绩。唯有被授权的RTI和华东师范大学的研

究员才会知道您的孩子的测试结果和问卷回答。其它任何人,包括学校、老师、美国教育

部、中国教育部或您都不会被告知。

如果您不希望您的孩子参与这次的研究项目,请撕下这封信下角,让您的孩子带回给管理

这次英文测验的研究员。如果您对此项研究或您的孩子的参与有任何问题,请洽询在您的

孩子的学校管理这次英文测验的研究员。感谢您配合这项重要的研究。

RTI International研究项目主管

Patricia Green 敬上

--------------------------------------------------------------

我不希望我的孩子参与这次的研究项目。
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaire

A Study of a Gaming Approach to English Language Learning for Middle School Students F-21

Attachment 6a. Consent Form for Participation in the Teacher Questionnaire – TFW 

敬爱的老师:

谢谢您自愿参与本项研究。这是对Open-Language Learning Initiative for English (OLLI 

English)所做的一项评估。这项研究是由美国教育部及中国教育部共同主办,而由在美国

的非营利组织RTI International进行实施。我们也请了华东师范大学一起合作。

此项研究是要了解英语学习软件是否有助于学生的英文程度。我们会测量学生在英文上的

听、读的能力。目前我们会请您填写一份问卷以便了解你在英语教学的经验和想法。问卷

大约花10分钟即可完成。您是约90位参加研究的中学老师之一。

您的参与是属于完全自愿的。您在任何时候都可以改变主意并决定停止参与,如果您决定

不要参与并不会受任何处分。您也可以决定不回答任何一个测验中的问题。您参与本项研

究不会遭受风险。您参与本项研究也不会享受到任何直接利益,但是会帮助我们了解教师

在英语教学中的经验。您的名字并不会被纪录在问卷上。我们被规定必须将所有资料加以

保密,您在问卷上的答案不会影响您在学校的身份或待遇。唯有被授权的RTI和华东师范

大学研究员才会知道您的问卷回答。其它任何人,包括您的学校、在学校的同事、美国教

育部、中国教育部或您的学生都不会知道。

管理这次问卷的研究员可以回答您有的任何问题,不管是关于此项研究或你的参与,如果

您有疑问请尽量发问。

在此先谢谢你的参与。

RTI International研究项目主管

Patricia Green敬上
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

F-22 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 7a. Consent Form for Participation in the Teacher Questionnaire – NO 

TFW

敬爱的老师:

谢谢您自愿参与本项研究。这是对Open-Language Learning Initiative for English (OLLI 

English)所做的一项评估。这项研究是由美国教育部及中国教育部共同主办,而由在美国

的非营利组织RTI International进行实施。我们也请了华东师范大学一起合作。

此项研究是要了解英语语言学习的有效性。我们会测量学生在英文上的听、读的能力。目

前我们会请您填写一份问卷以便了解你在英语教学的经验和想法。问卷大约花10分钟即

可完成。您是约90位参加研究的中学老师之一。

您的参与是属于完全自愿的。您在任何时候都可以改变主意并决定停止参与,如果您决定

不要参与并不会受任何处分。您也可以决定不回答任何一个测验中的问题。您参与本项研

究不会遭受风险。您参与本项研究也不会享受到任何直接利益,但是会帮助我们了解教师

在英语教学中的经验。您的名字并不会被纪录在问卷上。我们被规定必须将所有资料加以

保密,您在问卷上的答案不会影响您在学校的身份或待遇。唯有被授权的RTI和华东师范

大学研究员才会知道您的问卷回答。其它任何人,包括您的学校、在学校的同事、美国教

育部、中国教育部或您的学生都不会知道。

管理这次问卷的研究员可以回答您有的任何问题,不管是关于此项研究或你的参与,如果

您有疑问请尽量发问。

在此先谢谢你的参与。

RTI International研究项目主管

Patricia Green敬上
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaire

A Study of a Gaming Approach to English Language Learning for Middle School Students F-23

Attachment 8. Teacher Questionnaire

The following questions are about your experience and thoughts. There are no right or 

wrong answers. Please read all questions thoroughly and provide a response to the best of 

your ability. Directions are provided at each question. Please mark the box next to your 

answer choice, like this: √ For some questions, you will be asked to write your responses 

using words or numbers.

1. Are you male or female? Male Female

2. What year were you born? |_1_|_9_|___|___|

3. How many years have you taught English? |___|___| years

4. The following questions are about your English classes this semester. 

a. How many classroom hours of English classes do you teach on a weekly basis?

 |___|___|

b. How many English classes do you teach? |___| classes

c. What are the grade levels? 7

th grade 8th grade 9th grade

5. Do you have a degree in English? No Yes

6. The following questions are about the highest degree you have completed. Using the list on the 

bottom, please fill in the corresponding number.

a. What is the highest level of degree you have completed? |___|

b. What is the highest level of degree in English you have completed? |___| (Write

 0 if you do not have a degree in English)

List of Highest Degree Completed

(1) Less than High School

(2) High School Diploma

(3) 3-Year Normal High School Diploma

(4) Associate Degree, including 5-Year Normal College

(5) 4-Year University

(6) Advanced Degree

7. During this semester, have you participated in teachers’ training related to English teaching 

that was organized by your school? (Mark one answer)

No

Yes  How many days in total? |___|___| days
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

F-24 A Study of a Gaming Approach to English Language Learning for Middle School Students

8. During this semester, have you participated in teachers’ training related to English teaching 

that was organized by an entity other than your school? (Mark one answer)

No

Yes  How many days in total? |___|___| days

9. Currently, how experienced are you with using instructional technology to improve classroom 

instruction for English? (Mark one answer)

Not experienced at all

Less experienced

Somewhat experienced

Very experienced

10. Have you used computer software for teaching 8th grade English classes? (Mark one answer)

No

Yes  What kind of software? Please specify: __________________________

11. What are some teaching methods that you frequently use for teaching 8th grade English 

classes? (Mark all that apply)

Give lectures according to textbook chapters

Incorporate supporting materials for classroom instruction 

Design English language games and activities

Focus on test-taking skills

Something else  What are they? Please specify: 

_____________________________

12. In general, what percentage of time do you use Chinese (including lecture, oral questioning, 

dialogues, etc.) to teach 8th grade English classes? 

Less than 20%

20% - 50%

50% - 80%

More than 80% 

13. Do you have a computer at home?

Yes

No

14. Do you have access to the internet at home?

Yes

No
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaire

A Study of a Gaming Approach to English Language Learning for Middle School Students F-25

15. What is your username for the Forgotten World? 

16. Being as objective as you can, how motivated are your 8th grade students in terms of learning 

English?

Not motivated at all

Less motivated

Somewhat motivated

Very motivated

17. Currently, which of the following apply to you? (Mark all that apply)

I have traveled to English-speaking countries

I use English to speak with relatives, family, or friends

I read English language books and magazines, newspapers, and Web sites

18. How often do you watch English language TV programs for at least 15 minutes in a sitting? 

(Mark one answer)

Never

Once every few months

Once or twice a month

Once or twice a week

Three or four times a week

Everyday

19. How often do you use English to have online chat with people or write e-mails?

Never

Once every few months

Once or twice a month

Once or twice a week

Three or four times a week

Everyday

20. How important do you feel that learning English is to middle school students in this area? 

(Mark one answer)

Not important at all

Less important

Somewhat important

Very important

Thank you for completing this questionnaire!

We will keep your responses confidential.
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

F-26 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 8a. Teacher Questionnaire (Chinese)

本问卷主要是为了了解您在英语教学方面的经历和想法。任何回答都没有对错之分。希

望你能够认真阅读题目,并选择最适合的回答。请根据题目指示,在方框内打勾,如 

。有些题目会请您以文字或数字写出您的回答。 

1. 您的性别是: 男 ; 女

2. 您的出生年份是: |_1_|_9_|___|___|

3. 您从事英语教学有多少年了? |___|___| 年

4. 请回答下列关于您在本学期担任英语教学方面的问题。

 a. 您每周担任英语教学的课时数是:|___|___|

d. 您担任英语教学的班级数是: |___| 个班级

e. 有哪些年级? 7 年級 ; 8 年級 ; 9 年級

5. 您是不是英语专业毕业? 不是 ; 是

6. 下列问题有关于您的最高学历。请用题目下方的最高学历表,填入相关号码。

a. 您所取得的最高学历是: |___|

b. 您在英语专业方面取得的最高学历是: |___| (若不是英语专业毕业,填 0 )

最高学历表

(1) 高中以下

(2) 高中学历

(3) 3年制中师或中专学历

(4) 大学专科学历(含5年中师)

(5) 大学本科毕业

(6) 大学研究生毕业

7. 本学年您是否还参与过由本校组织,英语教学方面的教师培训活动?(单项选择)

没参与

有参与  总合计时间大约为:|___|___| 天
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaire

A Study of a Gaming Approach to English Language Learning for Middle School Students F-27

8. 本学年您是否还参与过由其它方面组织 (不包括本校),英语教学方面的教师培训

 活动?(单项选择)

没参与

有参与  总合计时间大约为:|___|___| 天

9. 目前,您在使用教育技术开展英语课堂教学方面的能力如何?(单项选择)

完全不会

比较弱

比较强

很强

10. 请问,您有沒有在8年級的英语教学中使用过计算机软件?(单项选择)

没用过

有用过  是什么样的计算机软件? 请写出: __________________________

11. 您在8年級的英语教学中常用的教学方法有哪些?(多项选择)

按照教材内容教师讲解为主

找一些辅助材料用于课堂教学

设计一些英语游戏活动

以围绕应付考试为主而教学

其他  是什么样的方式? 请写出:__________________________________

12. 一般來説,您在8年級的英语课堂中运用中文(包括讲解、提问、对话等)的时间 

 比例是多少?(单项选择) 

20%以内

20% 到50%

50% 到80%

80% 以上

13. 你在家里有电脑吗?(单项选择) 

有

没有
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

F-28 A Study of a Gaming Approach to English Language Learning for Middle School Students

14. 你在家里能上网吗?(单项选择) 

能

不能

15. 你的《被遗忘的世界》的用户名是什么? 。

16. 请客观地评价目前您所任教的8年級学生学习英语的积极性情况?(单项选择)

非常低

比较低

比较高

非常高

17. 目前,下列哪种情况适合于您? (多项选择)

我曾经到英语国家旅游过

我平时会用英语与亲戚、家人或者朋友对话

我平时会阅读英文书刊、报纸或者上英文网站阅读

18. 您多久收看有英语对话的电视节目,且每次至少看15分钟?(单项选择)

从来不看

几个月看 1次

每月看1到2次

每周看1到2次

每周看3到4次

每天看
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaire

A Study of a Gaming Approach to English Language Learning for Middle School Students F-29

19. 您多久会在网络上用英文与人聊天或写邮件? (单项选择)

从来不会

几个月 1次

每月1到2次

每周1到2次

每周3到4次

每天

20. 您认为,本地初中学生学习英语的重要性程度如何?(单项选择)

非常不重要

比较不重要

比较重要

非常重要

非常感谢您认真回答了本问卷!

我们将保密您的回答。
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

F-30 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 9. Student Questionnaire

The following questions are about your experience and thoughts in learning English. There 

are no right or wrong answers. Please read all questions thoroughly and provide a response 

to the best of your ability. Directions are provided at each question. Please mark the box 

next to your answer choice, like this: √ For some questions, you will be asked to write 

your response.

1. Are you a boy or a girl? Boy Girl

2. What year were you born? |_1_|_9_|_9_|___|

3. In what grade did you first start learning English at school? |___| grade

4. Currently, how important is learning English to you? (Mark one answer)

Not important at all

Not too important

Somewhat important

Very important

5. Currently, which of the following apply to you? (Mark all that apply) 

I have traveled to English-speaking countries

I would use English to speak with relatives, family, or friends

I would read English language books and magazines, newspapers, and Web sites

6. How often do you watch English language TV programs for at least 15 minutes in a

(Mark one answer)

Never

Once every few months

Once or twice a month

Once or twice a week

Three or four times a week

Everyday

7. How often do you use English to have online chat with people or write e-mails? (Mark one 

answer)

Never

Once every few months

Once or twice a month

Once or twice a week

Three or four times a week

Everyday
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaire

A Study of a Gaming Approach to English Language Learning for Middle School Students F-31

8. How often do you play computer games that have vocabulary in English? 

(Mark one answer)

Never

Once every few months

Once or twice a month

Once or twice a week

Three or four times a week

Everyday

9. Do you go to an English tutoring class outside of school? (Mark one answer)

Never

I have, but not anymore

I do, but have not before

I always have

10. Currently, how interested are you in learning English? (Mark one answer)

Not interested

Less interested

More interested

Very interested

11. Students in general may experience difficulty when they are learning English. Currently, how 

difficult are the following areas to you? (Mark one answer per row)

Content

Level of Difficulty

Not Difficult At 

All A Little Difficult Somewhat Difficult Very Difficult

a. Vocabulary

b. Grammar

c. Listening

d. Reading

e. Writing

f. Speaking

12. Realistically, do you expect to attend university some day? (Mark one answer)

No

Yes

Unsure
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

F-32 A Study of a Gaming Approach to English Language Learning for Middle School Students

13. Do you have a computer at home?

Yes

No

14. Do you have access to the internet at home?

Yes

No

15. How often do you play computer or video games?

Never

Once every few months

Once or twice a month

Once or twice a week

Three or four times a week

Everyday

16. What is your username for the Forgotten World? .

Thank you for completing this questionnaire!

We will keep your responses confidential.
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaire

A Study of a Gaming Approach to English Language Learning for Middle School Students F-33

Attachment 9a. Student Questionnaire

本问卷主要是为了了解你在学习英文方面的经历和想法。任何回答都没有对错之分。希

望你能够认真阅读题目,并选择最适合的回答。请根据题目指示,在方框内打勾,如 

。有些题目会指示你写出你的回答。

1. 你的性别是: 男孩; 女孩

2. 你的出生年份是: |_1_|_9_|_9_|___|

3. 你在学校中开始学习英语的年级是: |___| 年級

4. 目前,你认为学习英语重要吗?(单项选择) 

完全不重要

不太重要

有些重要

非常重要

5. 目前,下列哪种情况适合于你?(多项选择) 

我曾经到英语国家旅游过

我平时会用英语与亲戚、家人或者朋友对话

我平时会阅读英文书刊、报纸或者上英文网站阅读

6.你多久收看有英语对话的电视节目,且每次至少看15分钟?(单项选择) 

从来不看

几个月看 1次

每月看1 到2次

每周看1到2次

每周看3到5次

每天看
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

F-34 A Study of a Gaming Approach to English Language Learning for Middle School Students

7. 你多久会在网络上用英文与人聊天或写邮件?(单项选择) 

从来不会

几个月 1次

每月1 到2次

每周1到2次

每周3到5次

每天

8. 你多久在计算机上玩有英语单词的电子游戏?(单项选择) 

从来不玩

几个月玩 1次

每月玩1到2次

每周玩1到2次

每周玩3到4次

每天玩

9. 你有没有参加校外英语辅导班?(单项选择) 

从来没有过

以前有, 现在没有

以前没有,现在有

一直参加校外班

10. 目前,你学习英语的兴趣程度如何?(单项选择) 

没有

较小

较大

很大

 
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaire

A Study of a Gaming Approach to English Language Learning for Middle School Students F-35

11. 一般学生在英语学习中有时会遇到困难。目前,下列方面对你来说有多大困难? 

 (请在合适的方框内打勾,每行只能选一个) 

内容 困难程度

没有困难 困难较小 困难较大 困难很大

a. 单词

b. 语法

c. 听力

d. 阅读

e. 写作

f. 口语

12. 从实际出发,你期望将来能够进大学吗?(单项选择) 

不期望

有期望

不知道

13. 你在家里有电脑吗?(单项选择) 

有

没有

14. 你在家里能上网吗?(单项选择) 

能

不能
Appendix F – Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

F-36 A Study of a Gaming Approach to English Language Learning for Middle School Students

15. 你多久玩电脑游戏或其他电子游戏?(单项选择) 

从来不玩

几个月玩 1次

每月玩1到2次

每周玩1到2次

每周玩3到4次

每天玩

16. 你的《被遗忘的世界》的用户名是什么? 。

 

感谢你认真完成了本问卷! 

我们将对你的回答保密。
G-1

Appendix G.

Data Collection Procedures and Materials: 2010 PostTest Test Administrator Manual, Attached Informed 

Consent Forms, and Questionnaires

G-3

RTI Project 0209234.002.008

June, 2010

Test Administrator’s Manual

Evaluation of the E-Learning System: OpenLanguage Learning Initiatives for English

Main Study

Lanzhou, Gansu

China

Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-5

1. Background and Purpose of the Evaluation

The U.S. Department of Education [Policy and Program Studies Service (PPSS), within 

the Office of Planning, Evaluation, and Policy Development (OPEPD)], is sponsoring an 

evaluation of the E-language learning system named Open Language Learning Initiative for 

English. (OLLI English). The computer-based game developed to help students learn English is 

called The Forgotten Word (TFW).

RTI International (RTI) is carrying out the evaluation which is led by Dr. Patricia 

Green. RTI is collaborating with East China Normal University (ECNU) for the work in 

China. The project director at ECNU is Professor Zhu Yiming, a professor of the ECNU 

Department of Education.

This evaluation has been scheduled to take place in Chinese middle schools for two 

semesters from September 2009 to June 2010. The Chinese Ministry of Education identified 

Gansu Province as a site that fit the requirements for the evaluation. The provincial authorities 

selected the sampled schools. The evaluation aims to answer the following key evaluation 

question:

Do students exposed to TFW demonstrate increases in proficiency in reading, writing, 

listening, and speaking skills when compared to students without access to TFW, given the same 

curriculum and under conditions that allow reasonable access to the software?

The role of the test administrators is to:

1. Confirm (and update if needed) the current student roster with the student tracking form.

2. Distribute consent forms to all students and read aloud the form.

3. Administer reading and listening tests. 

4. Administer a questionnaire to all students participating in the survey about their thoughts 

and experience about learning English (attached to the listening test).

5. Administer a questionnaire to teachers participating in the program study. The 

questionnaire asks about the teachers‘ thoughts and experience about teaching English. There 

are two versions of the questionnaire: one for teachers in the ―treatment‖ schools that are 

using The Forgotten World and one for teachers in ―comparison‖ schools.

2. Verifying the student roster with the Student Tracking 

Form

2.1 Prior to Test Administration - Student Roster vs. Student 

Tracking Form

The first important step is to obtain a current student roster for each selected classroom at 

the school PRIOR to the test day (or at least 2 hours before testing is scheduled to begin). You 

will need to review each student name listed on the current student roster against the tracking 

form. A sample student tracking form is included in this manual as Attachment 1.
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-6 A Study of a Gaming Approach to English Language Learning for Middle School Students

■ If a student is missing from the student tracking form, you must add this student to the bottom 

of the tracking form on a line that has been assigned an ID but has no name. Please put the 

student‘s name on the list. It is very important that we know who has been added to which 

classroom and school. Please check with these students and make sure they include their 

TFW User ID on the questionnaire, or make a note that they do not have one assigned.

■ If you notice a spelling change, please update the student tracking form with the correct 

spelling.

■ If you have more than 3 additional students on the student list in the class NOT listed on the 

tracking form, you should assign new student IDs, using the same school and classroom IDs, 

but adding student IDs beginning with 601. 

2.2 On Test Day

All test materials have been bundled by classroom. Test materials for each student will be 

inside individual envelopes. The outside of each envelope will list the student‘s name and ID –

YOU MUST DISTRIBUTE THE ENVELOPES TO STUDENTS INDIVIDUALLY, 

MATCHING THE NAME ON THE OUTSIDE OF THE ENVELOPE TO THE CORRECT 

STUDENT. If you have added students to the student tracking form, you must give the newly 

added student the corresponding student packet ID.

The students should confirm that they have received the correct envelope by verifying the 

name on the outside of the envelope.

During the test, or after the test, please use the student tracking form to check off 

which students take the test. Please make a check after the student ID on the tracking form.

3. Consent Forms

3.1 Parent Letter 

We will give the schools a letter addressed to parents of the 8th grade students who are 

participating in the study, and ask the schools to distribute the letters to the parents. The letter 

describes the purpose of the study, students‘ participation, and confidentiality. Parents who do 

not wish to have their child participate in the study can tear off the bottom portion of the letter 

that says, ―I do not wish to have my child participate in this research‖ and return it to the test 

administrator. This student shall then be excluded from the study. Samples of the letter are 

included in English as Attachments 4 and 5 (and 4a and 5a in Chinese).

3.2 Student Assent Form 

Before the start of the English language assessment tests and student questionnaire, the 

test administrator should give each student a copy of the student assent form and read it aloud to 

the students. The form will explain the purpose of the study, the voluntary nature of their 

participation, and that they are asked to complete assessment tests for reading and listening, as 

well as a questionnaire about their thoughts and experience with learning English. The form also 

assures the students that their individual test scores or responses to the questionnaire will not be 

shared with their teachers, their parents, China‘s Ministry of Education, or the U.S. Department 

of Education. To ensure confidentiality, the students‘ name will not be recorded on the tests but 
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-7

will be represented by an ID number. If a student does not wish to participate, he or she shall be 

excluded from the study. Copies of the forms are provided in English as Attachments 2 and 3 

(and 2a and 3a in Chinese).

3.3 Teacher Consent Form for Participation

Before administering the teacher‘s questionnaire, the administrator should give a consent 

form to each teacher. The form will explain the purpose of the study, the voluntary nature of 

their participation, and that they are asked to complete a questionnaire about their thoughts and 

experience surrounding teaching English. The form also assures the teachers that individual 

responses to the questionnaire will not be shared with the school, other teachers, China‘s 

Ministry of Education, or the U.S. Department of Education. If a teacher does not wish to 

participate, he or she shall be excluded from the study. Copies of the forms are provided in 

English as Attachments 6 and 7 (and 6a and 7a-Chinese). A copy of the Teacher Questionnaire 

for treatment schools is provided as Attachment 8 and 8a. The Teacher Questionnaire for 

comparison schools is provided as Attachment 9 and 9a.

4. Assessment Tests and Questionnaires

As mentioned previously, you will be administering the reading test, the listening test and 

the student questionnaire to all students sampled for this study. It is very important that you 

follow the scripts for each test – each student must hear the same information and you must 

read/play the entire script for each test.

4.1 Reading Test

The reading test is also called the  ̳IPT‘ test and all items are in one booklet. Test 

administrators distribute the envelopes (that contain both tests and questionnaires) before 

beginning to read the directions for the reading test. After the envelopes are distributed, instruct 

students to take out the reading test booklet.

When all students have the reading test on their desk, instruct them to open the booklet 

and begin reading the script with the instructions for the first section.

The reading test has five sections and you will need to provide instructions (from the 

script) before each section.

A copy of the script (in Chinese and English) is provided in Attachment 10. This script 

must be practiced in advance of administering the test.

A copy of the reading test is provided as Appendix A.

4.2 Listening Test

Directions for the listening test must be played for the students using a cassette tape 

recorder. Before beginning the tape, ask students to take the listening test from the envelope and 

instruct them to open the booklet to the first page. Test administrators should explain to students 

how the bubbles should be marked and then play the standardized recording for the test. Make 
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-8 A Study of a Gaming Approach to English Language Learning for Middle School Students

sure that all students understand what to do before the recording is played. As part of the test 

instructions, the recording will cover the function of the bubbles again. The test instructions can 

be replayed, but the actual English prompts should only be played once.

A copy of the booklet containing the listening test and questionnaire is in Appendix B.

4.3 Student Questionnaire

The student questionnaire is the same booklet as the listening test. This part of the 

assessment asks the student how they feel about learning English. Some students may be getting 

tired of the test at this point, so you should encourage students to complete it and do their best.

4.4 Teacher Questionnaire

The teacher questionnaires will be distributed to eighth-grade English teachers on arrival 

to school and collected before departing from the school. A copy of the teacher questionnaire for 

treatment schools is provided in Attachment 8 and for comparison schools in Attachment 9.

5. Collecting the Test Booklets

When students have completed the test and questionnaires, collect all booklets from 

students. The tests SHOULD NOT be put back into the envelopes. Please leave the envelopes in 

the classroom or in a trash container at the school.

Thank the students and teacher for their participation before leaving the classroom.

Before leaving the school, please check off each booklet, including the teacher 

questionnaire, on the Student Tracking Form.

Please bundle books from each class together and place in mailing box (do not use boxes 

that were sent from U.S. as they may be damaged).

DO NOT include the student tracking form in the box with the tests. If for any reason a 

box would be lost or stolen, it is important that a person looking at the tests would be unable to 

link a test with a student name. Please give the tracking forms to Dr. Zhu. 

Thank you for your time and effort in this important China and U.S. evaluation!

6. Test Procedures

Before going to school, check that you have:

■ Copy of the Student Tracking Form for each class that you will be testing.

■ Envelopes with booklets for all students on Tracking Form(s).

■ Copies of Student Assent Forms.

■ Script with instructions for reading test.

■ Tape and tape recorder for listening test.

In the classroom:
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-9

Step 1: Before distribute test booklets to the students

■ Introduce yourself to the class and explain purpose of the study.

■ Give the consent forms to all students.

■ Read aloud the consent form statement to the students. 

Step 2: Distribute test booklets

■ Distribute the envelopes with test booklets by calling student names one by one. Test 

administrators MUST distribute the tests on their own. Do NOT ask teachers and students to 

help. Tell students that they should NOT open the envelope until everyone is ready.

■ Instruct students to take the reading booklet from the envelope together. Tell students NOT to 

write down their names on the test booklets. Give a brief introduction on the test booklets. 

Reading test is in a separate booklet. Listening test and student questionnaire are in one test 

booklet together. 

■ Ask students to check if the 9 digit number on the envelope matches with the numbers on the 

two test booklets. If there are any errors, make a note of the ones which don‘t match, and 

make changes on the test booklets and rosters accordingly. 

■ Tell students that the whole test takes one and a half hours. The order of the tests is reading 

first (60 minutes), listening second (20 minutes) and student questionnaire last (10 minutes). 

Step 3: Start reading test 

■ Start reading test first. Tell students that they don‘t need to answer questions on page 16. 

■ Test administrators must read the instructions by following test administrator manual on page 

35 (attachment 10). The instructions must be read VERBATIM and no sentence should be 

skipped. Test administrator MUST read the instruction on their own. After the first part 

instruction, test administrators need to wait to start the next part till all the students are done 

with first part of the questions. Ask the students who answer the questions faster to wait to 

start the next section. There are 5 sections in the test. 

■ Ask students to cover their answers by using the yellow envelope. Emphasize to the students 

that they must NOT look at other people‘s answers. If they don‘t know the answer, they may 

leave it blank, but NO copying from other students. Test administrators will need to 

REPETITIVELY remind the students through the whole test. 

■ The time distribution of the reading test is described as follows: 

 Part 1: Vocabulary 5-10 minutes

 Part 2: Vocabulary in context 5-10 minutes

 Part 3: Reading for understanding 15-30 minutes

 Part 4: Reading for life skills 8-15 minutes

 Part 5: Language usage 4-9 minutes

■ When students finish reading test, ask them to put the test booklets on the left corner of the 

table and start the listening test. 

Step 4: Start listening test and student questionnaire 

■ All schools should use tape player or broadcast system for listening test. 

■ If broadcast system is being used in the listening test, all the classes must start the listening 

test TOGETHER after reading test. 

■ Instruct students to take the listening test booklet from the envelope. Give instructions for 

filling out ―bubbles‖ and begin playing tape recorded instructions.
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-10 A Study of a Gaming Approach to English Language Learning for Middle School Students

■ When students have finished listening test, tell them to turn the page and complete the 

questionnaire. Remind the students to read the instruction on the questionnaire before start 

answering questions. 

■ When students work on the questionnaire, test administrators can rewind the tape for the next 

use. 

Step 5: Collect test booklets 

■ Before you collect the test booklets, tell students they do NOT need to put the finished test 

booklets into the envelope. Tell students to keep the student consent forms and envelope. 

■ Ask the students to put the finished test booklets on the corner of the table. Test 

administrators must collect the test booklets one for one and check the two booklets. After all 

the test booklets are collected, test administrators thank students for participating and ask 

students to dismiss. 

Step 6: After the test session:

■ Complete the Student Tracking Form by checking off the IDs of all completed booklets.

■ If any unusual circumstances arose during the test, attach an explanatory note to the Student 

Tracking Form.
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-11

Attachment 1. Student Tracking Form

School in English School in Chinese School ID Class# Serial Number Student Name School test ID Class test ID Student test ID

10 1 1 Mandarin 910 201 301

1 2 910 201 302

1 3 910 201 303

1 4 910 201 304

1 5 910 201 305

1 6 910 201 306

1 7 910 201 307

1 8 910 201 308

1 9 910 201 309

1 10 910 201 310

1 11 910 201 311

1 12 910 201 312

1 13 910 201 313

1 14 910 201 314

1 15 910 201 315

1 16 910 201 316

1 17 910 201 317

1 18 910 201 318

1 19 910 201 319

1 20 910 201 320

1 21 910 201 321

1 22 910 201 322

1 23 910 201 323

1 24 910 201 324

1 25 910 201 325

1 26 910 201 326

1 27 910 201 327

1 28 910 201 328
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-12 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 2. Assent Form – Student - TFW

Dear Student:

Thank you for volunteering to participate in this study. This study is an Evaluation of the Open-Language Learning 

Initiative for English (OLLI English). It is sponsored by the U.S. Department of Education, as part of a collaboration 

project with China‘s Ministry of Education. RTI International, a US-based non-profit research organization, is 

conducting this study. We have also asked East China Normal University (ECNU) to help us.

The purpose of the study is to examine the effectiveness of English language learning software on students‘ English 

proficiency. We are interested in how students listen and read in English. To do this, we will give the students some 

tests that will take about 1.5 hours to complete. In addition, we will ask you to fill out a questionnaire about your 

experience and thoughts with learning English and using The Forgotten World software. You are one of about 6,000 

middle school students participating in this study.

Your participation is voluntary. You may change your mind and not participate at any point. If you decide not to 

participate, there is no penalty. You may choose not to answer any question. There are no specific risks to you from 

taking part in the study. 

There are no other direct benefits to you from participating, but you will be helping us understand the effectiveness 

of English language learning software on students‘ English proficiency. Your name will not be recorded on the tests 

or the questionnaire and you are only represented by an ID. We are required to keep all information confidential and 

what you say or do on these tests will not affect your grades at school. Only authorized researchers at RTI and 

ECNU will know your test scores and your responses to the questionnaire. No one else will know, not your school, 

your teachers, the U.S. Department of Education, China‘s Ministry of Education, or your parents.

If you have any questions about this study or your participation, please feel free to ask the test administrator.

We thank you in advance for your help!

Patricia J. Green

Project Director

RTI International
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-13

Attachment 3. Assent Form – Comparison Students – no TFW

Dear Student:

Thank you for volunteering to participate in this study. This is a study of the Evaluation of the Open-Language 

Learning Initiative for English (OLLI English). It is sponsored by the U.S. Department of Education, as part of a 

collaboration project with China‘s Ministry of Education. RTI International, a US-based non-profit research 

organization, is conducting this study. We have also asked East China Normal University (ECNU) to help us.

The purpose of the study is to examine the effectiveness of English language learning. We are interested in how 

students listen and read in English. To do this, we will give the students some tests that will take about 1.5 hours to 

complete. In addition, we will ask you to fill out a questionnaire about your experience and thoughts with learning 

English. You are one of about 6,000 middle school students participating in this study.

Your participation is voluntary. You may change your mind and not participate at any point. If you decide not to 

participate, there is no penalty. You may choose not to answer any question. There are no specific risks to you from 

taking part in the study. There are no other direct benefits to you from participating, but you will be helping us 

understand the effectiveness of English language learning. Your name will not be recorded on the tests or the 

questionnaire and you are only represented by an ID. We are required to keep all information confidential and what 

you say or do on these tests will not affect your grades at school. Only authorized researchers at RTI and ECNU will 

know your test scores and your responses to the questionnaire. No one else will know, not your school, your 

teachers, the U.S. Department of Education, China‘s Ministry of Education, or your parents.

If you have any questions about this study or your participation, please feel free to ask the test administrator.

We thank you in advance for your help!

Patricia J. Green

Project Director

RTI International
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-14 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 4. Parent Letter – Treatment Schools -TFW Parents

Dear Parent or Guardian:

We are pleased to inform you that your child‘s school has been selected to participate in an important study called 

the Evaluation of the Open-Language Learning Initiative for English (OLLI English). It is sponsored by the U.S. 

Department of Education, as part of a collaboration project with China‘s Ministry of Education. RTI International, a 

US-based non-profit research organization, is conducting this study. We have also asked East China Normal 

University (ECNU) to help us.

The purpose of the study is to examine the effectiveness of English language learning software on students‘ English 

proficiency. We would like to measure students‘ English language proficiency in listening and reading. To do this, 

we will give the students some tests that will take about 1.5 hours to complete. In addition, we will ask your child to 

fill out a questionnaire about his or her experience and thoughts with learning English and using The Forgotten 

World software. Your child will be one of about 6,000 middle school students participating in this study. Your 

child‘s participation is voluntary. You or your child may withdraw from the study at any point. 

There is no penalty if you or your child decides not to participate. Your child may choose not to answer any 

question. There are no specific risks to your child from taking part in the study but he or she might feel slightly 

uncomfortable with the test administrator because your child might not have met him or her before. There are no 

other direct benefits to your child from participating, but he or she will be helping us understand the effectiveness of 

English language learning software on students‘ English proficiency. Your child‘s name will not be recorded on the 

tests or the questionnaire but will only be represented by an ID. We are required to keep all information confidential 

and your child‘s performance on these tests will not affect his or her grades at school. Only authorized researchers at 

RTI and ECNU will know your child‘s test scores and responses to the questionnaire. They will not be shared with 

the school, the teachers, the U.S. Department of Education, China‘s Ministry of Education, or with you. 

If you do not wish to have your child participate, please tear off the bottom portion of this letter and have your child 

give it to the test administrator. If you have any questions about this study or your child‘s participation, please feel 

free to ask the test administrator that will be at your child‘s school. We thank you in advance for your cooperation in 

this important study.

Sincerely,

Patricia J. Green

Project Director

RTI International

--------------------------------------------------------------------------------------------------------------------

I do not wish to have my child participate in this research.
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-15

Attachment 5. Parent Letter – Comparison Schools - Non TFW Parents

Dear Parent or Guardian:

We are pleased to inform you that your child‘s school has been selected to participate in an important study called 

the Evaluation of the Open-Language Learning Initiative for English (OLLI English). It is sponsored by the U.S. 

Department of Education, as part of a collaboration project with China‘s Ministry of Education. RTI International, a 

US-based non-profit research organization, is conducting this study. We have also asked East China Normal 

University (ECNU) to help us.

The purpose of the study is to examine the effectiveness of English language learning. We would like to measure 

students‘ English language proficiency in listening and reading. To do this, we will give the students some tests that 

will take about 1.5 hours to complete. In addition, we will ask your child to fill out a questionnaire about his or her 

experience and thoughts with learning English. Your child will be one of about 6,000 middle school students 

participating in this study. Your child‘s participation is voluntary. You or your child may withdraw from the study at 

any point. 

There is no penalty if you or your child decides not to participate. Your child may choose not to answer any 

question. There are no specific risks to your child from taking part in the study. There are no other direct benefits to 

your child from participating, but he or she will be helping us understand the effectiveness of English language 

learning in China. Your child‘s name will not be recorded on the tests or the questionnaire but will only be 

represented by an ID. We are required to keep all information confidential and your child‘s performance on these 

tests will not affect his or her grades at school. Only authorized researchers at RTI and ECNU will know your 

child‘s test scores and responses to the questionnaire. They will not be shared with the school, the teachers, the U.S. 

Department of Education, China‘s Ministry of Education, or with you. 

If you do not wish to have your child participate, please tear off the bottom portion of this letter and have your child 

give it to the test administrator. If you have any questions about this study or your child‘s participation, please feel 

free to ask the test administrator that will be at your child‘s school. We thank you in advance for your cooperation in 

this important study.

Sincerely,

Patricia J. Green

Project Director

RTI International

--------------------------------------------------------------------------------------------------------------------

I do not wish to have my child participate in this research.
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-16 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 6. Consent Form for Participation in the Teacher Questionnaire - TFW

Dear Teacher:

Thank you for volunteering to participate in this study. This study is an Evaluation of the Open-Language Learning 

Initiative for English (OLLI English). It is sponsored by the U.S. Department of Education, as part of a collaboration 

project with China‘s Ministry of Education. RTI International, a US-based non-profit research organization, is 

conducting this study. We have also asked East China Normal University (ECNU) to help us.

The purpose of the study is to examine the effectiveness of English-language learning software on students' English 

proficiency. We would like to measure students‘ English language proficiency in listening and reading. At this time, 

we are asking you to fill out a questionnaire about your experience and thoughts with teaching English and using 

The Forgotten World software. It will take about 10 minutes to complete. You will be one of about 90 middle school 

teachers participating in this study.

Your participation is voluntary. You may withdraw from the study at any point. There is no penalty if you decide 

not to participate. You may choose not to answer any question. There are no specific risks to you from taking part in 

the study. There are no other direct benefits to you from participating, but you will be helping us understand 

teachers‘ experiences in teaching English. Your name will not be recorded on the questionnaire, but will be 

represented by an ID. We are required to keep all information confidential and your responses in the questionnaire 

will not affect your status at school. Only authorized researchers at RTI and ECNU will know your responses to the 

questionnaire. They will not be shared with the school, others teachers, the U.S. Department of Education, China‘s 

Ministry of Education, or with your students.

If you have any questions about this study or your participation, please feel free to ask the questionnaire 

administrator. 

We thank you in advance for your help!

Sincerely,

Patricia J. Green

Project Director

RTI International
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-17

Attachment 7. Consent Form for Participation in the Teacher Questionnaire – NO 

TFW

Dear Teacher:

Thank you for volunteering to participate in this study. This study is an Evaluation of the Open-Language Learning 

Initiative for English (OLLI English). It is sponsored by the U.S. Department of Education, as part of a collaboration 

project with China‘s Ministry of Education. RTI International, a US-based non-profit research organization, is 

conducting this study. We have also asked East China Normal University (ECNU) to help us.

The purpose of the study is to examine the effectiveness of English-language learning. We would like to measure 

students‘ English language proficiency in listening and reading. At this time, we are asking you to fill out a 

questionnaire about your experience and thoughts with teaching English. It will take about 10 minutes to complete. 

You will be one of about 90 middle school teachers participating in this study.

Your participation is voluntary. You may withdraw from the study at any point. There is no penalty if you decide 

not to participate. You may choose not to answer any question. There are no specific risks to you from taking part in 

the study. There are no other direct benefits to you from participating, but you will be helping us understand 

experiences and thoughts about teaching English. Your name will not be recorded on the questionnaire, but will be 

represented by an ID. We are required to keep all information confidential and your responses in the questionnaire 

will not affect your status at school. Only authorized researchers at RTI and ECNU will know your responses to the 

questionnaire. They will not be shared with the school, others teachers, the U.S. Department of Education, China‘s 

Ministry of Education, or with your students.

If you have any questions about this study or your participation, please feel free to ask the questionnaire 

administrator. 

We thank you in advance for your help!

Sincerely,

Patricia J. Green

Project Director

RTI International
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-18 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 2a. Assent Form- Student TFW 

亲爱的同学:

谢谢你自愿参与本项研究。这是对Open-Language Learning Initiative for English (OLLI 

English) 

所做的一项评估。这项研究是由美国教育部及中国教育部共同主办,而由在美国的非营利

组织RTI International进行实施。我们也请了华东师范大学一起合作。

这项研究是为了要了解英语学习软件是否有助于学生的英文程度。我们希望了解学生们在

英文上的听、读的能力。研究办法是通过让学生完成一些大约需要1.5小时完成的测试。

我们也会请你填写一份问卷以便了解你在学习英文和使用《被遗忘的世界》软件的经验和

想法。你是约6000个参加研究的中学学生之一。

你的参与是属于完全自愿的。你在任何时候都可以改变主意并决定停止参与,如果你决定

不要参与并不会受任何处分。你也可以决定不回答任何一个测验中的问题。你参与本项研

究不会遭受风险。

你参与本项研究也不会享受到任何直接利益,但是会帮助我们了解英语学习软件在提高学

生英语熟练程度上的有效性。你的名字并不会被纪录在测验卷或问卷上,你的名字会用代

码代替。我们被规定必须将所有资料加以保密,你在进行测验时所说的或做的一切,都不

会影响你在学校的成绩。唯有被授权的RTI和华东师范大学的研究员才会知道你的测试结

果和问卷回答。其它任何人,包括你的学校、老师、美国教育部、中国教育部或你的父母

都不会知道。

管理这次英文测验的长辈可以回答你有的任何问题,不管是关于此项研究或你的参与,如

果你有疑问请尽量发问。

在此先谢谢你的参与。

RTI International研究项目主管

Patricia Green
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-19

Attachment 3a. Assent Form- Comparison Students – No TFW

亲爱的同学:

谢谢你自愿参与本项研究。这是对Open-Language Learning Initiative for English (OLLI 

English) 

所做的一项评估。这项研究是由美国教育部及中国教育部共同主办,而由在美国的非营利

组织RTI International进行实施。我们也请了华东师范大学一起合作。

这项研究是为了要了解英语语言学习的有效性。我们希望了解学生们在英文上的听、读的

能力。研究办法是通过让学生完成一些大约需要1.5小时完成的测试。我们也会请你填写

一份问卷以便了解你在学习英文的经验和想法。你是约6000个参加研究的中学学生之一

。

你的参与是属于完全自愿的。你在任何时候都可以改变主意并决定停止参与,如果你决定

不要参与并不会受任何处分。你也可以决定不回答任何一个测验中的问题。你参与本项研

究不会遭受风险。

你参与本项研究也不会享受到任何直接利益,但是会帮助我们了解英语语言学习的有效性

。你的名字并不会被纪录在测验卷或问卷上,你的名字会用代码代替。我们被规定必须将

所有资料加以保密,你在进行测验时所说的或做的一切,都不会影响你在学校的成绩。唯

有被授权的RTI和华东师范大学的研究员才会知道你的测试结果和问卷回答。其它任何人

,包括你的学校、老师、美国教育部、中国教育部或你的父母都不会知道。

管理这次英文测验的长辈可以回答你有的任何问题,不管是关于此项研究或你的参与,如

果你有疑问请尽量发问。

在此先谢谢你的参与。

RTI International研究项目主管

Patricia Green
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-20 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 4a. Parent Letter – TFW Parents

敬爱的父母或监护人:

我们很高兴通知您,您的孩子的学校已被选择参与一个非常重要的研究项目,叫 OpenLanguage Learning Initiative for English (OLLI 

English)。这项研究是由美国教育部及中国教育部共同主办,而由在美国的非营利组织

RTI International进行实施。我们也请了华东师范大学 一起合作。

此项研究是要了解英语学习软件是否有助于学生的英文程度。我们会测量学生在英文上的

听、读的能力。研究办法是通过让学生完成一些大约需要1.5小时完成的测试。此外,我

们也会请你的孩子填写一份问卷以便了解他在学习英文和使用《被遗忘的世界》软件的经

验和想法。您的孩子是大约6000个自愿参加此项研究的中学生之一。您的孩子的参与是

纯属自愿的,您或您的孩子在任何时候都可以改变主意并决定停止参与。

如果您或您的孩子决定不要参与,将不会受到任何处分。您的孩子可以决定不回答任何测

验中的问题。你的孩子参与本项研究不会遭受风险,但是他可能会觉得有些不习惯因为他

不一定认识管理这次英文测验和问卷的研究员。你的孩子参与本项研究也不会享受到任何

直接利益,但是会帮助我们了解英语学习软件在提高学生英语熟练程度上的有效性。您的

孩子的名字并不会被纪录在测验卷或问卷上,他的名字会用代码代替。我们被规定必须将

所有资料加以保密,您的孩子在测验上的表现不会影响他在学校的成绩。唯有被授权的R

TI和华东师范大学的研究员才会知道您的孩子的测试结果和问卷回答。其它任何人,包括

学校、老师、美国教育部、中国教育部或您都不会被告知。

如果您不希望您的孩子参与这次的研究项目,请撕下这封信下角,让您的孩子带回给管理

这次英文测验的研究员。如果您对此项研究或您的孩子的参与有任何问题,请洽询在您的

孩子的学校管理这次英文测验的研究员。感谢您配合这项重要的研究。

RTI International研究项目主管

Patricia Green 敬上

--------------------------------------------------------------

我不希望我的孩子参与这次的研究项目。
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-21

Attachment 5a. Parent Letter – Non TFW Parents

敬爱的父母或监护人:

我们很高兴通知您,您的孩子的学校已被选择参与一个非常重要的研究项目,叫 OpenLanguage Learning Initiative for English (OLLI 

English)。这项研究是由美国教育部及中国教育部共同主办,而由在美国的非营利组织

RTI International进行实施。我们也请了华东师范大学 一起合作。

此项研究是要了解英语语言学习的有效性。我们会测量学生在英文上的听、读的能力。研

究办法是通过让学生完成一些大约需要1.5小时完成的测试。此外,我们也会请你的孩子

填写一份问卷以便了解他在学习英文的经验和想法。您的孩子是大约6000个自愿参加此

项研究的中学生之一。您的孩子的参与是纯属自愿的,您或您的孩子在任何时候都可以改

变主意并决定停止参与。

如果您或您的孩子决定不要参与,将不会受到任何处分。您的孩子可以决定不回答任何测

验中的问题。你的孩子参与本项研究不会遭受风险,但是他可能会觉得有些不习惯因为他

不一定认识管理这次英文测验和问卷的研究员。你的孩子参与本项研究也不会享受到任何

直接利益,但是会帮助我们了解中国英语语言学习的有效性。您的孩子的名字并不会被纪

录在测验卷或问卷上,他的名字会用代码代替。我们被规定必须将所有资料加以保密,您

的孩子在测验上的表现不会影响他在学校的成绩。唯有被授权的RTI和华东师范大学的研

究员才会知道您的孩子的测试结果和问卷回答。其它任何人,包括学校、老师、美国教育

部、中国教育部或您都不会被告知。

如果您不希望您的孩子参与这次的研究项目,请撕下这封信下角,让您的孩子带回给管理

这次英文测验的研究员。如果您对此项研究或您的孩子的参与有任何问题,请洽询在您的

孩子的学校管理这次英文测验的研究员。感谢您配合这项重要的研究。

RTI International研究项目主管

Patricia Green 敬上

--------------------------------------------------------------

我不希望我的孩子参与这次的研究项目。
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-22 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 6a. Consent Form for Participation in the teacher Questionnaire – TFW 

敬爱的老师:

谢谢您自愿参与本项研究。这是对Open-Language Learning Initiative for English (OLLI 

English)

所做的一项评估。这项研究是由美国教育部及中国教育部共同主办,而由在美国的非营利

组织RTI International进行实施。我们也请了华东师范大学一起合作。

此项研究是要了解英语学习软件是否有助于学生的英文程度。我们会测量学生在英文上的

听、读的能力。目前我们会请您填写一份问卷以便了解你在英语教学和使用《被遗忘的世

界》软件的经验和想法。问卷大约花10分钟即可完成。您是约90位参加研究的中学老师

之一。

您的参与是属于完全自愿的。您在任何时候都可以改变主意并决定停止参与,如果您决定

不要参与并不会受任何处分。您也可以决定不回答任何一个测验中的问题。您参与本项研

究不会遭受风险。您参与本项研究也不会享受到任何直接利益,但是会帮助我们了解教师

在英语教学中的经验。您的名字并不会被纪录在问卷上。我们被规定必须将所有资料加以

保密,您在问卷上的答案不会影响您在学校的身份或待遇。唯有被授权的RTI和华东师范

大学研究员才会知道您的问卷回答。其它任何人,包括您的学校、在学校的同事、美国教

育部、中国教育部或您的学生都不会知道。

管理这次问卷的研究员可以回答您有的任何问题,不管是关于此项研究或你的参与,如果

您有疑问请尽量发问。

在此先谢谢你的参与。

RTI International研究项目主管

Patricia Green敬上
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-23

Attachment 7a. Consent Form for Participation in the Teacher Questionnaire – NO 

TFW

敬爱的老师:

谢谢您自愿参与本项研究。这是对Open-Language Learning Initiative for English (OLLI 

English)

所做的一项评估。这项研究是由美国教育部及中国教育部共同主办,而由在美国的非营利

组织RTI International进行实施。我们也请了华东师范大学一起合作。

此项研究是要了解英语语言学习的有效性。我们会测量学生在英文上的听、读的能力。目

前我们会请您填写一份问卷以便了解你在英语教学的经验和想法。问卷大约花10分钟即

可完成。您是约90位参加研究的中学老师之一。

您的参与是属于完全自愿的。您在任何时候都可以改变主意并决定停止参与,如果您决定

不要参与并不会受任何处分。您也可以决定不回答任何一个测验中的问题。您参与本项研

究不会遭受风险。您参与本项研究也不会享受到任何直接利益,但是会帮助我们了解教师

在英语教学中的经验。您的名字并不会被纪录在问卷上。我们被规定必须将所有资料加以

保密,您在问卷上的答案不会影响您在学校的身份或待遇。唯有被授权的RTI和华东师范

大学研究员才会知道您的问卷回答。其它任何人,包括您的学校、在学校的同事、美国教

育部、中国教育部或您的学生都不会知道。

管理这次问卷的研究员可以回答您有的任何问题,不管是关于此项研究或你的参与,如果

您有疑问请尽量发问。

在此先谢谢你的参与。

RTI International研究项目主管

Patricia Green敬上
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-24 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 8. Teacher Questionnaire (Treatment Schools)

Teacher ID : 

The following questions are about your experience and thoughts. There are no right or wrong 

answers. Please read all questions thoroughly and provide a response to the best of your ability. 

Directions are provided at each question. Please mark the box next to your answer choice, like 

this:  For some questions, you will be asked to write your response using words or numbers.

1. What year were you born? |_1_|_9_|___|___|

2. We would like to know if the training you received on using The Forgotten World was helpful. 

How helpful was the training to you? (Mark one answer)

Not helpful at all

Not so helpful

Somewhat helpful

Very helpful

3. How often did you use The Forgotten World in class? (Mark one answer)

Never

Once every few months

Once or twice a month

Once or twice a week

Three to four times a week

Everyday

4. How often did you organize students to use The Forgotten World outside of class at school? 

(Mark one answer)

Never

Once every few months

Once or twice a month

Once or twice a week

Three to four times a week

Everyday

5. Now that you have used The Forgotten World, realistically, how likely do you think you will 

use it as part of the curriculum in the future? (Mark one answer)

Not likely at all

Not too likely

Somewhat likely

Very likely
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-25

6. Could you tell us more about your answer in Question 5?

_________________________________________________________

__________________________________________________________

7. Realistically, how likely do you think you will use some other computer software as part of 

the curriculum in the future? (Mark one answer)

Not likely at all

Not too likely

Somewhat likely

Very likely

8. In general, how helpful do you think The Forgotten World is to students’ English learning?

(Mark one answer)

Not helpful at all

Not so helpful

Somewhat helpful

Very helpful

9. In what area do you think that The Forgotten World can be helpful to students’ English 

learning? (Mark one answer per row)

Area

Level of Helpfulness

Not helpful at all A little helpful Somewhat 

helpful Very helpful

a. Vocabulary

b. Sentence

c. Listening

d. Reading

e. Writing

f. Speaking

g. Test-taking

10. Compared to the past, has student’s motivation for learning English changed since using The 

Forgotten World? (Mark one answer)

No change at all

A little more motivated

Somewhat more motivated

A lot more motivated
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-26 A Study of a Gaming Approach to English Language Learning for Middle School Students

11. What were the two biggest challenges that you encountered while using The Forgotten World?

(Mark two answers)

Insufficient time for using the software

Limited support from the school

Problems with computers at the school computer laboratory

Concerns with negative impact on test scores

Other challenges -> Please specify: _____________________________

12. In your opinion, what are some important procedures that need to be implemented so that 

The Forgotten World could be used to improve English instruction and students’ English 

proficiency?

_________________________________________________________________

_________________________________________________________________

13. For which grade level would The Forgotten World be most helpful to the students? (Mark one 

answer)

7

th grade

8

th grade

9

th grade

14. How often did you collaborate or communicate with other teachers in using The Forgotten 

World?

Never

Once every few months

Once or twice a month

Once or twice a week

Three to four times a week

Everyday

15. Has using The Forgotten World changed the way you think about teaching?

Yes

No

If yes, please explain.________________________________________________

16. Which 8th grade classes are you teaching? __________________________

17. Please list ALL the TFW usernames you have been using and in which classroom.

TFW username: ______________________ , used in classroom ____________________

TFW username: ______________________ , used in classroom ____________________
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-27

TFW username: ______________________ , used in classroom ____________________

If you have TFW usernames for other use besides classroom teaching, please specify. 

_____________________________________________________________________

_____________________________________________________________________

Thank you for completing this questionnaire!

We will keep your responses confidential.
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-28 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 8a. 教师问卷 (实验学校)

教师代码和姓名:

本问卷主要是为了了解您在英语教学方面的经历和想法。任何回答都没有对错之分。希

望你能够认真阅读题目,并选择最适合的回答。请根据题目指示,在方框内打勾,如

。有些题目会请您以文字或数字写出你的回答。

1. 您的出生年份是: |_1_|_9_|___|___|

2. 您参与了The Forgotten 

World软件使用培训,请问,这个培训对您开展实际教学有多大的帮助?(单项选择)

没有帮助

帮助较小

帮助较大

帮助很大

3. 您在课堂中多久使用The Forgotten World 这个学习材料的?(单项选择)

从不使用

几个月使用 1次

每月使用1到2次

每周使用1到2次

每周使用3到4次

每天使用

4. 您在课外又是多久组织学生在学校使用The Forgotten World 这个学习材料的?

 (单项选择)

从不使用

几个月使用 1次

每月使用1到2次

每周使用1到2次

每周使用3到4次

每天使用
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-29

5. 经过前段时间的使用,从实际出发,您现在认为把The Forgotten World 作为

以后课程教学的一部分,可能性有多大?(单项选择)

完全没有可能

可能性不大

有较大可能

 非常有可能

6. 请说明在第5题中您所勾选该答案的原因:

________________________________________________________________________

_______________________________________________________________________

7. 用其它计算机软件作为以后课程教学的一部分,可能性有多大?(单项选择)

完全没有可能

可能性不大

有较大可能

非常有可能

8. 总的来说,您认为The Forgotten 

World这个学习软件对于学生学习英语的帮助有多大?(单项选择)

没有帮助

帮助较小

帮助较大

帮助很大
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-30 A Study of a Gaming Approach to English Language Learning for Middle School Students

9. 您认为The Forgotten World这个学习软件在哪些方面有助于促进学生学习

英语?(请在合适的方框内打勾,每行只能选一个)

内容

帮助程度

没有帮助 帮助较小 帮助较大 帮助很大

a. 单词

b. 语法

c. 听力

d. 阅读

e. 写作

f. 口语

g. 考试

10. 请您评价经过使用The Forgotten World 

这个学习材料,总体上班级学生在学习英语的积极性与以往相比,有没有变化?(单项

选择)

没有变化

有少许提高

有较大提高

有很大提高
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-31

11. 您在使用The Forgotten World 开展教学时,遇到了最大的2项困难是什么?

 (可选2项)

没有可以更多的使用时间

学校对这项工作的重视不够

学校计算机教室设备有问题

担心影响学生的考试成绩

其他困难  请写出:

________________________________________________________________________

________________________________________________________________________

12. 您认为,如果要使用The Forgotten 

World学习软件来改进英语教学和提高学生的英语水平,需要采取哪些措施?

请简要说明:____________________________________________________________

________________________________________________________________________

13. 您认为The Forgotten 

World这个学习软件对于那个年级的学生学习英语的帮助最大?(单项选择)

7 年级

8 年级

9 年级

14. 您在使用The Forgotten World学习软件时,多久与其他教师进行合作或交流?

从不合作或交流

几个月合作或交流 1次

每月合作或交流1到2次

每周合作或交流1到2次

每周合作或交流3到4次

每天合作或交流
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-32 A Study of a Gaming Approach to English Language Learning for Middle School Students

15. 使用The Forgotten World学习软件有没有改变你对于英文教学的想法?

有

没有

 如果有改变,请具体说明。_____________________________________________

16. 您现在教8年级的哪些班级?_____________________________

17. 请列出您所使用的所有The Forgotten World用户名以及在哪个班级教学中使用。

TFW 用户名_________________________, 在哪个班级使用______________________

TFW 用户名_________________________, 在哪个班级使用______________________

TFW 用户名_________________________, 在哪个班级使用______________________

如果您将The Forgotten World用户名用于课堂教学以外的其他活动,请具体说明。

_________________________________________________________________________

_________________________________________________________________________

谢谢您认真回答本问卷!

我们将保密您的回答。
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-33

Attachment 9. Teacher Questionnaire (Comparison Schools)

Teacher ID :

The following questions are about your experience and thoughts. There are no right or 

wrong answers. Please read all questions thoroughly and provide a response to the best of 

your ability. Directions are provided at each question. Please mark the box next to your 

answer choice, like this: √ For some questions, you will be asked to write your responses 

using words or numbers.

1. Are you male or female? Male Female

2. What year were you born? |_1_|_9_|___|___|

3. How many years have you taught English? |___|___| years

4. The following questions are about your English classes this semester. 

a. How many classroom hours of English classes do you teach on a weekly basis?

 |___|___|

f. How many English classes do you teach? |___| classes

g. What are the grade levels? 7

th grade 8th grade 9th grade

5. Do you have a degree in English? No Yes

6. The following questions are about the highest degree you have completed. Using the list on the 

bottom, please fill in the corresponding number.

a. What is the highest level of degree you have completed? |___|

b. What is the highest level of degree in English you have completed? |___| (Write

 0 if you do not have a degree in English)

List of Highest Degree Completed

(1) Less than High School

(2) High School Diploma

(3) 3-Year Normal High School Diploma

(4) Associate Degree, including 5-Year Normal College

(5) 4-Year University

(6) Advanced Degree

7. During this semester, have you participated in teachers’ training related to English teaching 

that was organized by your school? (Mark one answer)

No

Yes  How many days in total? |___|___| days
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-34 A Study of a Gaming Approach to English Language Learning for Middle School Students

8. During this semester, have you participated in teachers’ training related to English teaching 

that was organized by an entity other than your school? (Mark one answer)

No

Yes  How many days in total? |___|___| days

9. Currently, how experienced are you with using instructional technology to improve classroom 

instruction for English? (Mark one answer)

Not experienced at all

Less experienced

Somewhat experienced

Very experienced

10. Have you used computer software for teaching 8th grade English classes? (Mark one answer)

No

Yes  What kind of software? Please specify: __________________________

11. What are some teaching methods that you frequently use for teaching 8th grade English 

classes? (Mark all that apply)

Give lectures according to textbook chapters

Incorporate supporting materials for classroom instruction 

Design English language games and activities

Focus on test-taking skills

Something else  What are they? Please specify: 

_____________________________

12. In general, what percentage of time do you use Chinese (including lecture, oral questioning, 

dialogues, etc.) to teach 8th grade English classes? 

Less than 20%

20% - 50%

50% - 80%

More than 80% 

13. Do you have a computer at home?

Yes

No

14. Do you have access to the internet at home?

Yes

No
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-35

15. Being as objective as you can, how motivated are your 8th grade students in terms of learning 

English?

Not motivated at all

Less motivated

Somewhat motivated

Very motivated

16. Currently, which of the following apply to you? (Mark all that apply)

I have traveled to English-speaking countries

I use English to speak with relatives, family, or friends

I read English language books and magazines, newspapers, and Web sites

17. How often do you watch English language TV programs for at least 15 minutes in a sitting? 

(Mark one answer)

Never

Once every few months

Once or twice a month

Once or twice a week

Three or four times a week

Everyday

18. How often do you use English to have online chat with people or write e-mails?

Never

Once every few months

Once or twice a month

Once or twice a week

Three or four times a week

Everyday

19. How important do you feel that learning English is to middle school students in this area? 

(Mark one answer)

Not important at all

Less important

Somewhat important

Very important

Thank you for completing this questionnaire!

We will keep your responses confidential.
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-36 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 9a. 教师问卷 (对照学校)

教师代码及姓名 :

本问卷主要是为了了解您在英语教学方面的经历和想法。任何回答都没有对错之分。希

望你能够认真阅读题目,并选择最适合的回答。请根据题目指示,在方框内打勾,如 

。有些题目会请您以文字或数字写出您的回答。 

1. 您的性别是: 男 ; 女

2. 您的出生年份是: |_1_|_9_|___|___|

3. 您从事英语教学有多少年了? |___|___| 年

4. 请回答下列关于您在本学期担任英语教学方面的问题。

 a. 您每周担任英语教学的课时数是:|___|___|

h. 您担任英语教学的班级数是: |___| 个班级

i. 有哪些年级? 7 年級 ; 8 年級 ; 9 年級

5. 您是不是英语专业毕业? 不是 ; 是

6. 下列问题有关于您的最高学历。请用题目下方的最高学历表,填入相关号码。

a. 您所取得的最高学历是: |___|

b. 您在英语专业方面取得的最高学历是: |___| (若不是英语专业毕业,填 0 )

最高学历表

(1) 高中以下

(2) 高中学历

(3) 3年制中师或中专学历

(4) 大学专科学历(含5年中师)

(5) 大学本科毕业

(6) 大学研究生毕业
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-37

7. 本学年您是否还参与过由本校组织,英语教学方面的教师培训活动?(单项选择)

没参与

有参与  总合计时间大约为:|___|___| 天

8. 本学年您是否还参与过由其它方面组织 (不包括本校),英语教学方面的教师培训

 活动?(单项选择)

没参与

有参与  总合计时间大约为:|___|___| 天

9. 目前,您在使用教育技术开展英语课堂教学方面的能力如何?(单项选择)

完全不会

比较弱

比较强

很强

10. 请问,您有沒有在8年級的英语教学中使用过计算机软件?(单项选择)

没用过

有用过  是什么样的计算机软件? 请写出: __________________________

11. 您在8年級的英语教学中常用的教学方法有哪些?(多项选择)

按照教材内容教师讲解为主

找一些辅助材料用于课堂教学

设计一些英语游戏活动

以围绕应付考试为主而教学

其他  是什么样的方式? 请写出:__________________________________

12. 一般來説,您在8年級的英语课堂中运用中文(包括讲解、提问、对话等)的时间

 比例是多少?(单项选择)

20%以内

20% 到50%

50% 到80%

80% 以上
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-38 A Study of a Gaming Approach to English Language Learning for Middle School Students

13. 你在家里有电脑吗?(单项选择) 

有

没有

14. 你在家里能上网吗?(单项选择) 

能

不能

15. 请客观地评价目前您所任教的8年級学生学习英语的积极性情况?(单项选择)

非常低

比较低

比较高

非常高

16. 目前,下列哪种情况适合于您? (多项选择)

我曾经到英语国家旅游过

我平时会用英语与亲戚、家人或者朋友对话

我平时会阅读英文书刊、报纸或者上英文网站阅读

17. 您多久收看有英语对话的电视节目,且每次至少看15分钟?(单项选择)

从来不看

几个月看 1次

每月看1到2次

每周看1到2次

每周看3到4次

每天看
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-39

18. 您多久会在网络上用英文与人聊天或写邮件? (单项选择)

从来不会

几个月 1次

每月1到2次

每周1到2次

每周3到4次

每天

19. 您认为,本地初中学生学习英语的重要性程度如何?(单项选择)

非常不重要

比较不重要

比较重要

非常重要

非常感谢您认真回答了本问卷!

我们将保密您的回答。
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-40 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 10. Student Questionnaire

The following questions are about your experience and thoughts. There are no right or wrong 

answers. Please read all questions thoroughly and provide a response to the best of your ability. 

Directions are provided at each question. Please mark the box next to your answer choice, like 

this:  For some questions, you will be asked to write your response.

1. Are you a boy or a girl? Boy Girl

2. Currently, how important is learning English to you? (Mark one answer)

Not important at all

Not too important

Somewhat important

Very important

3. Currently, how interested are you in learning English? (Mark one answer)

Not interested

Less interested

More interested

Very interested

4. Currently, which of the following apply to you? (Mark all that apply) 

I have traveled to English-speaking countries

I would use English to speak with relatives, family, or friends

I would read English language books and magazines, newspapers, and Web sites

5. How often do you watch English language TV programs for at least 15 minutes in a sitting? 

(Mark one answer)

Never

Once every few months

Once or twice a month

Once or twice a week

Three or four times a week

Everyday

6. How often do you use English to have online chat with people or write e-mails? (Mark one 

answer)

Never

Once every few months

Once or twice a month

Once or twice a week

Three or four times a week
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-41

Everyday

7. How often do you play computer games that have vocabulary in English? (Mark one answer)

Do not include the time spent on The Forgotten World.

Never

Once every few months

Once or twice a month

Once or twice a week

Three or four times a week

Everyday

8. Do you go to an English tutoring class outside of school? (Mark one answer)

Never

I have, but not anymore

I do, but have not before

I always have 

9. Compared to when you were in 7th grade, has your motivation to learn English in 8th grade: 

(Mark one answer)

Lowered a lot

Lowered a little

Stayed about the same

Improved a little

Improved a lot 

10. How much fun is learning English in school? (Mark one answer)

No fun 

A little fun 

Some fun 

A lot of fun
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-42 A Study of a Gaming Approach to English Language Learning for Middle School Students

11. Students in general may experience difficulty when they are learning English. Currently, how 

difficult are the following areas to you? (Mark one answer per row)

Content

Level of Difficulty

Not Difficult At 

All A Little Difficult Somewhat Difficult Very Difficult

a. Vocabulary

b. Sentence

c. Listening

d. Reading

e. Writing

f. Speaking

12. Realistically, do you expect to attend university some day? (Mark one answer)

No

Yes

Unsure

Instruction: Students who have used The Forgotten World, please continue answering 

question 13 -17 below. Students who have never used The Forgotten World do NOT need to 

answer the questions below. 

13. How often did you use The Forgotten World at school? (Mark one answer)

Never

Once every few months

Once or twice a month

Once or twice a week

Three to four times a week

Everyday

14. How often did you use The Forgotten World outside of school? (Mark one answer)

Never

Once every few months

Once or twice a month

Once or twice a week

Three to four times a week

Everyday
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-43

15. In general, how helpful do you think The Forgotten World is to your English learning? (Mark 

one answer)

Don‘t know (Didn‘t use much or at all)

Not helpful at all

Not so helpful

Somewhat helpful

Very helpful

16. What were the two biggest challenges that you encountered while using The Forgotten World?

(Mark two answers)

Insufficient time for using the software

Limited support from the school

Problems with computers at the school computer laboratory

Concerns with negative impact on test scores

Other challenges -> Please specify: _____________________________

17. What is your username for the Forgotten World? .

Thank you for completing this questionnaire!

We will keep your responses confidential.
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-44 A Study of a Gaming Approach to English Language Learning for Middle School Students

Attachment 10a. Student Questionnaire

本问卷主要是为了了解你在学习英文方面的经历和想法。任何回答都没有对错之分。希

望你能够认真阅读题目,并选择最适合的回答。请根据题目指示,在方框内打勾,如 

。有些题目会指示你写出你的回答。

1. 你的性别是: 男孩; 女孩

 

2. 目前,你认为学习英语重要吗?(单项选择) 

完全不重要

不太重要

有些重要

非常重要

3. 目前,你学习英语的兴趣程度如何?(单项选择) 

没有

较小

较大

很大

4. 目前,下列哪种情况适合于你?(多项选择) 

我曾经到英语国家旅游过

我平时会用英语与亲戚、家人或者朋友对话

我平时会阅读英文书刊、报纸或者上英文网站阅读
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-45

5. 你多久收看有英语对话的电视节目,且每次至少看15分钟?(单项选择) 

从来不看

几个月看 1次

每月看1到2次

每周看1到2次

每周看3到4次

每天看

6. 你多久会在网络上用英文与人聊天或写邮件?(单项选择) 

从来不会

几个月1次

每月1到2次

每周1到2次

每周3到4次

每天

7. 你多久在计算机上玩有英语单词的电子游戏?(单项选择) 

 请不要包括使用The Forgotten World 的时间。 

从来不玩

几个月玩 1次

每月玩1到2次

每周玩1到2次

每周玩3到4次

每天玩

8. 你有没有参加校外英语辅导班?(单项选择) 

从来没有过

以前有, 现在没有

以前没有,现在有

一直参加校外班

9. 和你在7年级的时候相比,你在8年级时学习英语的积极性是:(单项选择)

下降较大

下降较小
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-46 A Study of a Gaming Approach to English Language Learning for Middle School Students

和以前一样

提高较小

提高较大

 

10. 你在学校学习英语时有多大的乐趣?(单项选择) 

没有乐趣

较小乐趣

较大乐趣

很大乐趣
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-47

11. 一般学生在英语学习中有时会遇到困难。目前,下列方面对你来说有多大困难? 

 (请在合适的方框内打勾,每行只能选一个) 

内容 困难程度

没有困难 困难较小 困难较大 困难很大

a. 单词

b. 语法

c. 听力

d. 阅读

e. 写作

f. 口语

 

12. 从实际出发,你期望将来能够进大学吗?(单项选择) 

不期望

有期望

不知道

填表说明:请使用过The Forgotten World 

学习软件的同学,继续回答下面的13题至17题。没有使用过的同学,不需要再回答了。 

 
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

G-48 A Study of a Gaming Approach to English Language Learning for Middle School Students

13.你在学校中多久使用The Forgotten World 学习软件?(单项选择) 

从不使用

几个月使用1次

每月使用1到2次

每周使用1到2次

每周使用3到4次

每天使用

14. 你在学校之外又是多久使用The Forgotten World 学习软件?(单项选择)

从不使用

几个月使用1次

每月使用1到2次

每周使用1到2次

每周使用3到4次

每天使用

15. 总的来说,你认为The Forgotten World 这个学习软件对于你学习英语的帮助 

 多大? (单项选择) 

不知道(很少使用或从未使用)

没有帮助

帮助较小

帮助较大

帮助很大

16. 你在使用The Forgotten World学习英语时,遇到了最大的2项困难是什么? 

 (可选2 项) 

没有可以更多的使用时间

学校对这项工作的重视不够

学校计算机教室设备有问题

担心影响考试成绩

其他困难  请写出:_______________________________________________

17. 你目前所使用的The Forgotten World的用户名是什么?________________ 
Appendix G – Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires

A Study of a Gaming Approach to English Language Learning for Middle School Students G-49

 

感谢你认真完成了本问卷。我们将保密你的回答! 

H-1

Appendix H.

Supplementary Tables

Appendix H – Supplementary Tables

Supplementary Tables H-3

Table H-1. Hierarchical Linear Models to Confirm Significant Findings for Test 

Score Gains Among Low-performing Students and Improved Motivation

Hierarchical models of test score change and probability of motivation or fun=1

Reading score change Listening score change

Model 1 Model 2 Model 1 Model 2

Spring 

motivation Spring fun

Intercept

6.61

(1.05)

0.000

6.64

(1.24)

0.000

5.01

(0.41)

0.000

4.59

(0.63)

0.000

-0.43

(0.17)

0.031

-0.24

(0.17)

0.189

Treatment

1.52

(1.22)

0.213

0.47

(1.47)

0.752

1.12

(1.31)

0.391

-0.83

(1.37)

0.547

0.50

(0.18)

0.007

0.29

(0.19)

0.122

Below median on 

fall test

-0.04

(0.60)

0.948

0.82

(1.04)

0.429

Treatmentt* 

Below 

median

2.24

(1.03)

0.029

4.14

(1.32)

0.002

Note: Coefficients reported in top line. Standard errors in parentheses. P-values (significance) in third line.

Significant findings (p<0.05) are bolded and italicized. N=3586 students, 62 classrooms, and 10 schools

Results are based on hierarchical linear models of change in reading or listening test score, 

predicted by enrollment in a treatment school and, where noted, membership in the lower half of 

the fall test score distribution. These are random-intercept models: the average test score change 

varies by class and school, but the effect of treatment assignment is fixed across all cases. For 

the index of motivation or fun, results are based on hierarchical nonlinear (logistic) models.
Appendix H – Supplementary Tables

H-4 Supplementary Tables

Table H-2. Mean and Design-corrected Standard Errors for Motivation, for 

Treatment and Comparison Groups

 

 

 Data Summary 

 

 Number of Clusters 10 (Schools) 

 Number of Observations 3640 

 

 Statistics 

 

 Std Error 

Variable N Mean of Mean 95% CL for Mean 

ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ 

SPRING_Q9_MOTIVATION 3567 0.453602 0.028563 0.38898782 0.51821711 

ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ 

 

 Domain Analysis: EXP 

 

 Std Error 

 EXP Variable N Mean of Mean 95% CL for Mean

ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ

TREATMENT SPRING_Q9_MOTIVATION 1787 0.509233 0.017536 0.46956446 0.54890224

COMPARISON SPRING_Q9_MOTIVATION 1780 0.397753 0.040592 0.30592670 0.48957892

ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ

 

 
H-5