1

PROGRAM FOR INTERNATIONAL STUDENT ASSESSMENT 

2012 (PISA:2012) VALIDATION STUDY 2015 FIELD TEST AND 

MAIN STUDY

REQUEST FOR OMB CLEARANCE

OMB# 1850-0900 v.2

SUPPORTING STATEMENT PART A

Submitted by:

National Center for Education Statistics

U.S. Department of Education

Institute of Education Sciences

Washington, DC

December 8, 2014
2

TABLE OF CONTENTS

PREFACE................................................................................................................ 3

A. JUSTIFICATION ............................................................................................... 3

A.1 Importance of Information ............................................................................................... 3

A.2 Purposes and Uses of Data ............................................................................................... 9

A.3 Use of Information Technology ....................................................................................... 9

A.4 Efforts to Identify Duplication.......................................................................................10

A.5 Minimizing Burden for Small Entities...........................................................................10

A.6 Frequency of Data Collection .........................................................................................10

A.7 Special Circumstances of Data Collection.....................................................................10

A.8 Consultations Outside NCES .........................................................................................10

A.9 Provision of Payments or Gifts to Respondents..........................................................10

A.10 Assurance of Confidentiality ...........................................................................................11

A.11 Sensitive Questions...........................................................................................................12

A.12 Estimates of Response Burden.......................................................................................12

A.13 Estimates of Cost to Respondents.................................................................................12

A.14 Cost to Federal Government ..........................................................................................12

A.15 Program Changes or Adjustments..................................................................................13

A.16 Publication Plans and Timetable.....................................................................................13

A.17 Display OMB Expiration Date .......................................................................................13

A.18 Exceptions to Certification Statement...........................................................................13

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL 

METHODS

B.1 Potential Respondent Universe and Sampling

B.2 Information Collection Procedures

B.3 Methods for Maximizing Response Rates

B.4 Tests of Procedures and Methods

B.5 Individuals Consulted on Study Design

Appendix A: Respondent Recruitment Materials

Appendix B: Draft Education and Skills Online Questionnaire Items
3

PREFACE

The Program for International Student Assessments (PISA) is an international assessment of 15-year-old

students sponsored by the Organization for Economic Cooperation and Development (OECD). In the 

United States, it is conducted by the National Center for Education Statistics (NCES), of the Institute of 

Education Sciences, U.S. Department of Education. PISA was first administered in 2000 and is conducted 

every three years. As part of PISA’s most recent administration, in 2012, PISA’s fifth cycle, a nationallyrepresentative sample of approximately 5,200 15-year-old students took the PISA assessments in the United 

States (OMB# 1850-0755).

PISA provides information about student performance in mathematics, science, and reading literacy, as well 

as problem-solving in the United States and more than 65 other participating countries and education 

systems. PISA is designed to assess how well students nearing the end of compulsory schooling are prepared 

for the challenges of further education and the workforce. To do so, PISA measures students’ literacy

(knowledge and skills to solve problems in real-life contexts) and collects information about students’ home 

background, attitudes, and experiences in and out of school through a questionnaire completed by students. 

Furthermore, principals of school attended by the participating students complete a questionnaire about

school policies and practices, providing additional context for the student measures.

NCES proposes to conduct a follow-up study with students who participated in PISA 20121to learn how 

performance on PISA relates to subsequent outcomes and skills of young adults. The follow-up study—

referred to in materials to potential respondents as the PISA Young Adult Follow-Up Study, and in this 

request in short as the PISA Validation Study — will provide information about how students’ skills and 

experiences at age 15, collected through PISA, relate to subsequent literacy, numeracy, and problem-solving 

skills, as well as educational attainment, education and work experiences, skills used in daily life, career 

intentions, and aspects of health and well-being. In fall 2015, when these students will be 18 years of age, they 

will be asked to take the web-based version of the OECD’s Program for the International Assessment of 

Adult Competencies (PIAAC) assessment and background questionnaire—the Education and Skills Online 

Assessment (ESO).

In November 2013, OMB approved the location and tracking of the PISA 2012 respondents (OMB 1850-

0900 v.1) to prepare for this follow-up study. This request is to (1) recruit the PISA 2012 sample respondents 

who have been successfully located and tracked; (2) administer ESO to a field test sample in the summer of 

2015; and (3) administer ESO to a main study sample in the fall of 2015. This submission describes the study 

rationale and design and presents the sample members recruitment materials (Appendix A) and the ESO 

background questionnaire (Appendix B).

A. JUSTIFICATION

A.1 Importance of Information

As part of a continuing cycle of international education studies, the United States, through the National 

Center for Education Statistics (NCES), is participating in several international assessments and surveys. The 

Program for International Student Assessment (PISA), sponsored by the Organization for Economic 

Cooperation and Development (OECD), is one of these studies (OMB# 1850-0755).

In light of growing concerns related to international economic competitiveness, the changing face of our 

workplace, and the expanding international marketplace in which we trade, knowing how our students and 

 

1 The sample of ~5,200 students took the PISA mathematics, science, and reading literacy assessment. PISA 2012 also included an assessment of financial literacy 

given to an additional ~1,100 students. In order to focus the Validation Study on the main PISA domains, the proposed study does not include the students who 

took financial literacy except in the field trial.
4

adults compare with their peers around the world has become an even more prominent issue than ever 

before. Nationwide, interest in understanding what other nations are doing to further the education 

achievement of their populations has increased beyond simple comparisons. Data at critical points during the 

education career of our students help inform policymakers in their efforts to guide and restructure the 

American education system. These critical points may occur during primary, secondary, or tertiary education, 

as well as extend into adult education and training programs. Consequently, generating comparative data 

about students in school, at the end of formal schooling, in postsecondary education, and about adults in the 

workplace and in the community has become an important focus for NCES.

PISA is part of the larger international program that NCES has actively participated in through collaboration 

with, and representation at, the OECD, the Asia-Pacific Economic Cooperation (APEC), and the 

International Association for the Evaluation of Educational Achievement (IEA). Collaboration with Statistics 

Canada, Eurostat, and ministries of education throughout the world helps to round out the portfolio of data 

NCES compiles. The United States has participated in PISA since its first administration in 2000.

PISA measures students' knowledge, skills, and competencies primarily in three subject areas –mathematics, 

science, and reading literacy. The overall strategy is to collect in-depth information on students’ capabilities in 

one of these three domains every three years so that detailed information on each becomes available every 

nine years. During each three-year survey cycle, the major focus is on one content domain, with a minor focus 

on the other two content domains. The major focus for the PISA data collection in 2012 was on mathematics 

literacy, with a minor focus on science and reading. PISA 2012 also included an assessment of problemsolving and financial literacy.

Investigating the link between PISA and young adults’ preparedness for life and work

PISA is designed to measure “how well students, at age 15, are prepared to meet the challenges they may 

encounter in future life” (OECD 2009, pg. 9). PISA does this by identifying, with input from participating 

countries and subject-matter experts, the key competencies that are believed to contribute to students’ future 

success, and by measuring students’ knowledge and skills in these areas. In short, PISA is designed to assess 

how well students nearing the end of compulsory schooling are prepared for further education or entry into 

the workforce and, more fundamentally, to contribute to society as functioning young adults.

Unlike other large-scale international assessments that are based on school curricula, such as the Trends in 

International Mathematics and Science Study (TIMSS) or the Progress in International Reading Literacy Study 

(PIRLS), PISA assessments are designed to reflect what students have learned both in and out of school and 

to measure students’ ability to apply that knowledge to real-world situations. For this reason, PISA has a 

unique validity challenge. TIMSS and PIRLS assessments can be validated by evaluating how well they cover 

mathematics, science, and reading curricula in the participating countries. However, comparing PISA 

measures with content standards or school curricula does not provide evidence that PISA assesses the 

competencies that matter for students’ “future success” and the “challenges they may encounter in the 

future.” To validate PISA measures, it is necessary to examine whether the competencies assessed by PISA 

predict future success.

Given the growing prominence of PISA in the United States, it is of policy interest to better understand how 

well PISA measures students’ preparedness for further education and the workforce. PISA is widely cited as 

evidence that U.S. students are not as prepared for the global economy as their peers in other highperforming countries, and such claims rest on the assumption that PISA is measuring the skills that students 

will need to succeed in the 21st century and the global economy.
5

The PISA Validation Study

NCES is requesting approval to conduct a follow-up study with students who participated in PISA 2012 to 

learn how performance on PISA relates to subsequent outcomes and skills of young adults. The follow-up 

study—referred to in materials to potential respondents as the PISA Young Adult Follow-Up Study (YAFS), 

and in this clearance request in short as the PISA Validation Study—will provide information about how 

students’ skills and experiences at age 15, collected through PISA, relate to subsequent literacy, numeracy, and 

problem-solving skills, as well as educational attainment, education and work experiences, and skills used in 

daily life. This will be accomplished by administering a web-based version of the OECD’s Program for the 

International Assessment of Adult Competencies (PIAAC). PIAAC will be used to measure the skills, 

outcomes, and experiences of students who previously participated in PISA. The PIAAC assessment, 

administered in 25 countries, focuses on cognitive and workplace skills deemed necessary for successful 

participation in 21st-century society and the global economy (OECD 2012). Specifically, PIAAC measures 

relationships between individuals' educational backgrounds, workplace experiences and skills, occupational 

attainment, use of information and communications technology, and cognitive skills in the areas of literacy, 

numeracy, and problem solving.

The sample for the PISA Validation Study will consist of the approximately 4,700 15-year-old students from 

the PISA 2012 national sample who took the PISA mathematics, reading, and science test and completed a 

Student Information Form with their contact information. PISA 2012 recruitment materials, including 

materials for parents, stated that students supplying contact information may be contacted by NCES for a 

future study. In fall 2015, when these students will be approximately 18 years of age, they will be asked to take 

a web-based version of the PIAAC assessment and background questionnaire developed by the OECD—the 

Education and Skills Online Assessment (ESO). The ESO is described in more detail below. A field trial of 

the Validation Study is currently planned for summer 2015 with a sample of 200 PISA 2012 students who 

provided contact information but took financial literacy test booklets and are not the target for the PISA 

Validation Study main data collection.

PIAAC was administered to a national sample of adults 16 to 65 years old in 2011-12. It examined adults’ 

literacy and skills in the information age, focusing on what are deemed key skills for individuals to participate 

successfully in the economy and society of the 21st century (Goodman, Finnegan, Mohadjer, Krenzke, and 

Hogan 2013). It assessed adults’ basic reading skills, reading literacy, numeracy, and problem solving in 

technology-rich environments (on or with a computer), and measured their ability to use computer and web 

applications to find, gather, and use information, and to communicate with others. PIAAC used a “Job 

Requirements Approach” to ask employed adults about the types and levels of a number of specific skills 

used in the workplace, including not only the use of reading and numeracy skills on the job, but also the use 

of physical skills (e.g., stamina and manual dexterity), people skills (e.g., public speaking, negotiating, working 

in a team), and information technology skills (e.g., using spreadsheets, writing computer code). PIAAC also 

asked about adults’ backgrounds, work and educational experiences, including the requirements of their main 

job in terms of the intensity and frequency of the use of the aforementioned skills.

The web-based version of the PIAAC assessment and background questionnaire (ESO) will be distributed by 

the OECD for use by countries, institutions, and individuals to support research and workforce development 

and training efforts. The ESO consists of two main components that are administered together: an 

assessment and a questionnaire. The ESO assessment, which is currently being finalized for distribution to 

countries by the OECD, was piloted in multiple countries and has been equated to PIAAC, allowing for 

international benchmarking. The ESO questionnaire is based on that used in PIAAC.
6

Administering the ESO to PISA 2012 participants

In the PISA Validation Study, the ESO assessment and background questionnaire will be administered to 

sample respondents in 2015 when they are approximately 18 years old, three years after they took PISA in 

2012. The assessment is designed to take on average 90 minutes to complete and the background 

questionnaire approximately 30 minutes, for a total of 120 minutes on average per respondent.

The ESO assessment measures a set of cognitive and non-cognitive skills deemed necessary for full 

participation in modern societies. These skills and knowledge include being able to understand and use 

printed and electronic texts, reason with numbers, and solve problems using information and communication 

technologies (see exhibit A-1, below, for an overview of the cognitive content of the ESO).

Exhibit A-1. Summary of the cognitive domains in the OECD Education and Skills Online (ESO) 

Assessment

Literacy Numeracy Problem solving in technologyrich environments 

Definition

Ability to understand, evaluate, use 

and engage with written texts to 

participate in society, achieve one’s 

goals, and develop one’s knowledge 

and potential. 

Ability to access, use, interpret and 

communicate mathematical 

information and ideas in order to 

engage in and manage the 

mathematical demands of a range of 

situations in adult life. 

Ability to use digital technology, 

communication tools and networks 

to acquire and evaluate information, 

communicate with others, and 

perform practical tasks. 

Content

Texts are characterized by

Medium:

 Print-based

 Digital

Format:

 Continuous or prose texts 

(narration, argumentation, or 

descriptions)

 Non-continuous or document 

texts (tables, lists, graphs)

 Mixed texts (combination of prose 

and document elements)

 Multiple texts (juxtaposition or 

linking of independently generated 

elements) 

Mathematical content, information 

and ideas:

 Quantity and number

 Dimension and shape

 Pattern, relationships, change

 Data and chance

Representations of mathematical 

content:

 Objects and pictures

 Numbers and symbols

 Diagrams, maps, graphs, tables

 Texts

 Technology-based displays 

Technology:

 Hardware devices

 Software applications

 Commands and functions

 Representations (text, graphics, 

video)

Nature of problems:

 Intrinsic complexity (number of 

steps, alternatives required for 

solution, complexity of 

computation and/or 

transformation, number of 

constrains)

 Explicitness of the problem 

statement (largely unspecified or 

described in detail 

Cognitive 

Strategies

 Access and identify

 Integrate and interpret (relating 

parts of text to one another)

 Evaluate and reflect on 

 Identify, locate or access

 Act upon and use (order, count, 

estimate, compute, measure, 

model)

 Interpret, evaluate and analyze

 Communicate 

 Setting goals and monitoring 

progress

 Planning

 Acquiring and evaluating 

information

 Using information 

Context

 Personal

 Work-related

 Community

 Education 

 Everyday life

 Work-related

 Society & Community

 Education 

 Personal

 Work-related

 Community 

Source: http://www.oecd.org/site/piaac/ENGLISH_Brochure%20Education%20and%20Skills%20Online.pdf.

The ESO questionnaire, in turn, asks about personal and home background and education and workforce 

experiences (all taken from the PIAAC questionnaire), and includes modules that were developed and piloted 

by the OECD for ESO. The PIAAC background questionnaires were more extensive than what will be 
7

included in the ESO, however. As shown in exhibit A-2, below, the content of the ESO questionnaire 

modules focus on three areas: (i) basic demographics, (ii) career interests and intentionality (CII) and (iii) 

behavioral performance competencies (BPC). It is possible to use one or more of the modules developed by 

the OECD, but it is not possible to add or remove items or edit items within the modules. That is, the ESO

modules must be administered intact.

Because the ESO must be administered as designed, NCES is planning to administer an ‘add-on’ 

questionnaire module, with 16 additional items administered in PIAAC but not included by the OECD in 

ESO. These additional questions address students’ recent and current educational status and experiences, 

including non-formal learning activities. This information is critical for realizing the aims of the PISA 

Validation Study, which is to examine the relationship between PISA and the paths that young adults choose 

after leaving high school. The themes covered by these additional survey items are also shown in exhibit A-2.

More detailed information on all of the non-cognitive survey items is included in Appendix B. Please note 

that because the ESO modules are designed to be administered online, detailed information on the survey 

items provided in Appendix B are shown in tables, as there is no paper version of the questionnaire.

Exhibit A-2. Summary of the non-cognitive modules in the OECD’s Education and Skills Online 

(ESO) Assessment

Non-cognitive survey 

module Content themes

Background + Educational 

Experiences (Core)

 Age

 Sex

 Highest level of education attained

 Country of origin

 Language spoken at home

 Native language

 Current employment status

 Current occupation

Career Interest and Intent

(CII)

 Inventory of vocational interests

 Intent to seek employment

 Intent to seek job training

 Job seeking behavior

Behavioral performance 

competencies (BPC)

 Well being

 Self-control

 Responsibility

 Sociability

 Curiosity

 Cooperation

 Generosity

 Adjustment

 Intellectual efficiency

 Ingenuity

 Achievement/Goal setting

 Dominance

 Organization and Order

Additional items not already 

included in ESO

 Current educational trajectory

 Degree/certification being sought

 General category of degree/certification 

being sought

 Non-formal learning activities (distance 

education; seminars)

 Additional learning activities

 Reasons for non-participation in learning 

activities

 Use of tutors and participation in 

apprenticeship programs 

Rationale for including CII and BPC modules

Of the original four modules being developed by the OECD for the ESO, NCES proposes to administer two 

(in addition to the demographic module). Both modules are derived from PIAAC modules of the same name 

which were field trialed and administered in the main PIAAC 2011-12 data collection. The Career Interests 

and Intentionality (CII) module will provide information about the employment status and vocational 

interests of students in the PISA Young Adult Follow-up Study, at approximately age 18, which can be related 

to their performance on PISA at age 15. Questions about employment status will provide detailed 

information about what students are doing for work and/or if they are seeking employment, which will be 
8

especially useful for understanding the current experiences of students who did not go on to higher 

education. For all students in the sample the module will provide information about occupational interests 

and intentions that can be related to performance on PISA.

The Behavioral Performance Competencies (BPC) module addresses a range of non-cognitive constructs, 

including well-being, self-control, cooperation, and responsibility. This module will provide a way to relate 

non-cognitive outcomes of students in early adulthood to earlier performance in reading, mathematics, and 

science, to address the question of how performance on PISA relates to successful participation in adult life.

As noted, both the CII and BPC modules have already been used in the field for PIAAC 2011-12 data 

collection so their extension to ESO does not require further testing.

Rationale for additional non-cognitive survey items

As indicated above, NCES proposes to augment the ESO questionnaire modules with 16 additional items as 

an ‘add-on’ module that, from a respondent perspective, will seamlessly blend with the ESO. The rationale 

for proposing these additional items is provided below. These additional items are also taken from the 

PIAAC 2011-12 questionnaires but were not included in the ESO modules distributed by the OECD. NCES 

believes these additional items will bring added analytic insight to the results of the PISA Validation Study.

It is estimated that over 80 percent of the U.S. 15-17 years-old cohort go on to some form of postsecondary 

education (ELS: 2002-2012)2. Moreover, some proportion of these young adults stays back in high school. 

Therefore, four questions on Current education study status (participation; level of degree; area of study) 

are proposed to be added. Given that some respondents may have started and stopped their education, or 

dropped out, within the last year, one question on Studies in the last 12 months is proposed to be added. 

Additionally, a portion of the respondents may be participating in some form of GED or basic skills courses, 

leading to three additional questions on Basic education courses in the last 12 months

(reading/writing/math, GED, some other adult education) proposed to be added. A small proportion of

young adults reports participating in apprenticeship programs. Therefore one question on the Formal 

apprenticeship in the last 12 months is proposed to be added. NCES is proposing to add these nine 

questions to allow for a more comprehensive tracking of the PISA-PIAAC Follow-up sample into various 

forms of formal education.

In addition to the nine questions above, another seven questions are being proposed to address the nonformal educational choices of young adults. While a large proportion of young adults participate in formal 

education, PIAAC data shows that 55 percent of U.S. 16-19 year olds also participated in non-formal 

education activities (PIAAC:2012)3. To ascertain whether respondents are participating in non-formal 

education, five questions on Non-formal learning activities (distance education, on-the-job courses, 

seminars, other private lessons) are proposed to be included. Finally, with the intent of tracking barriers to 

non-formal learning activities the respondents may experience, two questions on More/any learning 

activities, wanted, but could not participate in and reasons for non-participation are proposed to be 

included.

 

2 Lauff, E., and Ingels, S.J. (2013). Education Longitudinal Study of 2002 (ELS:2002): A First Look at 2002 High School Sophomores 10 Years Later (NCES 

2014-363). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved [11/11/2014] at 

http://nces.ed.gov/pubs2014/2014363.pdf

3 Organization for Economic Cooperation and Development (OECD), Program for the International Assessment of Adult Competencies (PIAAC), 2012. 

Retrieved from PIAAC IDE [11/11/2014].
9

As indicated earlier, respondents will be requested to complete both the cognitive and non-cognitive

instruments online. Appendix B includes the ESO non-cognitive survey items in the modules, including the 

additional items, proposed to be administered for this study.

A.2 Purposes and Uses of Data

PISA is intended to be an assessment at or near the end of compulsory education of how well students are 

prepared for further education or entry into the workforce and to contribute to society as functioning young 

adults. The PISA subject matter assessments are not developed by canvassing the curricula offered in 

participating countries, as is done in other large-scale international assessments. The PISA assessment 

frameworks and items are developed by conferring with subject matter experts around the world and 

representatives of participating countries about the mathematics, reading, and science competencies required 

for 15-year-olds to successfully transition to further education, the workforce, and otherwise as functional 

contributing members of society. For 2012, the mathematics framework was also circulated internationally 

among representatives of employers and postsecondary institutions for their comments on the 

appropriateness of aspects of the PISA mathematics framework.

Because of its emphasis on assessing what is thought to be important for later success rather than assessing 

what has been taught, PISA must turn to sources of evidence other than participating countries’ curricula in 

order to check that it is assessing what it intends to assess. Currently, the main source for PISA for this sort 

of validity checking is in the form of experts drawn from across participating countries, as described earlier.

Also, two other countries have performed longitudinal studies based on PISA. A longitudinal study in Canada 

has followed 30,000 students who participated in PISA 2000, gathering information every two years on their 

educational experiences, transitions, and attainment and employment outcomes (OECD 2010). A similar 

study drawing on a smaller sample was conducted in Denmark (Schleicher 2007).

What is lacking is an empirical linkage between PISA and the sorts of skills and outcomes required for 

successful participation in adult life. The proposed study will provide data on how performance on PISA, at 

age 15, relates to literacy, numeracy, and problem-solving skills; educational attainment and education and 

workforces experiences; skills use in daily life; and other non-cognitive outcomes at approximately age 18. For 

example, what thresholds of performance on PISA are associated the ability to successfully navigate the 

literacy, numeracy, and problem-solving demands of daily life in the adult world? What thresholds are 

associated with high school graduation and successful transition to higher education or entry-level workforce? 

How does performance on PISA relate to measures of subjective well-being at age 18, such as life satisfaction 

and healthy habits?

By linking performance on PISA to subsequent outcomes in young adulthood, the study will provide the sort 

of validity evidence needed to understand how well PISA is measuring the skills needed for successful 

participation in adult life, as it purports to do. The study will also strengthen our understanding of U.S. 

performance on PISA and its implications for U.S. college and career readiness and for the skills of our future 

workforce. This information can, in turn, also be used to inform the further development of PISA and future 

surveys of adult skills.

A.3 Use of Information Technology

Study instruments will be administered through the Internet. Respondents will be invited to log on to a secure 

website to take the assessment and questionnaire. The use of web-based instruments reduces the burden 

required for data collection, scoring, and data processing.
10

A.4 Efforts to Identify Duplication

Comparable U.S. data are not available from other data sources; nor would a PISA follow-up study make any 

data redundant. The purpose of this study is to look at the relationship of performance on PISA to 

performance on similar domains and to other outcomes after a period of time, to validate PISA’s measures. 

No other study in the United States has collected follow-up data from or about PISA students so this would 

be a unique source of data to accomplish this objective. The current study could produce some information 

that is similar to a subset of information from the High School Longitudinal Study (HSLS), which follows a 

similar age group. However, HSLS does not provide data that allow for the validation of PISA measures or 

that are directly comparable with data from other countries.

A.5 Minimizing Burden for Small Entities

This study will not impose burden on small businesses or small entities.

A.6 Frequency of Data Collection

Currently the main data collection is planned for fall 2015. A field trial of procedures will take place in 

summer 2015. Although there are currently no plans for a second follow-up study with the PISA 2012

students, NCES may decide at a later date to re-contact the cohort sample to collect additional data at another 

point in time on education and workforce experiences and skills in order to evaluate PISA measures against 

longer term outcomes.

A.7 Special Circumstances of Data Collection

No special circumstances of data collection are anticipated.

A.8 Consultations outside NCES

NCES has sought input from a wide range of individuals with knowledge of PISA and PIAAC. Both PISA 

and PIAAC are sponsored by the OECD and are coordinated internationally by a consortium of contractors 

responsible for various aspects of the studies. Key persons from the OECD and key contracting 

organizations involved in the international design, development, and operation of PISA and PIAAC are as 

follows.

Organization for Economic Cooperation and Development

Andreas Schleicher, Indicators and Analysis Division

2, rue André Pascal, 75775 Paris Cedex16, FRANCE

William Thorn, PIAAC

2, rue André Pascal, 75775 Paris Cedex16, FRANCE

Educational Testing Service

Irwin Kirsch, Project Director, ETS Corporate Headquarters

660 Rosedale Road, Princeton, NJ 08541 USA

A.9 Provision of Payments or Gifts to Respondents

As part of the planned efforts to reach response rate goals, NCES proposed giving sampled respondents the 

same incentive used for the PIAAC Main Study: a $50 cash card to thank the respondent for their time and 

effort spent taking the assessment and answering the survey. The amount of burden is equivalent to that of 

the PIAAC respondents. NCES received OMB approval to offer a $50 incentive for the PIAAC Main Study 
11

after the results of an incentive experiment during the 2010 PIAAC Field Test indicated that $50 improved 

response rates compared with lower amounts (OMB# 1850-0870). The $50 incentive was approved for this 

study in the Initial Contact and Address Updates request (OMB# 1850-0900 v.1).

Due to the unclustered sample, it would be prohibitively expensive to conduct an in-person data collection. A

web approach has been designed to administer the ESO to PISA 2012 students. During recruitment, 

respondents will be supplied with an unloaded cash card that will be loaded upon their completion of the 

ESO assessment and questionnaire. The card can be used by the respondent anywhere a MasterCard is 

accepted, once remuneration has been added to it. Until the survey is completed, the card will act as a visual 

incentive to the respondent to complete the survey.

A.10 Assurance of Confidentiality

Procedures for handling confidential aspects of the study that will be used in the PISA Validation Study will 

mirror those used in past administrations of PISA. Expertise in data security and confidentiality was a 

significant criterion in the selection of the PISA contractor. The plan for maintaining confidentiality includes 

signed confidentiality agreements and notarized nondisclosure affidavits obtained from all personnel who will 

have access to individual identifiers; personnel training regarding the meaning of confidentiality, particularly 

as it relates to handling requests for information and providing assurance to respondents about the protection 

of their responses; controlled and protected access to computer files under the control of a single data base 

manager; built-in safeguards concerning status monitoring and receipt control systems; and a secured and 

operator-manned in-house computing facility.

The physical and/or electronic transfer of personally identifiable information (particularly names and 

addresses) will be limited to the extent necessary to perform project requirements. This limitation includes 

both internal transfers (e.g., transfer of information between agents of Westat) and external transfers (e.g., 

transfers between Westat and NCES, or between Westat and another government agency assisting in data 

collection). All data files constructed to conduct the study will be maintained in secure network areas at 

Westat. These files will be subject to Westat’s regularly scheduled backup process. Backups are stored in 

secure facilities on site as well as off site. These data are stored and maintained in secure network and 

database locations where access is limited to those Westat staff who have specifically authorized access. 

Access is only granted once a staff member is assigned to the project and has completed the NCES Affidavit 

of Non-disclosure. Identifiers are maintained in files required to conduct survey operations that are physically 

separate from other research data and that are accessible only to sworn agency and contractor personnel.

The PISA Validation study will follow all applicable federal laws pertaining to security and confidentiality of 

personally identifiable information (PII). The laws pertaining to the collection and use of PII are clearly 

communicated in correspondence with participants, per NCES requirements (see materials in Appendix A). 

Materials will carry a statement addressing confidentiality as follows:

NCES is authorized to conduct this study under the Education Sciences Reform Act 

of 2002 (ESRA 2002, 20 U.S.C., § 9543). Your participation is voluntary and the 

information you provide may be used only for statistical purposes and may not be 

disclosed or used, in identifiable form for any other purpose except as required by 

law (ESRA 2002, 20 U.S.C., § 9573). Individual responses will be combined with 

those from other participants to produce summary statistics and reports.

Westat will deliver data files, accompanying software, and documentation to NCES at the end of the main 

study. Neither respondents’ names nor addresses will be included on any data file. Furthermore, prior to the 

release of any data file, NCES will conduct a disclosure risk analysis, per NCES standard protocol.
12

A. 11 Sensitive Questions

The survey does not include questions usually considered to be of a highly sensitive nature, such as items 

concerning religion, substance abuse, or sexual activity.

A. 12 Estimates of Response Burden

Table A-1 shows the estimated respondent burden for field trial and main study recruitment and data 

collection. There are combined total burden hours of 2,086 hours for the field test and main study data 

phases. These include communications with respondents asking them to formally participate in the survey, 

completion of the non-cognitive questionnaire, and completion of the cognitive assessment. At an estimated 

$7.25 per hour (federal minimum wage), the estimated cost to respondents for the total burden time 

associated with the address updates (2,086 hours) is $15,124.

Table A-1. Estimated burden to respondents*

Activity Sample 

size

Estimated 

response 

rate

Number of 

Respondents

Number of 

Responses

Estimated average 

burden (minutes)

Total burden 

(hours)

Address update requests - main 

study sample (5) 4,729 1.00 4,729 4,729 5 min per request25 min total 1,970

Address update requests – field 

trial sample (3) 1,081 1.00 1,081 1,081 5 min per request15 min total 270

Total burden address updates 5,810 5,810 2,240

Field trial

Communications 235 0.85 200 200 10 33

Questionnaire 235 0.85 200 200 30 100

Assessment 235 0.85 200 200 90 300

Data collection

Communications 3,447 0.85 2,930 2,930 10 488

Questionnaire 3,447 0.85 2,930 2,930 30 1,465

Assessment 3,447 0.85 2930 2,930 90 4,395

Total burden - data collections 

(communications and questionnaire) 3,130 6,260 2,086

*Total burden in the current request is for the field trial and main study communications and data collection of the questionnaire. Gray shaded 

rows indicate the portion of burden that was previously approved and completed, and entries in gray font represent burden associated with 

assessments, which is not subject to PRA approval and therefore not included in the burden totals.

A.13 Estimates of Cost to Respondents

Other than the burden time associated with completing the address updates and study instruments, the study 

imposes no additional cost to respondents.

A.14 Cost to Federal Government

The cost to the Federal Government for conducting the validation study is estimated to be $1,945,602 over a 

four-year period, based on the cost for the Validation Study task in the PISA 2015 contract (ED-IES-13-C0006). This includes tracing and tracking sample respondents, data collection, use of the OECD’s ESO 

instruments, and analysis and reporting. These figures include all direct and indirect costs of the project.
13

A.15 Program Changes or Adjustments

The apparent decrease in respondent burden for this collection is due to the fact that the last OMB approval 

was for two address updates in 2013-14 with all PISA 2012 sample members who supplied their contact 

information in 2012, while this 2015 field test and main study data collection will be conducted with only a 

subset of the original sample.

A.16 Publication Plans and Timetable

NCES will release a general audience report and technical/operations report, along with restricted-use data 

files. Electronic versions of each report will be made available on the NCES website. The expected data 

collection dates and a tentative reporting schedule are provided below.

.

Date Activity

Fall 2012 Collect contact information from PISA 2012 students (completed)

Fall 2013 – September 2015 Establish initial contact and conduct semi-annual contact with respondents for address 

updates

Summer 2015 Conduct field trial of procedures

Fall 2015 Web-based main study data collection 

2016-2017 Reporting: general audience report; survey operations report; restricted-use data-files

A.17 Display OMB Expiration Date

The OMB expiration date will be displayed on all data collection materials.

A.18 Exceptions to Certification Statement

No exceptions are requested to the "Certification for Paperwork Reduction Act Submissions" of OMB Form 

83-I.

References

Goodman, M., Finnegan, R., Mohadjer, L., Krenzke, T., and Hogan, J. (2013). Literacy, Numeracy, and Problem

Solving in Technology-Rich Environments Among U.S. Adults: Results from the Program for the International Assessment 

of Adult Competencies 2012: First Look (NCES 2014-008). U.S. Department of Education. Washington, DC: 

National Center for Education Statistics. Retrieved [date] from http://nces.ed.gov/pubsearch.

Organization for Economic Cooperation and Development and Statistics Canada. (1997). Literacy Skills for the 

Knowledge Society: Further Results from the International Adult Literacy Survey. Paris: OECD.

Organization for Economic Cooperation and Development and Statistics Canada. (2005). Learning a Living: 

First Results from the Adult Literacy and Life Skills Survey. Paris: OECD.

Organization for Economic Cooperation and Development. (2009). PISA 2009 Assessment Framework - Key 

Competencies in Reading, Mathematics and Science. Paris: Author.

Organization for Economic Cooperation and Development. (2010). Pathways to Success: How Knowledge and 

Skills at Age 15 Shape Future Lives in Canada. Paris: Author.

Organization for Economic Cooperation and Development. (2012). Literacy, Numeracy and Problem Solving in 

Technology-Rich Environments - Framework for the OECD Survey of Adult Skills. Paris: Author.

Schleicher, A. (2007). Can competencies assessed by PISA be considered the fundamental school knowledge 

15-year-olds should possess? Journal of Educational Change, 8: 349-357 (published online July 24, 2007).