1



## PROGRAM FOR INTERNAT **IONAL STUDENT ASSESS MENT**  **2012 (PISA** **:2012 )**  **VALIDATION STUDY 2015 FIELD TEST AND**  **MAIN STUDY REQUEST FOR OMB CLEA** **RANCE OMB# 1850**  **- 0900**  **v .**  **2** 

## SUPPORTING STATEMENT PART A

**Submitted by: National Center for Education Statistics**  **U.S. Department of Education Institute of Education Sciences**  **Washington, DC December**  **8 , 2014** 2 

## TABLE OF CONTENTS
**PREFACE ................................**  **................................ ................................**  **................ 3**  **A. JUSTIFICATION ................................**  **................................ ...............................**  **3** A.1 Importance of Information ................................ ................................ ............................... 3 A.2 Purposes and Uses of Data ................................ ................................ ............................... 9 A.3 Use of Information Technology ................................ ................................ ....................... 9 A.4 Efforts to Identify Duplication ................................ ................................ ....................... 10 A.5 Minimizing Burden for Small Entities ................................ ................................ ........... 10 A.6 Frequency of Data Collection ................................ ................................ ......................... 10 A.7 Special Circumstances of Data Collection ................................ ................................ ..... 10 A.8 Consultations Outside NCES ................................ ................................ ......................... 10 A.9 Provision of Payments or Gifts to Respondents ................................ .......................... 10 A.10 Assurance of Confidentiality ................................ ................................ ........................... 11 A.11 Sensitive Questions ................................ ................................ ................................ ........... 12 A.12 Estimates of Response Burden ................................ ................................ ....................... 12 A.13 Estimates of Cost to Respondents ................................ ................................ ................. 12 A.14 Cost to Federal Government ................................ ................................ .......................... 12 A.15 Program Changes or Adjustments ................................ ................................ .................. 13 A.16 Publication Plans and Timetable ................................ ................................ ..................... 13 A.17 Display OMB Expiration Date ................................ ................................ ....................... 13 A.18 Exceptions to Certification Statement ................................ ................................ ........... 13 **B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL**  **METHODS** B.1 Potential Respondent Universe and Sampling B. 2 Information Collection Procedures B. 3 Methods for Maximizing Response Rates B. 4 Tests of Procedures and Methods B. 5 Individuals Consulted on Study Design **Appendix A :**  **Respondent Recruitment Materials Appendix B:**  **Draft Education and Skills O nline**  **Q uestionnaire**  **Items**3 **PREFACE** The Program for International Student Assessments (PISA) is an international assessment of 15

-year- old students sponsored by the Organization for Economic Cooperation and Development (OECD) . In the United States, it is conducted by the National Center for Education Statistics (NCES) , of the Institute of Education Sciences, U.S. Department of Education . PISA was first administered in 2000 and is conducted every three years. As part of PISA’s most recent administration, in 2012, PISA’s fifth cycle, a nationally

representative sample of approximately 5,200 15

-year- old students took the PISA assessments in the U nited States (OMB# 1850

- 0755) . PISA provides information about student performance in mathematics, science, and reading literacy, as well as problem

- solving in the United States and more than 65 other participating countries and education systems. PISA is designed to assess how well students near ing the end of compulsory schooling are prepared for the challenges of further education and the workforce . T o do so , PISA measures students’ literacy (knowledge and skills to solve problems in real

-life contexts) and collect s information about students’ home background, attitudes, and experiences in and out of school through a questionnaire completed by students. Furthermore , principals of school attended by the participating students complete a questionnaire about school policies and practices, providing additional context for the student measures . NCES proposes to conduct a follow

- up study with students who participated in PISA 2012 1 to learn how performance on PISA relates to subsequent outcomes and skills of young adults. The follow

- up study

— referred to in materials to potential respondents as the PISA Young Adult Follow

- Up Study, and in this request in short as the PISA Validation Study

— will provide information about how students’ skills and experiences at age 15, collected through PISA, relate to subsequent literacy, numeracy, and problem

- solving skills, as well as educational attainment, education and work experiences, skills used in daily life, career intentions, and aspects of health and well

- being. In fall 2015, when these students will be 18 years of age, they will be asked to take the web

- based version of the OECD’s Program for the International Assessment of Adult Competencies (PIAAC) assessment and background questionnaire

—the Education and Skills Onlin e Assessment (ESO) . In November 2013 , OMB approved the location and tracking of the PISA 201 2 respondents (OMB 1850

- 0900 v.1 ) to prepare for this follow

- up study. This request is to (1) recruit the PISA 2012 sample respondents who have been successfully lo cated and tracked ; (2) administe r ESO to a field test sample in the summer of 2015 ; and (3) administer ESO to a main study sample in the fall of 2015 . This submission describes the study rationale and design and presents the sample members recruitment materials (Appendix A) and th e ESO background questionnaire ( Appendix B). **A. JUSTIFICATION A.1**  **Importance of Information** As part of a continuing cycle of international education studies, the United States, through the National Center for Education Statisti cs (NCES), is participating in several international assessments and surveys. The Program for International Student Assessment (PISA), sponsored by the Organization for Economic Cooperation and Development (OECD), is one of these studies (OMB# 1850

- 0755) . In light of growing concerns related to international economic competitiveness, the changing face of our workplace, and the expanding international marketplace in which we trade, knowing how our students and

1 The sample of ~5,200 students took the PISA mathematics, science , and reading literacy assessment. PISA 2012 also included an assessment of financial literacy given to an additional ~1,100 students. In order to focus the Validation Study on the main PISA domains, the proposed study does not include the students who took financial literacy except in the field trial.4

adults compare with their peers around the world has become an even more prominent issue than ever before. Nationwide, interest in understanding what other nations are doing to further the education achievement of their populations has increased beyond simple comparisons. Data at critical points during the education career of our students help inform policymakers in their efforts to guide and restructure the American education system. These critical points may occur during primary, secondary, or tertiary education, as well as extend into adult education and training programs. Consequently, generating comparative data about students in school, at the end of formal schooling, in postsecondary education, and about adults in the workplace and in the community has become an important focus for NCES. PISA is part of the larger international program that NCES has actively participated in through collaboration with, and representation at, the OECD, the Asia

- Pacific Economic Cooperation (APEC), and the International Association for the Evaluation of Educational Achievement (IEA). Collaboration with Statistics Canada, Eurostat, and ministries of education throughout the world helps to round out the portfolio of data NCES compiles. The United States has participated in PISA since its first administration in 2000. PISA measures students' knowledge, skills, and competencies primarily in three subject areas

–mathematics , science , and reading literacy. The overall strategy is to collect in

- depth information on student s’ capabilities in one of these three domains every three years so that detailed information on each becomes available every nine years. During each three

-year survey cycle, the major focus is on one content domain, with a minor focus on the other two content domains. The major focus for the PISA data colle ction in 2012 was on mathematics literacy, with a minor focus on science and reading. PISA 2012 also included an assessment of problem

s olving and financial literacy. **Investigating the link between PISA and young adults’**  **preparedness for life and work** PISA is designed to measure “how well students, at age 15, are prepared to meet the challenges they may encounter in future life” (OECD 2009, pg. 9). PISA does this by identifying, with input from participating countries and subject

- matter experts, the key competencies that are believed to contribute to students’ future success , and by measur

ing students’ knowledge and skills in these areas. In short, PISA is designed to assess how well students nearing the end of compulsory schooling are prepared for further e ducation or entry into the workforce and, more fundamentally, to contribute to society as functioning young adults. Unlike other large

- scale international assessments that are based on school curricula, such as the Trends in International Mathematics and S cience Study ( TIMSS) or the Progress in International Reading Literacy Study (PIRLS), PISA assessments are designed to reflect what students have learned both in and out of school and to measure students’ ability to apply that knowledge to real

-world situations. For this reason, PISA has a unique validity challenge. TIMSS and PIRLS assessments can be validated by evaluating how well they cover mathematics, science, and reading curricula in the participating countries . However, c omparing PISA measures with c ontent standards or school curricula does not provide evidence that PISA assesses the competencies that matter for students’ “future success” and the “challenges they may encounter in the future.” To validate PISA measures , it is necessary to examine wheth er the competencies assessed by PISA predict future success. Given the growing prominence of PISA in the United States, it is of policy interest to better understand how well PISA measures students’ preparedness for further education and the workforce. PISA is widely cited as evidence that U.S. students are not as prepared for the global economy as their peers in other high

performing countries, and such claims rest on the assumption that PISA is measuring the skills that students will need to succeed in th e 21 st century and the global economy.5

**The PISA Validation Study** NCES is requesting approval to conduct a follow

- up study with students who participated in PISA 2012 to learn how performance on PISA relates to subsequent outcomes and skills of young adults. The follow

- up study

—referred to in materials to potential respondents as the PISA Young Adult Follow

- Up Study (YAFS) , and in this clearance request in short as the PISA Validation Study

— will provide information about how students’ skills and experi ences at age 15, collected through PISA, relate to subsequent literacy, numeracy, and problem

- solving skills, as well as educational attainment, education and work experiences, and skills used in daily life. This will be accomplished by administering a web

- based version of the OECD’s Program for the International Assessment of Adult Competencies (PIAAC) . PIAAC will be used to measure the skills, outcomes, and experiences of students who previously participated in PISA. The PIAAC assessment, administered in 25 countries, focuses on cognitive and workplace skills deemed necessary for successful participation in 21 st

- century society and the global economy (OECD 2012). Specifically, PIAAC measures relationships between individuals' educational background s , workplace experiences and skills, occupational attainment, use of information and communications technology, and cognitive skills in the areas of literacy, numeracy, and problem solving. The sample for the PISA Validation Study will consist of the approximately 4,700 15

-year- old students from the PISA 2012 national sample who took the PISA mathematics, reading, and science test and completed a Student Information Form with their contact information. PISA 2012 recruitment materials, including materials for parent s, stated that students supplying contact information may be contacted by NCES for a future study . In fall 2015, when these students will be approximately 18 years of age, they will be asked to take a web

- based version of the PIAAC assessment and background questionnaire developed by the OECD

—the Education and Skills Online Assessment (ESO). The ESO is described in more detail below. A field trial of the Validation Study is currently planned for summer 2015 with a sample of 200 PISA 2012 students who provid ed contact information but took financial literacy test booklets and are not the target for the PISA Validation Study main data collection . PIAAC was administered to a national sample of adults 16 to 65 years old in 2011

- 12 . It examined adults’ literacy and skills in the information age, focusing on what are deemed key skills for individuals to participate successfully in the economy and society of the 21st century (Goodman, Finnegan, Mohadjer, Krenzke, and Hogan 2013) . It assessed adults’ basic reading s kills, reading literacy, numeracy, and problem solving in technology

- rich environments (on or with a computer), and measured their ability to use computer and web applications to find, gather, and use information, and to communicate with others. PIAAC use d a “Job Requirements Approach” to ask employed adults about the types and levels of a number of specific skills used in the workplace , including not only the use of reading and numeracy skills on the job , but also the use of physical skills (e.g., stamina and manual dexterity), people skills (e.g., public speaking, negotiating, working in a team), and information technology skills (e.g., using spreadsheets, writing computer code) . PIAAC also asked about adults’ background s , work and educational experiences , including the requirements of their main job in terms of the intensity and frequency of the use of the aforementioned skills. The web

- based version of the PIAAC assessment and background questionnaire (ESO) will be distributed by the OECD for use by countries, institutions , and individuals to support research and workforce development and training efforts. The ESO consists of two main components that are administered together: an assessment and a questionnaire. The ESO assessment, which is currently being finalized for distribution to countries by the OECD, was piloted in multiple countries and has been equated to PIAAC, allowing for international benchmarking. The ESO questionnaire is based on that used in PIAAC.6

**Administering the ESO to PIS** **A 2012 participants** In the PISA Validation Study, the ESO assessment and background questionnaire will be administered to sample respondents in 2015 when they are approximately 18 years old, three years after they took PISA in 2012 . The assessment is designed to take on average 90 minutes to complete and the background questionnaire approximately 30 minutes , for a total of 120 minutes on average per respondent . The ESO assessment measures a set of cognitive and non

- cognitive skills deemed necessary for full participation in modern societies. These skills and knowledge include being able to understand and use printed and electronic texts, reason with numbers, and solve problems using information and communication technologies (see exhibit A

- 1, below, for an overview of the cognitive content of the ESO). **Exhibit A -**  **1. Summary of the cognitive domains in the OECD Education and Skills Online (ESO)**  **Assessment**

**Literacy Numeracy**  **Problem solving in technology****** **rich environments**
**Definition** Ability to understand, evaluate, use

and engage with written texts to participate in society, achieve one’s goals, and develop one’s knowledge and potential. Ability to access, use, interpret and communicate mathematical information and ideas in order to engage in and manage the mathematical demands of a range of situations in adult life. Ability to use digital technology, communication tools and networks to acquire and evaluate information, communicate with others, and perform practical tasks.

**Content** Texts are characterized by

Medium:  Print

- based  Digital Format:  Continuous or prose texts (narration, argumentation , or descriptions)  Non

- continuous or document texts (tables, lists, graphs)  Mixed texts (combination of prose and document elements)  Multiple texts (juxtaposition or linking of independently generated elements) Mathematical content, information and ideas:  Quantity and number  Dimension and shape  Pattern, relationships, change  Data and chance Representations of mathematical content:  Obj ects and pictures  Numbers and symbols  Diagrams, maps, graphs, tables  Texts  Technology

- based displays Technology:  Hardware devices  Software applications  Commands and functions  Representations (text, graphics, video) Nature of problems:  Intrinsic complexity (number of steps, alternatives required for solution, complexity of computation and/or transformation, number of constrains)  Explicitness of the problem statement (largely unspecified or described in detail **Cognitive Strategies**   Access and identify

 Integrate and interpret (relating parts of text to one another)  Evaluate and reflect on  Identify, locate or access  Act upon and use (order, count, estimate, compute, measure, model)  Interpret, evaluate and analyze  Communicate  Setting goals and monitoring progress  Planning  Acquiring and evaluating information  Using information **Context**  Personal

 Work

- related  Community  Education  Everyday life  Work

- related  Society & Community  Education  Personal  Work

- related  Community Source: http://www.oecd.org/site/piaac/ENGLISH_Brochure%20Education%20and%20Skills%20Online.pdf . The ESO questionnaire , in turn, asks about personal and home bac kground and education and workforce experiences (all taken from the PIAAC questionnaire) , and includes modules that were developed and piloted by the OECD for ESO . The PIAAC background questionnaires were more extensive than what will be7

included in the ESO, however. As shown in exhibit A

- 2, below, the content of the ESO questionnaire modules focus on three areas: (i) basic demographics, (ii ) career interests and intentionality (CII) and ( iii ) behavioral performance competencies (BPC) . It is possible to use one or more of the modules developed by the OECD , but it is not possible to add or remove items or edit items within the modules. That is, the ESO modules must be administered intact . Because the ESO must be administered as designed , NCES is planning to administer an ‘add

- on’ questionnaire module , with 16 additional items administered in PIAAC but not included by the OECD in ESO . These additional questions address students’ recent and current educational status and experiences, including non - formal learning activities. This information is critical for realizing the aims of the PISA Validation Study, which is to examine the relationship between PISA and the paths that young adults choose after leaving high school . The themes covered by these additional survey items are also shown in exhibit A

- 2. More detailed information on all of the non

- cognitive survey items is included in Appendix B. Please note that because the ESO modules are designed to be administered online , detailed information on the survey items provided in Appendix B are shown in tables, as there is no paper version of the questionnaire. **Exhibit A -**  **2. Summary of the**  **non -**  **cognitive modules in the OECD’s Education and Skills Online**  **(ESO) Assessment Non**  **- cogni** **tive survey module**  **Content theme s**  **Background + Educational Experiences**  **(Core)**  Age  Sex  Highest level of education attained  Country of origin  Language spoken at home  Native language  Current employment status  Current occupation **Career Interest and Intent (CII)**   Inventory of vocational interests  Intent to seek employment  Intent to seek job training  Job seeking behavior **Behavioral performance competencies**  **(BPC)**  Well being  Self

- control  Responsibility  Sociability  Curiosity  Cooperation  Generosity  Adjustment  Intellectual efficiency  Ingenuity  Achievement/Goal setting  Dominance  Organization and Order **Additional items not already included in ESO**   Current educational trajectory  Degree/certification being sought  General category of degree/certification being sought  Non

- formal learning activities (distance education; seminars)  Additional learning activities  Reasons for non

- participation in learning activities  Use of tutors and participation in apprenticeship programs **Rationale for including CII and BPC modules** Of the original four modules being developed by the OECD for the ESO, NCES proposes to administer two (in addition to the demographic module). Both modules are derived from PIAAC modules of the same name which were field trialed and administered in the main PIAAC 2011

- 12 data collection. The Care er Interests and Intentionality (CII) module will provide information about the employment status and vocational interests of students in the PISA Young Adult Follow

- up Study, at approximately age 18, which can be related to their performance on PISA at age 15. Questions about employment status will provide detailed information about what students are doing for work and/or if they are seeking employment, which will be8

especially useful for understanding the current experiences of students who did not go on to higher education. For all students in the sample the module will provide information about occupational interests and intentions that can be related to performance on PISA. The Behavioral Performance Competencies (BPC) module addresses a range of non

- cognitive constructs, including well

- being, self

- control, cooperation, and responsibility. This module will provide a way to relate non - cognitive outcomes of students in early adulthood to earlier performance in reading, mathematics, and science, to address the question of how performance on PISA relates to successful participation in adult life. As noted, both the CII and BPC modules have already been used in the field for PIAAC 2011

- 12 data collection so their extension to ESO does not require further testing. **Rationale for additional non -**  **cognitive survey items** As indicated above, NCES proposes to augment the ESO questionnaire modules with 16 additional items as an ‘add

- on’ module that, from a respondent perspective, will seamlessly blend with the ESO. The rationale for proposing these additional items is provided below . These additional items are also taken from the PIAAC 2011

- 12 questionnaires but were not included in the ESO modules distributed by the OECD. NCES believes these additional items will bring added analytic insight to the results of the PISA Validation Study. It is estimated that over 80 percent of the U.S. 15

- 17 years

- old co hort go on to some form of post secondary education (ELS: 2002

- 2012) 2 . Moreover, some proportion of these young adults stays back in high school. Therefore, four questions on **Current educ**  **ation stud**  **y status** (participation; level of degree; area of study) are proposed to be added. Given that some respondents may have started and stopped their education, or dropped out, within the last year, one question on **Studies in the last 12 months** is proposed to be added. Additionally, a portion of the respondents may be participating in some form of GED or basic skills courses, leading to three additional questions on **Basic education courses in the last 12 months** (reading/writing/math, GED, some other adult education) proposed to be added. A small proportion of young adults report s participating in apprenticeship programs . Therefore one question on the **Formal apprenticeship in the last 12 months**  is proposed to be added . NCES is proposing to add these nine questions to allow for a more comprehensive tracking of the PISA

- PIAAC Follow

- up sample into various forms of formal educatio n. In addition to the nine questions above, another seven questions are being proposed to address the non

formal educational c hoices of young adults. While a large proportion of young adults participate in formal education, PIAAC data shows that 55 percent of U.S. 16

- 19 year olds also participated in non

- formal education activities (PIAAC:2012) 3
. To ascertain whether respondents are participating in non

- formal education , five questions on **Non -**  **formal learning activities** (distance education, on

- the

- job courses, seminars, other private lessons) are proposed to be included. Finally, with the intent of tracking barriers to non - formal learning activities the respondents may experience , two questions on **More/any learning activities, wanted, but could not participate in**  and reasons for non

- participation are proposed to be included.

2 Lauff, E., and I

ngels, S.J. (2013). Education Longitudinal Study of 2002 (ELS:2002): A First Look at 2002 High School Sophomores 10 Years Later (NCES 2014 - 363). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved [11/11/2014] at http://nces.ed.gov/pubs2014/2014363.pdf 3 Organization for Economic Cooperation and Development (OECD), Program for the I nternational Assessment of Adult Competencies (PIAAC), 2012. Retrieved from PIAAC IDE [11/11/2014].9

As indicated earlier, respondents will be requested to complete both the cognitive and non

- cognitive instruments online. Appendix B includes the ESO non - cognitive survey items in the modules, including the additional items, proposed to be administered for this study. **A.2 Purposes and Uses of Data**  PISA is intended to be an assessment at or near the end of compulsory education of how well students are prepared for further education or entry into the workforce and to contribute to society as functioning young adults. The PISA subject matter assessments are not developed by canvassing the curricula offered in participating countries, as is done in other large

- scale international assessments. The PISA assessment frameworks and items are developed by conferring with subject matter experts around the world and representatives of participating countries about the mathematics, reading, and science competencies required for 15

-year- olds to successfully transition to further education, the workforce, and otherwise as functional contributing members of society. For 2012, the mathematics framework was also circulated internationally among representatives of employers and postsecondary institutions for their comments on the appropriateness of aspects of the PISA mathematics framework. Because of its emphasis on assessing what i s thought to be important for later success rather than assessing what has been taught, PISA must turn to sources of evidence other than participating countries’ curricula in order to check that it is assessing what it intends to assess. Currently, the main source for PISA for this sort of validity checking is in the form of experts drawn from across participating countries , as described earlier . Also, t wo other countries have performed longitudinal studies based on PISA. A longitudinal study in Canada has followed 30,000 students who participated in PISA 2000, gathering information every two years on their educational experiences, transitions, and attainment and employment outcomes (OECD 2010). A similar study drawing on a smaller sample was conducte d in De nmark (Schleicher 2007). What is lacking is an empirical linkage between PISA and the sorts of skills and outcomes required for successful participation in adult life. The proposed study will provide data on how performance on PISA , at age 15 , relates to literacy, numeracy, and problem

- solving skills; educational attainment and education and workforces experi ences; skills use in daily life; and other non

- cognitive outcomes at approximately age 18. For example, what thresholds of performance on PISA are asso ciated the ability to successfully navigate the literacy, numeracy, and problem

- solving demands of daily life in the adult world? What thresholds are associated with high school graduation and successful transition to higher education or entry

-level workfo rce? How does performance on PISA relate to measures of subjective well

- being at age 18 , such as life satisfaction and healthy habits ? By linking performance on PISA to subsequent outcomes in young adulthood, the study will provide the sort of validity evidence needed to understand how well PISA is measuring the skills needed for successful participation in adult life, as it purports to do . The study will also strengthen our understanding of U.S. performance on PISA and its implications for U.S. college and career readiness and for the skills of our future workforce. This information can, in turn, also be used to inform the further development of PISA and future surveys of adult skills. **A.3 Use of Information Technology**  Study instruments will be administered through the Internet. Respondents will be invited to log on to a secure web site to take the assessment and questionnaire. The use of web

- based instruments reduces the burden required for data collection, scoring, an d data p rocessing.10

**A.4 Efforts to Identify Duplication**  Comparable U.S. data are not available from other data sources; nor would a PISA follow

- up study make any data redundant. The purpose of this study is to look at the relationship of performance on PISA to perf ormance on similar domains and to other outcomes after a period of time , to validate PISA ’s measures . No other study in the United States has collected follow

- up data from or about PISA students so this would be a unique source of data to accomplish this o bjective. The current study could produce some information that is similar to a subset of information from the High School Longitudinal Study ( HSLS), which follows a similar age group . H owever, HSLS does not provide data that allow for the validation of PI SA measures or that are directly comparable with data from other countries . **A.5 Minimizing Burden for Small Entities**  This study will not impose burden on small businesses or small entities. **A.6 Frequency of Data Collection**  Currently the main data collectio n is planned for fall 2015. A field trial of procedures will take place in summer 201 5 . Although there are currently no plans for a second follow

- up study with the PISA 2012 students, NCES may decide at a later date to re

- contact the cohort sample to collect additional data at another point in time on education and workforce experiences and skills in orde r to evaluate PISA measures against longer term outcomes . **A.7 Special Circumstances of Data Collection**  No special circumstances of data collection are anticipated. **A.8 Consultations**  **outside NCES**  NCES has sought input from a wide range of individuals with knowledge of PISA and PIAAC. Both PISA and PIAAC are sponsored by the OECD and are coordinated internationally by a consortium of contractors responsib le for various aspects of the studies. Key persons from the OECD and key contracting organizations involved in the international design, development, and operation of PISA and PIAAC are as follow s . **Organization for Economic Cooperation and Development**

Andreas Schleicher, Indicators and Analysis Division 2, rue André Pascal, 75775 Paris Cedex16, FRANCE William Thorn, PIAAC 2, rue André Pascal, 75775 Paris Cedex16, FRANCE **Educational Testing Service** Irwin Kirsch, Project Director, ETS Corporate Headquarters 660 Rosedale Road, Princeton, NJ 08541 USA **A.9 Provision of Payments or Gifts to Respondents**  As part of the planned efforts to reach response rate goals, NCES propose d giving sampled respondents the same incentive used for the PIAAC Main Study: a $50 cash card to thank the respondent for their time and effort spent taking the assessment and answering the survey . The amount of burden is equivalent to that of the PIAAC respondents. NCES received OMB approval to offer a $50 incentive for the PIAAC Main Study11

after the results of an incentive experiment during the 2010 PIAAC Field Test indicated that $50 improved response rates compared with lower amounts (OMB# 1850

- 0870) . The $50 incentive was approved for this study in the Initial Contact and Address Updates request (OMB# 1850

- 0900 v.1). Due to the unclustered sample, it would be prohibitively expensive to conduct an in

- person data collection. A web approach has been designed to administer the ESO to PISA 2012 students. During recruitment, respondents will be supplied with an unloaded cash card that will be loaded upon their completion of the ESO assessment and questionnaire . The card can be used by the respondent anywhere a MasterCard is accepted , once remuneration has been added to it. Until the survey is completed, the card will act as a visual incentive to the respondent to complete the survey. **A.10 Assurance of Confidentiality**  Procedures for handling confidential aspects of the study that will be used in the PISA Validation Study will mirror those used in p ast administrations of PISA. Expertise in data security and confidentiality wa s a significant criterion in the selection of the PISA contractor. The plan for maintaining confidentiality includes signed confidentiality agreements and notarized nondisclosure affidavits obtained from all personnel who will have access to individual identifiers ; personnel training regarding the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses; controlled and protected access to computer files under the control of a single data base manager; built

-in safeguards concerning status monitoring and receipt control systems; and a secured and operator- manned in

- h ouse computing facility . The physical and/or electronic transfer of personally identifiable information (particularly names and addresses) will be limited to the extent necessary to perform project requirements. This limitation includes both internal trans fers (e.g., transfer of information between agents of Westat) and external transfers (e.g., transfers between Westat and NCES, or between Westat and another government agency assisting in data collection). All data files constructed to conduct the study will be maintained in secure network areas at Westat. These files will be subject to Westat’s regularly scheduled backup process. Backups are stored in secure facilities on site as well as off site. These data are stored and maintained in secure network and database locations where access is limited to those Westat staff who have specifically authorized access. Access is only granted once a staff member is assigned to the project and has completed the NCES Affidavit of Non

- disclosure. Identifiers are maintain ed in files required to conduct survey operations that are physically separate from other research data and that are accessible only to sworn agency and contractor personnel . The PISA Validation study will follow all applicable federal laws pertaining to s ecurity and confidentiality of personally identifiable information (PII). The laws pertaining to the collection and use of PII are clearly communicated in correspondence with participants, per NCES requirements (see materials in Appendix A). Materials will carry a statement addressing confidentiality as follows:

NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C., § 9543). Your p articipation is voluntary and the information you provide may be used o nly for statistical purposes and may not be disclosed or used, in identifiable form for any other purpose except as required by law (ESRA 2002, 20 U.S.C., § 9573). Individual responses will be combined with those from other participants to produce summary statistics and reports. Westat will deliver data files, accompanying software, and documentation to NCES at the end of the main study. Neither respondents’ names nor addresses will be included on any data file. Furthermore, prior to the release of any data file, NCES will conduct a disclosure risk analysis, per NCES standard protocol.12

**A. 11 Sensitive Questions**  The survey does not include questions usually considered to be of a highly sensitive nature, such as items concerning religion, substance abuse, or s exual activity. **A. 12 Estimates of Response Burden**  Table A

- 1 shows the estimated respondent burden for field trial and main study recruitment and data collection. There are combined total burden hours of 2,086 hours for the field test and main study data p hases. These include communications with respondents asking them to formally participate in the survey, completion of the non

- cognitive questionnaire, and completion of the cognitive assessment. At an estimated $7.25 per hour (federal minimum wage), the es timated cost to respondents for the total burden time associated with the address updates (2,086 hours) is $ 15,124. **Table A -**  **1. Estimated burden to respondents** * **Activity Sample**  **size Estimated**  **r esponse**  **rate Number**  **of**

**Respondents Number of** 

**Responses Estimated a**  **verage**

**burden (minutes) Total burden** 

**(hours)** Address update requests - main study sample (5) 4,729 1.00 4,729 4,729 5 min per request 25 min total 1,970 Address update requests – field trial sample (3) 1,081 1.00 1,081 1,081 5 min per request 15 min total 270 Total burden address updates 5,810 5,810 2,240 **Field trial** Communications 235 0.85 200 200 10 33 Questionnaire 235 0.85 200 200 30 100 Assessment 235 0.85 200 200 90 300 **Data collection** Communications 3,447 0.85 2 , 930 2,930 10 488 Questionnaire 3,447 0.85 2 , 930 2,930 30 1,465 Assessment 3,447 0.85 2930 2,930 90 4,395 **Total burden -**  **data collections (communications and questionnaire)**  **3,130 6,260**  **2,086** *Total burden in the current request is for the field trial and main study communications and data collection of the questionnaire . Gray shaded rows indicate the portion of burden that was previously approved and complete d , and entries in gray font represent burden associated with assessments, which is not subject to PRA approval and therefore not included in the burden totals. **A.13 Estimates of Cost to Respondents**  Other than the burden time associated with completing the address updates and study instruments, the study imposes no additional cost to respondents. **A.14 Cost to Federal Government**  The cost to the Federal Government for conducting the validation study is estimated to be $1,945,602 over a four

-year period , based on the cost for the Validation Study task in the PISA 2015 contract (ED

- IES

- 13

- C

0006) . This includes tracing and tracking sample respondents, data collection, use of the OECD’s ES O instruments, and analysis and reporting. These figures include all direct and indirect costs of the project.13

**A.15 Program Changes or Adjustments**  The ap parent de crease in respondent burden for this collection is due to the fact that the last OMB approval was for two address updates in 2013

- 14 with all PISA 2012 sample members who supplied their contact information in 2012, while this 2015 field test and main study data collection will be conducted with only a subset of the original sample. **A.16 Publication Plans and Timetable**  NCES will release a general audience report and technical/operations report, along with restricted

- use data files. Electronic versions of each report will be made available on the NCES website. The expected data collection dates and a tentative reporting schedule are provided below.

. **Date Activity**  Fall 2012 Collect contact information from PISA 2012 students (completed) Fall 2013

– September 2015 Establish initial contact and conduct semi

- annual contact with respondents for address updates Summer 2015 Conduct field trial of procedures Fall 2015 Web

- based main study data collection 2016

- 2017 Reporting: general audience report; survey operations report; restricted

- use data

- files **A.17 Display OMB Expiration Date**  The OMB expiration date will be displayed on all data collection materials. **A.18 Exceptions to Certification Statement**  No exceptions are requested to the "Certification for Paperwork Reduction Act Submissions" of OMB Form 83

- I. **References** Goodman, M., Finnegan, R., Mohadjer, L., Krenzke, T., and Hogan, J. (2013). Literacy, Numeracy, and Problem Solving in Technology

-Rich Environments Among U.S. Adults: Results from the Pr ogram for the International Assessment of Adult Competencies 2012: First Look (NCES 2014

- 008). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved [date] from http://nces.ed.gov/pubsearch. Organization for Econ omic Cooperation and Development and Statistics Canada. (1997). Literacy Skills for the Knowledge Society: Further Results from the International Adult Literacy Survey. Paris: OECD. Organization for Economic Cooperation and Development and Statistics Canada. (2005). Learning a Living: First Results from the Adult Literacy and Life Skills Survey. Paris: OECD. Organization for Economic Cooperation and Development. (2009). PISA 2009 Assessment Framework

- Key Competencies in Reading, Mathematics and Science. P aris: Author. Organization for Economic Cooperation and Development. (2010). Pathways to Success: How Knowledge and Skills at Age 15 Shape Future Lives in Canada. Paris: Author. Organization for Economic Cooperation and Development . (2012). Literacy, Numer acy and Problem Solving in Technology

-Rich Environments

- Framework for the OECD Survey of Adult Skills. Paris: Author. Schleicher, A. (2007). Can competencies assessed by PISA be considered the fundamental school knowledge 15

-year- olds should possess? Journal of Educational Change, 8: 349

-357 (published online July 24, 2007).