National Assessment Program--Science Literacy Year 6 Technical Report, 2006
Author: Wu, Margaret
Date: 2008-00-00
Type: Numerical/Quantitative Data
Subjects: Foreign Countries,Scientific Literacy,Science Achievement,Comparative Analysis,Sampling,Test Items,Research Reports,Measures (Individuals),Measurement,Test Construction,Surveys,Program Development,Consultants,Vendors,Item Analysis,Psychometrics,Evaluation Methods,Geographic Location,Gender Differences,Indigenous Populations,Student Characteristics,Urban Schools,Rural Schools,Media Literacy,Scientific Concepts,Science Tests,Science Curriculum,Data Analysis,Grade 6,National Standards,Objective Tests,Performance Based Assessment,Student Evaluation,Program Effectiveness,Educational Testing,Test Content,Test Results,Test Use,Educational Policy,Program Descriptions,Educational Indicators,Educational Assessment,Educational Objectives,National Programs,Benchmarking,Academic Standards,Scoring,Scaling,Item Response Theory,Test Interpretation,Difficulty Level,Equated Scores,Records (Forms)

In July 2001, the Ministerial Council on Education, Employment, Training and Youth Affairs (MCEETYA) agreed to the development of assessment instruments and key performance measures for reporting on student skills, knowledge and understandings in primary science. It directed the newly established Performance Measurement and Reporting Taskforce (PMRT), a nationally representative body, to undertake the national assessment program. The PMRT commissioned the assessment in July 2001 for implementation in 2003. The Primary Science Assessment Program (PSAP)--as it was then known--tested a sample of Year 6 students in all States and Territories. PSAP results were reported in 2005. The National Assessment Program--Science Literacy was the first assessment program designed specifically to provide information about performance against MCEETYA's National Goals for Schooling in the Twenty-First Century. MCEETYA has since also endorsed similar assessment programs to be conducted for Civics and Citizenship, and Information and Communications Technology (ICT). The intention is that each assessment program will be repeated every three years so that performance in these areas of study can be monitored over time. The first cycle of the program was intended to provide the baseline against which future performance could be compared. PMRT awarded the contract for the second cycle of science testing, for 2006, to a consortium of Educational Assessment Australia (EAA) and Curriculum Corporation (CC). Educational Measurement Solutions (EMS) was sub-contracted to CC to provide psychometric services. The Benchmarking and Educational Measurement Unit (BEMU) was nominated by PMRT to liaise between the contractors and PMRT in the delivery of the project. The Science Literacy Review Committee (SLRC), comprising members from all States, Territories and sectors, was a consultative group to the project. This technical report aims to provide detailed information with regard to the conduct of the National Assessment Program--Science Literacy 2006 so that valid interpretations of the 2006 results can be made, and future cycles can be implemented with appropriate linking information from past cycles. Further, a fully documented set of the National Assessment Program--Science Literacy procedures can also provide information for researchers who are planning surveys of this kind. The methodologies used in the National Assessment Program--Science Literacy 2006 can inform researchers of the current developments in large-scale surveys. They can also highlight the limitations and suggest possible improvements in the future. Consequently, it is of great importance to provide technical details on all aspects of the survey. This report is divided into nine chapters. Chapter 2 provides an outline of the test development and test design processes, including trialling and item selection, and the assessment domains of scientific literacy. The sampling procedures across jurisdictions, schools and classes are discussed in Chapter 3. Chapter 4 includes information about how the tests were administered and marked, including coding for student demographic data and participation or non-inclusion. It also provides an explanation of the reporting processes. Chapter 5 details the processes involved in computing the sampling weights. Chapter 6 provides an extensive analysis of all items included in the final test forms, including item difficulties based on Rasch modelling. Scaling and item calibration procedures leading to the placement of items and student scores within the Proficiency levels of the Scientific Literacy Progress Map are outlined in Chapter 7. Chapter 8 discusses the processes used to equate the 2003 assessment and the 2006 assessment. Chapter 9 provides information about the proficiency scale used for reporting the results, including the cut-scores for each of the levels and the placement of all test items within the levels. Appended are: (1) National Year 6 Primary Science Assessment Domain; (2) Sample School Reports; (3) Item Pool Feedback; (4) Student Participation Form; (5) Technical Notes on Sampling; (6) Programming Notes on Sampling; (7) Characteristics of the Proposed 2006 Sample; (8) Variables in File; and (9) ConQuest Control File for Producing Plausible Values. (Contains 54 tables, 11 figures and 19 footnotes.) [For main report, &quot;National Assessment Program--Science Literacy Year 6 Report, 2006,&quot; see ED534659. For school release materials, &quot;National Assessment Program--Science Literacy Year 6 School Release Materials, 2006,&quot; see ED534661.]