ethanporcaro/eric-fulltext-meta huggingface dataset:
Measuring Opportunity to Learn and Academic Language Exposure for English Language Learners in Elementary Science Classrooms. CRESST Report 767
Author: Martinez, Jose Felipe
Date: 2010-01-00
Type: Reports - Research
Subjects: Academic Discourse,Elementary School Science,Observation,Second Language Learning,Teacher Surveys,Psychometrics,Grade 4,Science Instruction,English (Second Language),Educational Opportunities,Limited English Speaking,Comparative Analysis,Educational Practices,Qualitative Research,Statistical Analysis,Interviews,Correlation,Language Proficiency,Elementary Education

The present study piloted a survey-based measure of Opportunity to Learn (OTL) and Academic Language Exposure (ALE) in fourth grade science classrooms that sought to distinguish teacher practices with ELL (English language learner) and non-ELL students. In the survey, participant teachers reported on their instructional practices and the context in their science classrooms. A small sub-sample was also observed teaching a lesson in their classroom on two occasions. The pilot data were used to investigate basic psychometric properties of the survey: specifically (a) the dimensions underlying the survey items, in particular whether OTL and ALE are distinct or overlapping features or dimensions of science instruction and (b) the match between information reported by teachers in the survey, and that collected by classroom observers. Qualitative analyses of observation and teacher open ended responses in the survey informed the interpretation of the quantitative analysis results and provided useful insights for refining the survey instrument to better capture the classroom experiences of ELL students. Appendices include: (1) Teacher Survey of Science Instruction (CA Version); (2) Classroom Observation Protocol; (3) Debriefing Interview; (4) OTL/ALE Survey Item Descriptives: Comparisons of ELL and Non-ELL students; (5) Questions; (6) Correspondence Between Teacher Survey Responses and Classroom Observations; (7) Patterns of Item inter-correlation by Scale; and (8) Additional High-Contrast Case Studies. (Contains 5 tables and 4 footnotes.)
A Conceptual Framework for Assessing Performance in Games and Simulations. CRESST Report 771
Author: Koenig, Alan D.
Date: 2010-07-00
Type: Reports - Research
Subjects: Military Training,Fire Protection,Computers,Games,Computer Simulation,Models,Bayesian Statistics,Probability,Formative Evaluation

The military's need for high-fidelity games and simulations is substantial, as these environments can be valuable for demonstration of essential knowledge, skills, and abilities required in complex tasks. However assessing performance in these settings can be difficult--particularly in non-linear simulations where more than one pathway to success or failure may exist. The challenge lies not in capturing the raw data arising from game-play, but in interpreting what a player's actions and decisions mean in the broader context of cognitive readiness for a particular job function or task. The aim of our current research is to develop a conceptual framework for assessing complex behaviors in non-linear, 3-D computer-based simulation environments. Central to this framework is the incorporation of both a domain ontology (which depicts the key constructs and relationships that comprise the domain being simulated), and one or more Bayesian networks (which catalog the probabilities of various sequences of actions related to the constructs in the ontology). For the current research, the domain is damage control related to fire-fighting onboard naval ships, and the two key constructs being assessed are situation awareness and decision-making. A 3-D, computer-based simulation depicting the interior of a naval ship has been developed. Assuming the role of a damage control investigator, the player is tasked with identifying, addressing, and reporting on a variety of potential, imminent, and existing fires and fire hazards. Using a dynamic Bayesian network, all actions and decisions related to situation awareness, communications, and decision-making are evaluated and recorded in real time, and are used for both formative and summative assessments of performance. Using this conceptual framework, our goal is to provide a generic model of assessment that can be incorporated into both new and pre-existing computer-based simulations that depict cognitively complex scenarios. (Contains 1 table and 3 figures.)
What Works? Common Practices in High Functioning Afterschool Programs across the Nation in Math, Reading, Science, Arts, Technology, and Homework--A Study by the National Partnership. The Afterschool Program Assessment Guide. CRESST Report 768
Author: Huang, Denise
Date: 2010-03-00
Type: Reports - Research
Subjects: Educational Strategies,After School Programs,Program Effectiveness,Professional Development,Educational Practices,Best Practices,STEM Education,Educational Indicators,Educational Assessment,Delivery Systems,Program Descriptions,Program Content,Program Design,Guides,Program Evaluation,Academic Standards

In an effort to identify and incorporate exemplary practices into existing and future afterschool programs, the U.S. Department of Education commissioned a large-scale evaluation of the 21st Century Community Learning Center (CCLC) program. The purpose of this evaluation project was to develop resources and professional development that addresses issues relating to the establishment and sustainability of afterschool programs. Fifty-three high functioning programs representative across eight regional divisions of the nation, including rural and urban programs, community-based and school district related programs, were identified using rigorous methods. Exemplary practices in program organization, program structure, and especially in content delivery were studied. The findings were synthesized into the Afterschool Toolkit that was made available to programs nationwide via the world-wide-web. Professional development was conducted consistently and extensively throughout the nation. Appendices include: (1) Goal Setting and Curriculum Practices: Standards-based Instruction; (2) Goal Setting and Curricular Practices: Instructional Strategies; (3) Evaluative Structures and Program Impact: Day-school Teacher Perception of Changes (Figures G-J); (4) Validation Models by Contents Technology Validation Model; and (5) The Afterschool Program Assessment System. (Contains 22 tables and 24 figures.)
Developing High-Quality Assessments that Align with Instructional Video Games. CRESST Report 774
Author: Vendlinski, Terry P.
Date: 2010-10-00
Type: Reports - Evaluative
Subjects: Learning Theories,Student Evaluation,Educational Research,Video Games,Numbers,Instructional Materials,Inferences,Educational Games,Item Analysis,Reliability,Test Items,Pretests Posttests,Teaching Methods,Knowledge Level

The evaluation of educational interventions requires assessments that consistently (reliably) produce data from which accurate (valid) inferences about the test subjects can be made for some stated purpose. Despite codified definitions of all these terms, there remains vibrant debate about the assessment design process and how measures of technical quality should be determined and reported. More importantly, there seems to be little consistency in the process of developing assessment items that demonstrate consistently high technical quality and that are conceptually aligned with instruction. We present a process, rooted in the work of Baker (1974), in which a theory of learning drives the development of knowledge specifications that then drive the integrated development of assessment items and forms and instructional materials. In this case, the instruction is in the form of an educational video game about adding rational numbers. We discuss the development of the knowledge and item specifications, an initial version of the instructional game based on these specifications, and report the consistently high technical quality (internal consistency and test-retest reliability--Cronbach's alpha of 0.9 to 0.94). We discuss as well the ability of pretest measures to predict subsequent student performance (inferential validity--Spearman's rho of 0.69) in the game and on similarly developed assessment measures. Alignment between the instructional game and assessment items is also discussed. Appendices include: (1) Knowledge Specifications; and (2) Puppetman Screen Shots from the First Version Tested with Students. (Contains 7 tables and 5 footnotes.) [This work was supported under the Educational Research and Development Centers Program, PR/Award Number R305C080015.]
Automatic Assessment of Complex Task Performance in Games and Simulations. CRESST Report 775
Author: Iseli, Markus R.
Date: 2010-08-00
Type: Reports - Research
Subjects: Performance Tests,Performance Based Assessment,Decision Making Skills,Military Training,Models,Computer Simulation,Computers,Games,Fire Protection,Cognitive Processes,Inferences,Probability,Bayesian Statistics,Expertise,Scoring,Comparative Analysis,Cost Effectiveness

Assessment of complex task performance is crucial to evaluating personnel in critical job functions such as Navy damage control operations aboard ships. Games and simulations can be instrumental in this process, as they can present a broad range of complex scenarios without involving harm to people or property. However, &quot;automatic&quot; performance assessment of complex tasks is challenging, because it involves the modeling and understanding of how experts think when presented with a series of observed in-game actions. When assessing performance, human expert scoring can be limiting, as it depends on subjective observations of in-game player's performance, which in turn is used to interpret their mastery of key associated cognitive constructs. We introduce a computational framework that incorporates the automatic performance assessment of complex tasks or action sequences as well as the modeling of real-world, simulated, or cognitive processes by modeling player actions, simulation states and events, conditional simulation state transitions, and cognitive construct dependencies using a dynamic Bayesian network. This novel approach combines a state-space model along with a probabilistic framework of Bayesian statistics, which allows us to draw probabilistic inferences about a player's decision-making abilities. Through this process, a comparison of human expert scoring and dynamic Bayesian network scoring is presented. The use of the computational framework using a dynamic Bayesian network presented in this report can help reduce or eliminate the need for human raters and decrease the time to score. This has the benefit of potentially reducing costs. In addition, it can facilitate the efficient aggregation, standardization, and reporting of the scores. (Contains 3 tables and 5 figures.)
Examining Practices of Staff Recruitment and Retention in Four High-Functioning Afterschool Programs: Extended Study from the National Afterschool Partnership Report. CRESST Report 769
Author: Huang, Denise
Date: 2010-03-00
Type: Reports - Research
Subjects: Occupational Mobility,Educational Opportunities,Faculty Development,Professional Development,Recruitment,After School Programs,Qualitative Research,Measures (Individuals),Employment Qualifications,Parent Attitudes,Interviews,Salaries,Altruism,Motivation,Work Environment,Expectation,Goal Orientation,Decision Making,Labor Turnover

This study describes how staff qualifications, decisions on staffing procedures, and professional development opportunities support the recruitment and retention of quality staff members. Four high-functioning programs were identified. Qualitative procedures and instruments were designed to capture staff and parents' academic perspectives about relationships and professional development. Study findings revealed that all staff across the four afterschool programs consistently reported an intrinsic reason for working in their program. Interview data implied that program incentives such as a career ladder and an ascending pay scale were not enticing enough to recruit or retain staff. The decisions to stay with a program tend to be altruistic in nature, such as to provide academic, social, or emotional support for the students. Thus, at these four programs, the motivation for the staff to stay with the programs could be the organized environments, clear program structures, open communication, clear program goals, consistent expectations, positive relationships, and program climates that foster staff efficacy in &quot;making a difference&quot; in their students' lives. Thus, promoting strategies in enhancing staff efficacy, such as empowering staff with decision-making and providing professional development opportunities to enhance their professional skills could help programs to recruit and retain quality staff members. Appendices include: (1) California Program Site Summary; (2) Florida Program Site Summary; (3) Indiana Program Site Summary; and (4) Texas Program Site Summary. (Contains 5 tables and 8 footnotes.)
What Probably Works in Alternative Assessment. CRESST Report 772
Author: Baker, Eva L.
Date: 2010-07-00
Type: Reports - Evaluative
Subjects: Alternative Assessment,Evaluation Methods,Evaluation Research,Performance Based Assessment,Literature Reviews,Military Service,Check Lists,Simulation,Test Bias,Test Validity

This report provides an overview of what was known about alternative assessment at the time that the article was written in 1991. Topics include beliefs about assessment reform, overview of alternative assessment including research knowledge, evidence of assessment impact, and critical features of alternative assessment. The author notes that in the short term, alternative assessment will generate negative news about student learning and will require massive support to make it a successful reform strategy. (Contains 5 tables and 3 footnotes.)
The Unintended Consequences of an Algebra-for-All Policy on High-Skill Students: The Effects on Instructional Organization and Students' Academic Outcomes
Author: Nomi, Takako
Date: 2010-00-00
Type: Reports - Research
Subjects: Educational Policy,Algebra,Graduation Requirements,Student Diversity,Ability,Knowledge Level,Grade 9,Outcomes of Education,Context Effect,Mathematics Achievement,Educational Change,High Achievement,School Organization

The purpose of this study is to understand how a policy that provided college-prep coursework for low-skill students may affect instructional organization within schools, and how such effects on instructional organization may have unintended consequences on academic outcomes of high-skill students who were not targeted by the policy. The author and her colleagues focus on a ninth-grade algebra-for-all policy implemented in Chicago public schools (CPS) and address two research questions; (1) to what extent did a policy that required algebra for all students in ninth grade affect classroom academic composition?; and (2) for high-skill students who were not targeted by the policy, how did the policy affect their academic outcomes? The results showed potential challenges of a curricular policy of expanding a college-prep curriculum to all students: schools are likely to detrack math classrooms when the policy merely required them to eliminate remedial coursework, and this resulted in lower peer ability levels for high-skill students. Consequently, their test scores suffered from this policy. This study suggests that simply mandating a college-prep curriculum for all students is not sufficient enough to improve academic outcomes of all students. An appendix presents: Statistical Models to estimate the effect of an algebra-for-all policy on students' academic outcomes among high-skill students. (Contains 3 tables and 11 figures.)
A Bayesian Perspective on Methodologies for Drawing Causal Inferences in Experimental and Non-Experimental Settings
Author: Kaplan, David
Date: 2010-00-00
Type: Reports - Evaluative
Subjects: Evidence,Educational Research,Educational Policy,Models,Measurement,Statistical Inference,Bayesian Statistics,Behavioral Sciences,Item Response Theory,Experiments

In recent years, attention in the education community has focused on the need for evidenced-based research, particularly educational policies and interventions that rest on "scientifically based research". The emphasis on scientifically based research in education has led to a corresponding increase in studies designed to provide strong warrants for causal claims. These studies have been conducted relying almost entirely on analytic procedures that are rooted in the frequentist (classical) paradigm of statistics. Over the last decade, however, significant advances have been made in the area of Bayesian statistical inference, owing mostly to computational developments and readily available software (see e.g. Gilks, Richardson, & Spiegelhalter, 1996). With these advances have come important applications of Bayesian methods to problems in the social and behavioral sciences. However, few applications of Bayesian methods to educational problems can be found, with the exception of Bayesian statistical methods for item response theory models in educational measurement (e.g. Fox & Glas, 2001). This talk proposes that Bayesian methods offer an alternative analytic paradigm that can support and enhance scientifically based research in the educational sciences.
Cognitive Underpinnings of Preschool Literacy and Numeracy
Author: Kidd, Julie K.
Date: 2010-00-00
Type: Reports - Research
Subjects: Preschool Children,Disadvantaged Youth,Preschool Curriculum,Numeracy,Emergent Literacy,Basic Skills,Abstract Reasoning,Differences,Comprehension,Art Activities,Cognitive Development,National Norms

The present research represents a test of the effect of adding seriation instruction to oddity instruction to produce an advantage in both forms of abstraction. Pasnak et al. (2007) and Kidd, Pasnak, Gadzichowski, Ferral-Like, &amp; Gallington (2008) have shown that at risk kindergartners profit academically from instruction in both oddity and seriation. This study extends this work into the preschool classroom, testing whether oddity and seriation instruction benefits at risk preschoolers in both literacy and numeracy. The research was conducted in seven Head Start preschools in Alexandria, Virginia, an urban community just south of Washington DC. That the literacy and numeracy control lessons were effective is shown by the high scores the children in those groups made in the specific domain in which they were instructed, However, the children who received the literacy or art instruction scored below national norms on numeracy, and the children taught numeracy or art  likewise scored below norms on literacy, despite the enrichment offered by their Head Start program. Such outcomes are likely when children have been raised in impoverished homes. Lacking the cognitive enrichment that would be found in many middle-class homes, they were probably functioning closer to the floor provided by their inherent abilities than to the ceiling. At least, that is one of the assumptions on which this research was based. Lagging in normal cognitive development, they would be less apt to understand and profit from preschool activities aimed at fostering literacy and numeracy. That the cognitive group matched the numeracy group in numeracy, and the literacy group in literacy, testifies to the importance of the advantages in abstraction that group had gained. These advantages appear to have enabled it to better understand the normal preschool curriculum, which offered plenty of chances to improve on literacy and numeracy. Preschool learning activities assume that children can detect relevant differences and understand the relations between big, medium, small, and so forth. When children are deficient in these abstractions, many of the instructional activities are over their heads. The authors conclude that strengthening children's understanding of differences on one dimension and unidimensional ordering enables them to gain more from learning opportunities in the preschool classroom.
Using State Tests vs. Study-Administered Tests to Measure Student Achievement: An Empirical Assessment Based on Four Recent Randomized Evaluations of Educational Interventions
Author: Zhu, Pei
Date: 2010-00-00
Type: Reports - Evaluative
Subjects: Academic Achievement,Program Effectiveness,State Standards,Testing Programs,Scores,Reliability,Achievement Tests,Comparative Analysis,Futures (of Society),Intervention

For this project, the authors use data from four IES-sponsored randomized studies to examine some of the key issues identified in May et. al. (2009). The first set of questions focuses on issues related to using state tests: (1) Do studies meet the assumptions needed for combining impacts on state tests across grades and/or states?; (2) How sensitive is the estimated impact on state tests to different strategies for rescaling test scores and/or aggregating results across states?; (3) Do estimated impacts on state tests differ from estimated impacts on the study-administered test? Under what conditions is this most likely to happen?; (4) What is the reliability of study-administered versus state tests, especially for the study-targeted population?; and (5) What are the implications for precision of using state test scores rather than a study-administered test to measure achievement in the baseline period? The findings provide valuable empirical evidence of the trade-offs between using state tests and study-administered test and have implications for designing future RCT (randomized controlled trial) studies. (Contains 7 footnotes.)
Physics Doctorates One Year Later: Data from the Follow-Up Survey of Degree Recipients from the Classes of 2007 and 2008. Focus On
Author: Mulvey, Patrick
Date: 2010-11-00
Type: Reports - Research
Subjects: Physics,Doctoral Degrees,Graduates,Surveys,Educational Attainment,Followup Studies

Each fall the Statistical Research Center conducts its Survey of Enrollments and Degrees, which asks all degree-granting physics and astronomy departments in the US to provide information concerning the numbers of students they have enrolled and counts of recent degree recipients. In connection with this survey, the authors ask for the names and contact information for their recent degree recipients. This degree recipient information is used to conduct their follow-up survey in the winter following the academic year in which they received their degrees. The data in this focus on comes from that survey. (Contains 2 figures and 2 tables.)