# Presentation: 481187

## GLOBE Year 10 Evaluation

**Presented at Annual GLOBE Conference**
- August 1, 2005 • Prague, Czech Republic
**Bill Penuel & Willow Sussex**
- SRI International

## Research Questions for Y10

- What are the effects of GLOBE ONE on participating scientists and schools?
- What effect does GLOBE have on students’ science inquiry skills?
- What makes GLOBE professional development effective?

## Research Questions for Y10

- What are the effects of GLOBE ONE on participating scientists and schools?
*Case study of GLOBE ONE*  (in progress)
- What effect does GLOBE have on students’ science inquiry skills?
*Quasi-experimental student achievement study*
- What makes GLOBE professional development effective?
*Large-scale teacher survey study*

## GLOBE ONE Case Study Results

**“****GLOBE ONE Evaluation Results”**
- Presentation is today
- Session B
- 15:00-16:30
**Themes explored:**
- Science and education in partnership
- Role of GLOBE ONE in fostering student research

## GLOBE’s Effects on Science Inquiry

- Quasi-Experimental Design
- Focus on middle-school students
- All teachers recruited were GLOBE-trained (to reduce selection bias)
- Some teachers selected to implement GLOBE, others as control group
- Used a proxy pre-test: Self-reported science grades
- Implementation measures
- Weekly teacher logs
- Student survey
- Outcome Measures
- Atmosphere Knowledge Test (Penuel et al., 2002)
- Atmosphere Inquiry Skills Test (Penuel et al., 2002)

## Sample

| Grade Level: | Intended GLOBE | Intended Comparison | 
TOTALS |
| --- | --- | --- | --- |
| 6th | 76 | 69 | 145 |
| 7th | 82 | 37 | 119 |
| 8th | 298 | 319 | 617 |
| TOTALS | 425 | 456 | 881 |

## Sample

| Actual Implementation Level: | Intended GLOBE | Intended Comparison | 
TOTALS |
| --- | --- | --- | --- |
| Extensive GLOBE | 127 | 30 | 157 |
| Some GLOBE | 298 | 182 | 480 |
| No GLOBE | 0 | 244 | 244 |
| TOTALS | 425 | 456 | 881 |

## Student Achievement Measures

**Atmosphere Knowledge Test**
- Test of 14 multiple choice and short answer items
- Focused on core concepts often covered in Atmosphere investigation area
- Piloted in 2002; single scale had good reliability
- Revised with input from science advisors
- New factor structure poor; scale is unreliable (α = .55)
**Atmosphere Inquiry Skills Test**
- Test of 15 multiple choice, short answer, and extended response items
-  Organized around a single performance task
- Piloted in 2002; single scale had good reliability
- Revised with input from science advisors
- New factor structure has good reliability (overall, plus 3 factors related to planning, data analysis: tables, data analysis: charts)

## Preliminary Results

- No effect of being in the treatment group (no significant differences among high, medium, and low groups either)
- Regression models did not find data collection, learning activities, or data analysis to be significant predictors of student inquiry skills
- Self-reported grades in science (the pre-test measure) was strongly predictive of inquiry scores (p < .0005)

## Interpreting the Results

- Treatment Fidelity and Contamination
- GLOBE teachers may have taught material earlier in the year
- Several of the comparison teachers taught GLOBE, despite instructions from the researchers to refrain from implementation
- Teacher Effects
- Earlier positive findings may have been partly the result of unmeasured teacher effects; this sample consisted of groups of teachers equally capable of implementing GLOBE
- “Student Effects” Prior Achievement
- With one exception (Y9), earlier studies did not account for initial differences in student achievement; prior achievement was a significant predictor in all models
- Measurement error for inquiry items not as likely an explanation
- Factor structure very similar to structure of NAEP inquiry items

## What Makes GLOBE PD Effective?

- From “Training” to “Professional Development”
- GLOBE has made the shift from a single 40-hour training session to providing teachers with a broader range of professional development experiences
- Individual teachers’ experiences in these activities vary
- Professional development goals (improved knowledge and practice) have not been examined
- Specific evaluation research questions:
- To what extent do features of effective PD identified in the research matter for GLOBE implementation?
- To what extent do features of effective PD identified in the research make a difference in teachers’ knowledge and overall teaching practice?

## Garet et al.’s (2001) Path Analysis

***Sponsorship***

***Structural Features***

***Core Features***

***Outcomes***

- Sponsor (District vs. Grantee)

- Time Span

- Contact Hours

- Collective Participation

- Focus on Content Knowledge

- Active Learning Strategies

- Coherence

- Enhanced Knowledge and Skill

- Changes to Teaching Practice

**.27**

**.21**

- Form (Traditional    vs. Reform)

**.13**

**.08**

**.27**

**.08**

**.10**

**.05**

**.13**

**.31**

**.30**

**.26**

**.14**

**.44**

**.42**

**.33**

**.21**

**.11**

**.09**

## Our Hierarchical Model

**Partner Level:**
- What do _partners do_ to influence GLOBE implementation?
- Emphasis on student inquiry in their activities
- Emphasis on supporting implementation planning
- Emphasis on building needed science content knowledge (protocols and learning activities)
**Teacher Level:**
- What of GLOBE do teachers implement?
- What about _teachers’ experience_ of professional development influences GLOBE implementation?
- Coherence with other reform initiatives
- Duration
- Form of professional development
- Participation with other teachers from their school

## Sample

- A total of 454 teachers from 28 of the most active GLOBE partnerships
- All partners completed survey in 2004
- Teachers in the sample had all received their initial protocol training in the past 2 years
- 34% response rate for teacher survey.  Non-response bias analysis indicated:
- No significant differences between respondents and non-respondents with respect to school’s SES or ethnic makeup
- Respondents were more likely to report data (more active)
- A small percentage in both groups reported data

## Explaining Protocol Implementation

**48% of survey respondents were implementing protocols**
- What predicts protocol implementation:
- Active learning strategies (β = .09, p = .05)
- Coherence (β = .68 p = .001)
- What is unrelated to protocol implementation:
- Content knowledge focus (β = -.12, p = .38)
- Inquiry focus in training (β = -.03, p = .44)
- Form (β = .08, p = .56, β = .04, p = .72)
- Participation with other teachers in training (β = .25, p = .19)

## Explaining Data Reporting

**27% of survey respondents reported data to the GLOBE Data Archive**
- What predicts data reporting:
- Inquiry focus in training (β = .18, p = .01)
- What is negatively related to data reporting:
- Content knowledge focus (β = -.63, p = .01)
- What is unrelated to protocol implementation:
- Active learning strategies (β = .02, p = .79)
- Form (β = -.04, p = .76, β = -.01, p = .93)
- Participation with other teachers in training (β = -.31, p = .14)
- Coherence (β = -.10 p = .30)

## Explaining Changes to Knowledge

## Explaining Changes to Knowledge

- What predicts changes in teachers’ knowledge:
- Coherence (β = .60, p = .001)
- What is unrelated to knowledge:
- Content knowledge focus (β = -.01, p = .84)
- Inquiry focus in training (β = -.01, p = .30)
- Active learning strategies (β = .02, p = .14)
- Form (β = .05, p = .17, β = .05, p = .17)
- Participation with other teachers in training (β = .09, p = .44)

## Explaining Reported Changes in Science Teaching Practice

## Explaining Reported Changes in Science Teaching Practice

- What predicts changes to practice:
- Form (β = .17, p = .01, β = .02, p = .72)
- Participation with other teachers in training (β = .17, p = .001)
- Coherence (β = .28, p = .001)
- What is unrelated to changes to practice:
- Content knowledge focus (β = -.03, p = .59)
- Inquiry focus in training (β = .009, p = .44) - surprise
- Active learning strategies (β = .009, p = .55)

## PD Design Implications

**Promotion of Implementation May Require:**
- Understanding of teachers’ contexts and goals for professional development
- Dedicated time for implementation planning
- Emphasis on inquiry in initial protocol training
- Equipment and technical support with set up
**Promotion of Teacher Growth May Require:**
- Understanding of teachers’ contexts and goals for professional development
- Use of classroom-based strategies (e.g., mentoring)
- Participation with other teachers

## Recommendations for NGG

- GLOBE needs to consider the feasibility of *multiple* *implementation models* from teachers’ viewpoints and the possible effects on teaching and learning for each:
- A time-limited project model (the NGG)
- A service-learning or community action model
- A student research model
- A unit model
- A model of GLOBE embedded within systemic reform
- Evaluation of implementation models requires *small-scale, controlled studies of student effects* rather than large achievement studies
- Effects will vary for each model
- Measures needed will also vary

## Recommendations for NGG

- In moving to a project-based model for GLOBE, the program must *still grapple with school realities* that make it difficult for teachers to implement:
- Ensuring coherence with system-level goals for student and teacher learning
- Need for ongoing pedagogical and technical support for a partner
- Need for more varied resources to facilitate making GLOBE a meaningful learning experience (e.g., making science more ‘real’, conducting student investigations)
- Evaluation of projects will likely require a *case-based approach* to research, with embedded quantitative studies