**National Mathematics Advisory Panel**

# Instructional Practices Task Group

Progress Report

Miami Dade College, Miami, FL

June 6, 2007

## Task Group Members

-   Russell Gersten

-   Camilla Benbow

-   Douglas Clements

-   Bert Fristedt

-   Tom Loveless

-   Vern Williams

-   Joan Ferrini-Mundy, Ex Officio

-   Irma Arispe, Ex Officio

## Report on the Role of Technology

**Fundamental Question:** What is the role of technology including
computer software, calculators and graphing calculators, in mathematics
instruction and learning?

### Three Sections

-   Description of *categories* of software

-   Synthesis of extant reviews, emphasizing meta-analyses

-   NMP's own meta-analysis of individual studies, *focusing on
    calculators*

+----------+----------------------------------+-----------------------+
| **Ca     | **Typical Pedagogy**             | **Possible Features** |
| tegory** |                                  |                       |
+----------+----------------------------------+-----------------------+
| Drill    | Linear                           | Sequence              |
| and      |                                  |                       |
| Practice | Repetitious                      | Management            |
|          |                                  |                       |
|          | Presentation of task, student    | Feedback              |
|          | response, feedback               |                       |
|          |                                  | Controlled            |
|          |                                  | introduction of new   |
|          |                                  | items                 |
|          |                                  |                       |
|          |                                  | Distributed practice  |
|          |                                  |                       |
|          |                                  | Reinforcement         |
|          |                                  | schedules             |
+----------+----------------------------------+-----------------------+
| T        | Linear progression with various  | Sequence              |
| utorials | amounts of branching             |                       |
|          |                                  | Management            |
|          | Didactic presentation and,       |                       |
|          | sometimes, Socratic dialogue,    | Feedback              |
|          | presentation of information,     |                       |
|          | questioning, and feedback        | Instructional events  |
|          | depending on the response;       |                       |
|          | branching to explanations or     |                       |
|          | review                           |                       |
+----------+----------------------------------+-----------------------+
| Sim      | Nonlinear;                       | Integrated with       |
| ulations | exploratory/inquiry-oriented     | tutorials or teaching |
|          |                                  | tools                 |
|          | Provides a model of real-world   |                       |
|          | or mathematical situation in     | Appropriate           |
|          | which students act; then         | simulation            |
|          | responds to student input        |                       |
|          | following that model             |                       |
+----------+----------------------------------+-----------------------+
| Games    | Provides a set of tools and/or   | Math emphasis         |
|          | miniature \"world\" as setting   |                       |
|          | for attempting to achieve a goal | Intrinsic math        |
|          | within a framework of rules      |                       |
|          |                                  | Manipulation of       |
|          | Provides clear goals, a set of   | concepts              |
|          | artificial rules, and elements   |                       |
|          | of competition                   | Motivational elements |
+----------+----------------------------------+-----------------------+
| Tools    | Specific functions (calculator,  | Integration/data      |
| and      | graphing, computer-based         | communication across  |
| Problem  | laboratories, geometric          | tools (or with other  |
| Solving  | construction, CAS)               | software categories)  |
|          |                                  |                       |
|          | Problem Solving may include      | Specific feature sets |
|          | presentation of problems and     |                       |
|          | feedback                         |                       |
+----------+----------------------------------+-----------------------+
| Pro      | Specific language                | Math emphasis         |
| gramming |                                  |                       |
+----------+----------------------------------+-----------------------+
| Internet | Type: General information        |                       |
|          | search/retrieval, \"WebQuest,\"  |                       |
|          | other                            |                       |
+----------+----------------------------------+-----------------------+
| Tools    | Type: electronic blackboard,     | Integration/data      |
| for      | demonstration/display,           | communication across  |
| Teachers | management system (CMI; may      | tools (or with other  |
|          | include practice generator),     | software categories)  |
|          | item/test/practice generator,    |                       |
|          | classroom interaction (each      |                       |
|          | student has device)              |                       |
+----------+----------------------------------+-----------------------+

## 

## 

## Synthesis of Reviews

-   Caveats: Many studies included in previous reviews, including
    meta-analyses, would not have met IP's criteria.

### All CBI

-   All subjects\
    (math + others)

    -   Median .35

-   Math only

    -   Median .345

-   Problem solving

    -   Median .215

  ---------------------------------------------------- ------------------
  **Meta-analysis**                                    **Pooled effect
                                                       size**

  ***General achievement***                            

  Fletcher-Flinn & Gravatt (1995)                      .24

  Kulik & Kulik (1991)                                 .30

  Ryan (1991)                                          .31

  Lee (1990)                                           .32

  Khalili & Shashaani (1994)                           .38

  Hartley (1978)                                       .41

  Niemiec & Walberg (1984)                             .42

  Kulik (1994)                                         .48

  ***Mathematics achievement***                        

  Slavin, Lake & Groff (2007)                          .18

  Christmann, Badgett & Lucking (1997)                 .18

  Slavin & Lake (2007)                                 .19

  Kuchler (1999)                                       .32

  Fletcher-Flinn & Gravatt (1995)                      .32

  Burns & Bozeman (1981)                               .37

  Chambers (2002)                                      .37

  Kulik & Kulik (1991)                                 .37

  Lee (1990)                                           .38

  Khalili & Shashaani (1994)                           .52

  ***Problem solving***                                

  Lee (1990)                                           .17

  Chambers (2002)                                      .20

  Hembree (1992)                                       .23

  Kuchler (1999)                                       .24
  ---------------------------------------------------- ------------------

##### 

### CBI Compared to Other Interventions for Individualization

-   CBI \< Ind. Tutoring

-   CBI \> Others

-   CBI \< Acc. classes

-   CBI \> Others

  -------------------------------------------------------------------------------------------------------------------------------
  **Table 3. Comparison                                                                                            
  of effects sizes for                                                                                             
  different mathematics                                                                                            
  interventions for                                                                                                
  individualization**                                                                                              
  --------------------- ----------- ------------- ------------- -------------- ----------- ------------ ---------- --------------
  **Study**             **Pooled                                                                                   
                        effect                                                                                     
                        size**                                                                                     

                        *CBI/CAI*   *Individual   *Individual   *Programmed                                        
                                    tutoring*     learning      instruction*                                       
                                                  packets*                                                         

  Hartley (1978)        .41         .60           .16           .11                                                

                        *CBI*       *Peer         *Learning     *Programmed    *Mastery    *Grouping*   *Special   *Accelerated
                                    tutoring*     packages*     instruction*   learning*                classes,   classes,
                                                                                                        gifted*    gifted*

  Kulik (1994)          .48         .38           .19           .07            .10         .19          .50        .93
  -------------------------------------------------------------------------------------------------------------------------------

### CBI Compared to Other Math Interventions

-   CBI \< "Processes"

-   CBI \> Curricula

  --------------------------------------------------------------------------------------
  **Table 4.                                                               
  Comparison of                                                            
  effects sizes for                                                        
  different                                                                
  mathematics                                                              
  interventions**                                                          
  ----------------- ------------- ---------- ------------- --------------- -------------
  **Study**                       CBI        Mathematics   Instructional   Cooperative
                                             curricula     processes ^a^   learning

  Slavin & Lake     Elementary    .19        .10           .29             .29
  (2007)                                                                   

  Slavin et al.,    Secondary     .18        .05           .21             .32
  2007                                                                     

  ^a^ E.g.                                                                 
  metacognitive                                                            
  strategy                                                                 
  instruction,                                                             
  individualized                                                           
  instruction,                                                             
  mastery learning,                                                        
  comprehensive                                                            
  school reform;                                                           
  includes                                                                 
  cooperative                                                              
  learning.                                                                
  --------------------------------------------------------------------------------------

### By Goal

-   Computation

    -   Median .45

-   Concepts

    -   Median .45

-   Problem solving

    -   Median .225

  -----------------------------------------------------------------------
  **Table 5. Effects of CBI             
  on specific educational               
  goals**                               
  --------------------------- --------- ---------------------------------
  **Study**                   Pooled    Study information
                              effect    
                              size      

  ***Computation***                     

  Kuchler (1999)              .39       Secondary

  Lee (1990)                  .45       All grades

  Burns & Bozeman (1981)      .69       All grades

  ***Concept development***             

  Lee (1990)                  .37       All grades

  Kuchler (1999)              .53       Secondary

  ***Problem solving***                 

  Lee (1990)                  .17       All grades

  Burns & Bozeman (1981)      .20       All grades

  Kuchler (1999)              .21       Elementary

  Kuchler (1999)              .24       Secondary

  Hembree (1992)              .23       All grades, CAI

  Hembree (1992)              .29       All grades, calculators
  -----------------------------------------------------------------------

### Contextual and Implementation Variables

-   Variance suggests other variables at work

-   Contextual variables (e.g., settings, such as urban/suburban/rural
    and student or family characteristics)

-   Implementation variables (e.g., duration, use of the intervention as
    a supplement or substitution for conventional instruction, fidelity
    of implementation; support and availability of resources, funds, and
    time; setting within the school)

-   Needs to examine; also need more comprehensive and nuanced reporting
    and analysis

**Contextual**

-   Grade level

    -   No consistent\
        pattern

        -   Ability

    ```{=html}
    <!-- -->
    ```
    -   Low \> other

    -   High, perhaps

  ---------------------------------------------------------------------------------------------------
  **Table 7 Contextual                                                             
  Variables**                                                                      
  ----------------------- ------------------ ------------------ ------------------ ------------------
  **Study**               **Pooled effect                                          
                          size**                                                   

  *Age/grade level*       *Pre-school*       *Elementary*       *Middle/Junior*    *High school*

  Fletcher-Flinn &        .55                .26 (grades 1-6)                      .20 (grades 7-12)
  Gravatt (1995)                                                                   

  Hartley (1978)                             .42 (grades K-8)                      .30 (grades 9-12)

  Lee (1990)                                 .37                .37                .32
  (achievement)                                                                    

  Lee (1990) (problem                        .18                .24                -.01
  solving)                                                                         

  Chambers (2002)                            .43                .62                .58
                                             \[weighted=.20\]   \[weighted=.24\]   \[weighted=.67\]

  Kuchler (1999,                                                .35                .37
  secondary)                                                                       

  Khalili & Shashaani                        .34                .11                .62
  (1994)                                                                           

                                                                                   

                                                                                   

  *Ability level*         *Low*              *Average*          *High/Average*     *High*

  Chambers (2002)         .68                .54 \[weighted=                       .59 \[weighted =
                          \[weighted=.35\]   .92\]                                 .61\]

  Hartley (1978)          .44                .36 (middle)                          .31

  Kuchler (1999,          .55                .52                                   -.23
  secondary)                                                                       

  Lee (1990)              .41                .15                                   .26

  Fletcher-Flinn &        .08                                                      .16
  Gravatt (1995)                                                                   
  ---------------------------------------------------------------------------------------------------

**Contextual**

-   Gender

```{=html}
<!-- -->
```
-   Males\
    benefit more

    -   SES

```{=html}
<!-- -->
```
-   Low \> other

-   Few studies

  --------------------------- -------------- -------------- --------------
  *Gender*                    *Males*        *Females*      

  Lee (1990)                  .32            .10            

  Lee (1990) (attitude        .22            .03            
  outcome)                                                  

  Hartley (1978)              .24            .23            

  Kuchler (1999, secondary)   .44            .45            

  Fletcher-Flinn & Gravatt    -.02           .36            
  (1995)                                                    

                                                            

  *SES*                       *Low*          *Average*      *High*

  Lee (1990)                  .28            .35            .32

  Kuchler (1999, secondary)   .82            .12            
  --------------------------- -------------- -------------- --------------

Implementation

-   Supplement better

-   Classroom better in elementary, possibly all

-   Researcher/\
    teacher better developer

-   And better if specific audience

+----------------------------------+-----------------+-----------------+
| *Substitute v supplement*        | *Substitute*    | *Supplement*    |
+----------------------------------+-----------------+-----------------+
| Kuchler (1999, secondary)        | .28             | .38             |
+----------------------------------+-----------------+-----------------+
| Hartley (1978)                   | .08             | .47             |
+----------------------------------+-----------------+-----------------+
| Lee (1990) (achievement)         | .31             | .40             |
+----------------------------------+-----------------+-----------------+
| Lee (1990) (problem solving)     | .05             | .30             |
+----------------------------------+-----------------+-----------------+
|                                  |                 |                 |
+----------------------------------+-----------------+-----------------+
| *Computer laboratory vs.         | *Laboratory*    | *Classroom*     |
| classroom*                       |                 |                 |
+----------------------------------+-----------------+-----------------+
| Khalili & Shashaani (1994)       | .47             | .27             |
|                                  |                 |                 |
| (jr high school)                 |                 |                 |
+----------------------------------+-----------------+-----------------+
| Khalili & Shashaani (1994)       | .23             | .57             |
|                                  |                 |                 |
| (high school)                    |                 |                 |
+----------------------------------+-----------------+-----------------+
| Kuchler (1999, secondary)        | .16             | .48             |
+----------------------------------+-----------------+-----------------+
| Lee (1990)                       | .38             | .69             |
+----------------------------------+-----------------+-----------------+
|                                  |                 |                 |
+----------------------------------+-----------------+-----------------+
| *Developer*                      | *Exp. /teacher* | *Commercial*    |
+----------------------------------+-----------------+-----------------+
| Kuchler (1999, secondary)        | .51^a^          | .36 ^a^         |
+----------------------------------+-----------------+-----------------+
| Lee (1990)                       | .48             | .34             |
+----------------------------------+-----------------+-----------------+
|                                  |                 |                 |
+----------------------------------+-----------------+-----------------+
| *Audience*                       | *Specific*      | *General*       |
+----------------------------------+-----------------+-----------------+
| Lee (1990)                       | .54             | .29             |
+----------------------------------+-----------------+-----------------+

#### Where's the fidelity?

*The following are Findings on Specific Categories of Instructional
Software\
*

**Drill and Practice**

-   Greater on computation, less on concepts/application

-   Positive on attitude

-   All grades, \> males, *only* category where substitution better

-   Recommended for automaticity

### Tutorials

-   Slightly greater on concepts and applications

-   Often most effective of CBI categories

-   All grades, \> higher grades, supplement

-   \> Researcher/teacher, specific audience

-   Recommended for new subject matter

### Tools

-   Tools and exploratory environments have inconsistent, but positive
    results.

-   Calculators: One of most important areas, still being analyzed and
    synthesized.

**Calculators (K-12)**

-   Ranges of effect sizes

    -   17 to .44

    -   -.21 to .72

    -   -.15 to .74

-   Pooled effect sizes

    -   .21

    -   .29

        -   Instruction with, testing *without* calculators

    ```{=html}
    <!-- -->
    ```
    -   Operational skills: .17

    -   Selectivity skills: .30

    -   Computational, conceptual, PS, all *ns*

        -   Testing (and instruction) *with* calculators

    ```{=html}
    <!-- -->
    ```
    -   Selectivity skills, *ns*

    -   *All* other areas, significant, .33 (PS) to .44 (conceptual)

        -   Attitudes positive

> (Ellington)

-   Testing *without* calculators

```{=html}
<!-- -->
```
-   Operational skills: .14, all grade levels *except* -.15 at grade 4

-   Problem solving: .20, lower for grades 4 and 7 (.08)

-   Computational, conceptual, PS, all *ns*

    -   Testing *with* calculators

```{=html}
<!-- -->
```
-   Operational skills, .33 low ability

-   Computation, .64 average ability

-   Problem solving, all .45

-   Selectivity subscale, .33 average

> (Hembree)

**Graphing Calculators (MS/HS)**

-   Testing *without* calculators

```{=html}
<!-- -->
```
-   Procedural: *-*.21, *ns*

-   Conceptual: .38

    -   Testing *with* calculators

```{=html}
<!-- -->
```
-   Procedural: .52

-   Conceptual: .72

### Computer Programming

-   Positive effects on achievement, greatest on concepts and
    applications, especially geometric

-   Positive effects on problem solving and attitudes, may be more than
    other CBI categories

-   \> average SES

-   Supplemental. Logo more effective

#### Features and Pedagogical Strategies

-   Too little on features

-   Pedagogy

    -   Working technology and support staff

    -   Adequate implementation fidelity, esp. time

    -   Integration with curriculum

    -   Formative assessment

    -   Social learning, especially collaboration, considered

    -   Teacher monitoring and mediation

## Report on Individual Differences---Gifted

-   Wide range of achievement in any age group

-   How to be responsive to these individual differences so that all
    students make progress and can achieve their potential

-   For the gifted, curriculum needs to be differentiated by level,
    complexity, breadth/depth, and pacing

### Four Ways to Differentiate

-   Enrichment

-   Acceleration

-   Homogeneous Grouping

-   Individualization

Amount of adjustment required depends upon level of giftedness

Acceleration and Enrichment

### Methodology

-   Extensive literature, including meta-analyses, but only 7-9 Studies
    met the rigorous methodological criteria

-   3 acceleration studies

-   2 self-paced learning

-   2 enrichment

-   1 combination

### Outcomes-1

-   No effect of acceleration on SAT-M scores g=.007 (N=2)

-   Accelerants took more elective math courses and more often majored
    in math g=.244, p=.05 (N=2)

-   Accelerants gained several years in their education, performed
    equally well or slightly better on host of variables but at an
    earlier age (N=3)

### Outcomes-2

-   Self-paced learning effective, g=.45 (N=2)

-   Self-paced learning plus enrichment even more effective, g=.65,
    p\<.001 (N=1)

-   Enrichment, mixed results (N=2)

### Tentative Conclusions

-   Increasing the pace and level of instruction for gifted youth is
    beneficial

-   Acceleration is effective

-   Enrichment might be a positive enhancement but by itself yields
    mixed results

-   Much research has been conducted but few individual studies meet
    methodologically rigorous criteria

## Report on Student-Centered vs. Teacher-Directed Instruction Update

-   Revised cooperative and peer-assisted learning

-   Main Finding: Team-Assisted Learning (TAI) effect size 0.340
    (p\<.01), computation, six studies.

-   Identified three experimental or quasi-experimental studies
    comparing SCI to TDI.

### Three Studies\--SCI vs. TDI

-   Hopkins and DeLisi (1997), 3rd and 5th grades, computation, sig
    effects are for girls only, favoring didactic (ES=1.14, p\<.001).

-   Muthukrishna and Borkowski (1995), 3rd grade, part-whole problem
    strategy, sig effects for far transfer of form only (ES=0.58,
    p\<.05).

-   Brenner e al. (1997), pre-algebra middle school, representing
    function problems (ES=0.88, p\<.05) but not sig. for getting correct
    answer.

### Tentative Conclusions

-   Research in its current state will not settle the great debate
    between SCI and TDI.

-   Effective practices that have been identified are
    situational---depends on context and outcome sought.

-   TDI often assumed to be control. Need more studies of
    teacher-directed instruction as treatment