# Presentation: 312236

## An Implementation of Vicarious Learning with Deep-Level Reasoning Questions in Middle School and High School Classrooms 

**Barry Gholson, Art Graesser, and Scotty Craig**
**University of Memphis**

**Notes:**
I thought we send in a title, such as the one for cognitive science that we sent earlier.  

## Memphis Systems: K12 and College

**AutoTutor**

**iSTART**

**MetaTutor**

**ALEKS - math**

**Tutor Agent**

**ARIES**

**IDRIVE**

## Art Graesser (PI)
Zhiqiang Cai
Patrick Chipman
Scotty Craig
Don Franceschetti
Barry Gholson
Xiangen Hu
Tanner Jackson
Max Louwerse
Danielle McNamara
Andrew Olney 
Natalie Person
Vasile Rus

- Zhiqiang Cai
- Patrick Chipman
- Scotty Craig
- Don Franceschetti
- Barry Gholson
- Xiangen Hu
- Tanner Jackson
- Max Louwerse
- Danielle McNamara
- Andrew Olney
- Natalie Person
- Vasile Rus

**Learn by conversation in natural language**
- Graesser, A.C., Chipman, P., Haynes, B.C., & Olney, A. (2005).  AutoTutor: An intelligent tutoring system with mixed-initiative dialogue.  *IEEE Transactions in Education, 48*, 612-618.
- VanLehn, K., Graesser, A.C., Jackson, G.T., Jordan, P., Olney, A., & Rose, C.P. (2007).  When are tutorial dialogues more effective than reading?  *Cognitive Science, 31, *3-62.

-   **What is AutoTutor?**

## AutoTutor

**AutoTutor**

## LEARNING GAINS OF TUTORS(effect sizes) 

**  ****.42****	****	****Unskilled human tutors**
**	****	****	****(Cohen, Kulik, & Kulik, 1982)**
**  ****.80****	****	****AutoTutor (14 experiments)**
**	****	****	****(Graesser and colleagues)**
**1.00****	****	****Intelligent tutoring systems                     **
**	****	****	****PACT ****(Anderson, Corbett, Aleven, Koedinger)**
**	****	****	****Andes, Atlas ****(VanLehn)**
**	****	****	****Diagnoser ****(Hunt, Minstrell)**
**	****	****	****Sherlock**** (Lesgold)**
**(?)****	****	****Skilled human tutors **
**	****	****	****(Bloom, 1987)**

## Is an intelligent interactive tutor really needed?

**Vicarious Learning****. Perhaps observing a scripted dialogue can be just as effective.**
**Deep Questions****. Perhaps a dialogue organized around deep questions may be just as effective.**

**Notes:**
I am not sure what to do with this slide. I think it needs to be replaced.

## Why Vicarious Learning?

**Observation is an important learning method**
**Recall (Baker-Ward, Hess, & Flannagan, 1990) **
**Language (Akhtar et al., 2001, Huston & Wright, 1998) **
**Cultural norms (Ward, 1971; Metge, 1984) **
**Vicarious learning can be as effective as interactive learning.**
**Human tutoring if observers collaborate ****(Chi, Hausman, & Roy, in press; Craig, Vanlehn, & Chi, 2007)**
**Intelligent tutoring when guided by deep questions ****(Craig et al, 2006)**
**Provides a cost effective method that can easily be integrated into classrooms.**

## Facts about Deep Questions

- Students and teachers are not inclined to ask deep questions (Dillon, 1988; Graesser & Person, 1994).
- Training students to ask deep questions facilitates comprehension (Rosenshine, Meister & Chapman, 1996).
- Vicarious learning is effective when students observe animated conversational agents asking deep questions (Craig, Gholson, Ventura, & Graesser, 2000; Craig, et al., 2006; Gholson & Craig, 2006).

**Notes:**
I just do not think this slide works as a transition.

## Deep-level reasoning questions

- Deep-level reasoning question
- A question that facilitates logical, causal, or goal-oriented reasoning
- Example: Shallow vs. Deep questions
- What is a type of circulation? (shallow)
- What is required for Systemic Circulation to occur? (deep)

**Notes:**
The size, layout and aesthetics of the slide needs to be improved.  
SDC: How? I did what I could.

I don’t understand the meaning of the deep question.

## The Contest

-   Interactive computer tutor (Interactive condition)
-                          vs.
-   Vicarious learning from dialogue with deep reasoning questions (Dialogue condition)
-                          vs.
-   Monologue (Monologue condition)

## Q-Dialogue versus Monologue

-      Agent 1: The sun experiences a force of gravity due to the earth, which is equal in magnitude and opposite in direction to the force of gravity on the earth due to the sun.

**Agent 2:**** How does the earth's gravity affect the sun?**

**Agent 2:**** How does the gravitational force of the earth **
**    ****affect the sun?**

**Agent 1:**** The force of the earth on the sun will be equal and opposite to the force of the sun on the earth**

## Laboratory results with multiple choice dataCraig, Sullins, Witherspoon, & Gholson, (2006).  Cognition & Instruction.

- College students and computer literacy
- Three Conditions:
- Interactive (AutoTutor)
- Yoked vicarious (AutoTutor sessions)
- Q-Dialogue with deep questions

**Cohen’s d effect size**

## Memphis City School Study I

- Middle and high school students in two domains
- Computer literacy: Grades 8 & 10
- Physics: Grades 9 & 11
- Three Conditions:
- Interactive (AutoTutor)
- Dialogue (Monologue with deep questions)
- Monologue (AutoTutor Ideal Answers)

## Impact of condition as a function of prior knowledge Memphis City School Study I 

**Cohen’s d effect size**

**Notes:**
Great slide.  

Better title needed: Impact of condition as a function of prior knowledge.   

## Classroom Research

-   Standard classroom teaching
-                          vs.
-   Vicarious learning from dialogue with deep reasoning questions
-                          vs.
-   Monologue

## Overview of biology studyMemphis City School Study II

- 8th grade biology (circulatory system)
- Day 1
- Pretesting
- Gholson (multiple choice)
- Azevedo (matching, labeling, flow diagram, mental model shift)
- Days 2-6
- 30-35 minutes of vicarious dialogue, vicarious monologue, or standard classroom instruction
- 10 minutes to answer essay questions
- Day 7
- 15-20 minutes of vicarious or interactive review
- Day 8
- Posttests
- Gholson (multiple choice)
- Azevedo (matching, labeling, flow diagram, mental model shift)

## Azevedo and Gholson test resultsMemphis City School Study II

**Cohen’s d effect size**

- Mental model shift

**Notes:**
Pretest and Posttest showed significant learning gains in all conditions 
Labeling test showed differences among groups in favor of standard pedagogy.
Labeling - Shallow level identification task

## Daily essay questions Memphis City School Study II

**Cohen’s d effect size**

**Effect size compared to standard classroom**

**Dialogue **
**vs. standard **
**pedagogy**

**Monologue **
**vs. standard **
**pedagogy**

**Notes:**
Art, let me know what you think of this. Do you think this is confusing?

## Conclusions

**Vicarious learning is effective when students observe animated  conversational agents asking deep questions.**
**Deep-level reasoning questions effect replicates in computer literacy and Newtonian Physics (8****th****-11****th****).**
**Vicarious learning is most effective for learners with low domain knowledge.**
**Vicarious learning transfers to classroom settings for daily essays, but not for the primarily more shallow one day delayed tests.**

**Notes:**
Also, classroom shows advantages for shallow learning (labeling). -- This would be said on the last point, but I did not think we had space to put it on the slide.

## ?

## Memphis City School Study IIDesign

| Class | Format | Conditions |
| --- | --- | --- |
| 
1 | 
vicarious | Monologue |
|  |  | Dialogue |
| 
2 | 
vicarious | Monologue |
|  |  | Dialogue |
| 
3 | 
interactive | Regular classroom instruction |

**Notes:**
I’ll add color to make this easier to read.  I left it at 3 classes instead of 6 for simplicity.
This makes it look like we went into fewer classrooms. 

## Memphis City School Study II

**Using vicarious learning to teach course content at Snowden Middle School**
**8****th**** Graders**
**Our first foray into the circulatory system domain**

## Memphis City School Study IIMaterials

- Students in vicarious conditions observe the virtual tutoring session via laptop computer in the classroom
- Students in the interactive condition receive the regular classroom instruction** **
- 2 Pretests developed by
- Gholson (multiple choice)
- Azevedo (matching, labeling, flow diagram, mental model shift)
- 3 Posttests developed by
- Gholson & Azevedo (identical to pretest)

## Memphis City School Study IIProcedure

- Day 1
- Pretesting
- Days 2-6
- 30-35 minutes of vicarious or interactive instruction in the circulatory system
- 10 minutes to answer review questions after instruction
- Day 7
- 15-20 minutes of vicarious or interactive review
- Day 8
- Posttests (Gholson and Azevedo)

## Alternative Predictions

**1.  Interactive hypothesis:**
**     ****Interactive > Q-Dialog = Monolog**
**2.  Dialogic hypothesis:**
**     ****Interactive = Q-Dialog > Monolog**
**3.  Deep question hypothesis:**
**	****Q-Dialog > Interactive ****≥**** Monolog**

## Learning Conceptual Physics

- Four conditions:
- Read Nothing
- Read Textbook
- AutoTutor
- Human Tutor

## What are Deep-Level Reasoning Questions? 
(Graesser and Person,1994)

LEVEL 1: SIMPLE or SHALLOW 
1. Verification  		Is X true or false? Did an event occur?
2. Disjunctive 	  	Is X, Y, or Z the case?
3. Concept completion          Who? What? When? Where?
4. Example 	 	What is an example or instance of a category?).

LEVEL 2: INTERMEDIATE 
5. Feature specification 	What qualitative properties does entity X have?
6. Quantification 	                What is the value of a quantitative variable?  How much? 
6. Definition questions 	What does X mean?
8. Comparison		How is X similar to Y?  How is X different from Y?

LEVEL 3: COMPLEX or DEEP
9. Interpretation                     What concept/claim can be inferred from a pattern of data?
10. Causal antecedent	Why did an event occur? 
11. Causal consequence	What are the consequences of an event or state? 
12. Goal orientation               What are the motives or goals behind an agent’s action?
13. Instrumental/procedural What plan or instrument allows an agent to accomplish a goal? 
14. Enablement                      What object or resource allows an agent to accomplish a goal?
15. Expectation		Why did some expected event not occur?
16. Judgmental                      What value does the answerer place on an idea or advice?

- (Graesser and Person,1994)
- LEVEL 1: SIMPLE or SHALLOW
- 1. Verification  		Is X true or false? Did an event occur?
- 2. Disjunctive 	  	Is X, Y, or Z the case?
- 3. Concept completion          Who? What? When? Where?
- 4. Example 	 	What is an example or instance of a category?).
- LEVEL 2: INTERMEDIATE
- 5. Feature specification 	What qualitative properties does entity X have?
- 6. Quantification 	                What is the value of a quantitative variable?  How much?
- 6. Definition questions 	What does X mean?
- 8. Comparison		How is X similar to Y?  How is X different from Y?
- LEVEL 3: COMPLEX or DEEP
- 9. Interpretation                     What concept/claim can be inferred from a pattern of data?
- 10. Causal antecedent	Why did an event occur?
- 11. Causal consequence	What are the consequences of an event or state?
- 12. Goal orientation               What are the motives or goals behind an agent’s action?
- 13. Instrumental/procedural What plan or instrument allows an agent to accomplish a goal?
- 14. Enablement                      What object or resource allows an agent to accomplish a goal?
- 15. Expectation		Why did some expected event _not_ occur?
- 16. Judgmental                      What value does the answerer place on an idea or advice?

## Learning Environments with Agents developed at University of Memphis

| AutoTutor | Understanding science & technology  |
| --- | --- |
| MetaTutor | Learning how to learn and think |
| iSTART | Deep reading |
| SEEK  | True versus false information on the web |
| iDRIVE | Deep question asking and answering |
| HURAA  | Reasoning about research ethics |
| ARIES | Scientific reasoning |
| iMAP | Multi-channel communication |

## Memphis City School Study IResults - Overall

*Cohen’s d*

**Cohen’s d effect size**

**Notes:**
The Cohen’s d label on the y axis needs to be larger and bold, with better font (never italics).

Condition needs to be on the x-axis, not effect size.

## Other Collaborations with Agents  at University of Memphis

| iDRIVE | Question answering in science & technology | Gholson |
| --- | --- | --- |
| MetaTutor | Metacognition in science | Azevedo |
| iMAP | Multichannel commun ication with maps | Louwerse |
| SEEK | Critical stance while exploring web | Wiley, Goldman |
| ARIES | Critical Reasoning in science | Millis, Britt, Magliano, Wiemer-Hastings |

## Conclusions and summary

- Deep-level question effect - Deep-level question dialog improves learning over an interactive session, yoked vicarious session, & monolog session with same content
- (Craig, et al., 2006)
- Effect replicates in computer literacy and Newtonian Physics.
- Effect transfers to classroom settings

## Questions in Newtonian physics

-    The sun exerts a gravitational force on the earth as the earth moves in its orbit around the sun. Does the earth pull equally on the sun?  Explain why?

## Expectations and misconceptions in Sun & Earth problem

**EXPECTATIONS**
**The sun exerts a gravitational force on the earth. **
**The earth exerts a gravitational force on the sun. **
**The two forces are a third-law pair.**
**The magnitudes of the two forces are the same.**
**MISCONCEPTIONS**
**Only the larger object exerts a force. **
**The force of earth on sun is less than that of the sun on earth.**

## Misconceptions

| contact forces exerted after contact ceases |
| --- |
| vertical forces might have a non-zero horizontal component |
| heavier objects fall faster |
| heavier objects accelerate faster for the same non-gravitational force |
| air resistance non negligable |
| freefall means constant velocity  |
| lighter object exerts no force on a larger object |
| nonzero net force but no acceleration  |
| same force means same acceleration regardless of mass |
| action and reaction force acts on same body |
| 0 force implies slowing down |
| 0 force implies speeding up  |
| 0 force implies 0 velocity |
| (no autotutor equiv) 0 acceleration implies 0 velocity |
| action and reaction force do not have same magnitude |
| After an object is dropped or thrown the only force acting on it is gravity |
| Gravitational force acts *only* in the vertical direction |
| Inanimate object exerts no/less force in interaction  |
| Object that has been hit exerts no/less force in interaction  |
| Accelerations of both objects equal during interaction  |
| Only masses of part of compound body considered  |
| The force acting on a body is dependent on the mass of the body  |
| Action and reaction force have same directions  |
| Acceleration considered relative to accelerated reference frame  |
|  |

## Force equals mass times acceleration

|  | Pretest
Essay | Pretest
MC | Training | Posttest Essay | Posttest 
MC |
| --- | --- | --- | --- | --- | --- |
| All-or-none
Learning | X00X | X0XXXX0XXX XXX0XXX0XX
XX0X0X    
 | X0XX0X1XX1 | X1X1 | XX1XX1XXXX
X11XXXXXX1
XX1XXX |
| Variable Learning | X10X
 | X0XXXX0XXX XXX1XXX0XX
XX0X0X    
 | X0XX1X1XX0
 | X1X0
 | XX1XX0XXXX
X11XXXXXX0
XX1XXX
 |
| No   Learning | X00X
 | X0XXXX1XXX XXX0XXX0XX
XX1X0X    
 | X0XX0X1XX0 | X1X0 | XX0XX0XXXX
X10XXXXXX0
XX1XXX
 |
| Refresher  Learning
 | X00X
 | X0XXXX1XXX XXX1XXX1XX
XX1X1X    
 | X1XX1X1XX1 | X1X1 | XX1XX1XXXX
X11XXXXXX1
XX1XXX
 |

## Conceptual Physics(Graesser, Jackson, et al., 2003)

**Three conditions:**
**AutoTutor**
**Read textbook **
**Read nothing**

## Impact of Monolog  versus Dialog on recall and questions in a transfer task  (Craig, Gholson, Ventura, & Graesser, 2000)

- Learning about computer literacy with conversational agents.
- Monolog on computer literacy content
- Dialog with added deep questions
- Recall of content in training task
- Transfer tasks on new material
- Students instructed to generate questions about new computer literacy topics
- Recall of content of new material

## Impact of Dialog versus Monolog on recall and questions in a transfer task  (Craig, Gholson, Ventura, & Graesser, 2000)

## Managing One AutoTutor Turn

**Short feedback on the student’s previous turn**
**Advance the dialog by one or more dialog moves that are connected by discourse markers**
**End turn with a signal that transfers the floor to the student**
**Question**
**Prompting hand gesture**
**Head/gaze signal**

## Expectation and Misconception-Tailored Dialog: Pervasive in AutoTutor & human tutors 

**Tutor asks question that requires explanatory reasoning **
**Student answers with fragments of information, distributed over multiple turns**
**Tutor analyzes the fragments of the explanation**
***Compares to a list of expected good idea units***
***Compares to a list of expected errors and misconceptions***
**Tutor posts goals & performs dialog acts to improve explanation**
***Fills in missing expected good idea units (one at a time)***
***Corrects expected errors & misconceptions (immediately)***
**Tutor handles periodic sub-dialogues**
***Student questions***
***Student meta-communicative acts                                    (e.g., What did you say?)***

## Dialog Moves During Steps 2-4

**Positive immediate feedback: ****“Yeah”    “Right!”**
**Neutral immediate feedback: ****“Okay”  “Uh huh”**
**Negative immediate feedback: ****“No”  “Not quite”**
**Pump for more information: ****“What else?”**
**Hint: ****“What about the earth’s gravity?” **
**Prompt for specific information: ****“The earth exerts a gravitational force on what?”**
**Assert: ****“The earth exerts a gravitational force on the sun.”**
**Correct: ****“The smaller object also exerts a force. ”**
**Repeat: ****“So, once again, ...”  **
**Summarize: ****“So to recap,...”**** **
**Answer student question: **

## Procedure

**Gates-McGinitie reading test**
**& Pretest**

**Posttest**

**Interactive, Monologue, **
**or Dialogue instruction**

## Memphis City School Study(342 students)

- 	2 x 2 x 3 Design

| Age | Subject | Condition |  |  |
| --- | --- | --- | --- | --- |
|  |  | Dialogue | Monologue | Interactive |
| 
8th & 9th | Computer |  |  |  |
|  | Physics |  |  |  |
| 
10th & 11th | Computer |  |  |  |
|  | Physics |  |  |  |

## Multiple Choice Test Results

**Multiple Choice Test Results**
**Physics & Computer Literacy**

## How to cover a single expectation

**The earth exerts a gravitational force on the sun.**
**Who articulates it: student, tutor, or both?**
**Fuzzy production rules drive dialog moves**
**Progressive specificity drives dialog moves **
**	****Hint ******** Prompt ******** Assertion cycles**
**Strategies tailored to student knowledge and abilities**

## How does AutoTutor compare to comparison conditions on tests of deep comprehension?

**0.80** sigma compared to pretest, doing nothing, and reading the textbook
**0.22 **compared to reading relevant textbook segments
**0.07** compared to reading succinct script
**0.13** compared to AutoTutor delivering speech acts in print
**0.08** compared to humans in computer-mediated conversation
**-0.20 **compared to AutoTutor enhanced with interactive 3D simulation
**ZONE OF PROXIMAL DEVELOPMENT**

## Memphis City School Study IIVicarious Interface

## Memphis City School Study II

**Question:  How will the vicarious conditions perform next to interaction with a human teacher?**

**Notes:**
This is one route we could take to set up the results.