# Presentation: 103227

## Building Students Metacognitive Skills through Interactions with Computer-based Teachable Agents

**Building Students Metacognitive Skills through Interactions with Computer-based Teachable Agents**

- _Gautam Biswas_
- _[gautam.biswas@vanderbilt.edu](mailto:gautam.biswas@vanderbilt.edu)_
*Dept of EECS & ISIS*
*Vanderbilt University*
- Collaborators: Dan Schwartz, Kefyn Catley
- Postdoc, Students (at Vanderbilt): Rod Roscoe, John Wagster, Hogyeong Jeong, Nancy Morabito, Jim Segedy, Garrett Linn
*Supported by Dept. of Education IES, and NSF REESE Awards*

## Goals of our work

- Learn Science through principles that apply across domains
- Processes, Entities, Relations, Interdependence, and Balance
- Preparation for Future Learning
- Students should become independent learners, even when they move away from the computer environment
- Learning for oneself – ability to assess one’s learning progress
- Learning is never a one step process
- Cognitive tasks and Metacognitive strategies

## Outline of Talk

- Our Approach to Learning by Teaching
- Betty’s Brain, a Teachable Agent
- Learning Science by creating Causal Concept Maps
- Assessment through self-other monitoring
- Adaptive Tutoring
- Providing Metacognitive support in support of Preparation for future learning
- Experimental Studies
- Current/Future Work

## Betty’s Brain

- Query

- Teach

- Quiz

**Additional resources**

- Mentor Agent
- 	Mr. Davis

- Text Resources

- Betty can explain her answers

## Teachable Agents

- Teachable Agents

- Students teach computer agent using visual representations
- Agent’s performance based on what she is taught
- Students re-teach agent so that they may do better, (and in that process they learn)

- Agents only learn what they are taught explicitly by student
- No machine learning algorithms drive our agent
- Learning through social interactions
- Shared representations
- Shared responsibility

## Learning Science

- By creating visual concept map structures
- Entities
- e.g., fish, macroinvertebrates, dissolved oxygen
- Relations
- causal: fish consume macroinvertebrates
- increase decrease effects
- Causal Reasoning
- Cause-effect relations extended to chain of events
- Fish  waste  bacteria  nutrients  algae
- Interdependence
- Multiple dependencies: everything depends on each other

**IES 2008: Adaptive Tutoring seminar**

## Metacognition to aid Learning 

- Metacognition to aid Learning

- Metacognition describes two component processes
- Ability to monitor one’s cognitive activities
- Ability to take appropriate regulatory steps when problem is detected
- Implemented as Self-regulated learning strategies
-  Involves multiple aspects when learning
- Setting goals
- Planning
- Seeking help
- Monitoring one’s own learning
- ....

## Monitoring when Problem Solving

- _Self monitoring_ (cf. to self explanation) requires two coordinated processes
- Ability to generate solution steps
- Analyze and correct for discrepancies
- Our approach: _Self-other monitoring _while teaching (& learning for oneself)
- Provide support to help student’s organize their own learning
- Betty: demonstrates self-regulated learning behaviors by example
- Mentor: provides additional support and hints

**IES 2008: Adaptive Tutoring seminar**

## Example regulation strategies

| Regulation
Goal | Pattern
Description | Betty’s response |
| --- | --- | --- |
| MONITORING BY 
ASKING QUERIES | Successive quiz requests but no queries asked of Betty in between quizzes
 | I’m still unsure of this material and I would like to do well. Mr. Davis said “take the quiz only if you think you will do well.”
(Betty refuses to take quiz) |
| MONITORING THROUGH 
EXPLANATIONS | Multiple requests for Betty to give an answer but no request for explanation | Let’s see, you have asked me a lot of questions, but you have not asked for my explanations lately. Please make me explain my answers so you will know if I really understand. |
| TRACKING PROGRESS | The most recent quiz score is significantly worse than the previous score | I would really like to do better. Please check the resources, teach me, and make sure I understand by asking me questions that are on the quiz. My explanation will help you find out why I am making mistakes in my answers. Also, be sure to check out the new tips from Mr. Davis. |

## Example regulation strategies

| Regulation
Goal | Pattern
Description | Mentor’s response |
| --- | --- | --- |
| MONITORING THROUGH 
EXPLANATIONS | Multiple requests for Betty to give an answer but no request for explanation | Without asking Betty to explain her answers, you may not know whether she really understands the chain of events that you have been trying to teach her.  Click on the Explain button to see if she explains her answer correctly. |
| TRACKING PROGRESS | The most recent quiz score is significantly worse than the previous score | Betty did well on the last quiz.  What happened this time?  Maybe you should try re-reading some of the resources and asking Betty more questions so that you can make sure she understands the material. |
| SETTING LEARNING GOALS
 | Betty is asked a question that she cannot answer for the second time | I’ve seen this kind of difficultly with teaching other students in the past.  You should look for missing links between concepts or links that are in the wrong direction. |

## Mentor: other forms of help

- On-Demand Help: Students select which kind of helps they need
*Pedagogical examples*
- “What should I teach Betty?”
- “How can I be sure that Betty learns what I have taught?”
*Learning examples*
-  “How do I know that I know enough to teach?”
*Domain-content examples *
- General – What domain content is relevant, chains of reasoning
- Specific: “I need help on the quiz.”
- Help after quiz taken: Adaptive
- ICS & LBT systems –  where errors have occurred in concept map and possible fixes
- SRL groups – what to read so as to do generate a more correct map

**Notes:**
Contrasting to the situation in ITSs where the tutor determines when students need help, in our environment, asking for help is a part of the learning process.  Students need to be able to evaluate their learning progress as the development of metacognitive strategies.  However, the mentor agent can refuse to give help at all or give help in a different category than that the user asks for.

## Experimental Studies

## Betty’s Brain: Experimental Studies

- Fifth-grade students teach and learn about river ecosystems in several 45-min. sessions, and complete written pre/post tests
- _Domain:_ River ecosystems: interdependence and balance involving: (i) Food Chain, (ii) Photosynthesis and Respiration, and (iii) Waste cycle
-  They later participate in a transfer (PFL) phase where they learn a new domain (e.g., nitrogen cycle on land, or global warming).
- We have compared several versions of the system:
**ICS** – create a map(no teach) with content feedback
**LBT** – teach Betty with content feedback
**T-SRL** – teach Betty with SRL feedback
**M-SRL** – create a map (no teaching) with SRL feedback

## Data Analysis

- Data Analysis

- Performance – learning of domain content
- Number of correct concepts + links in students’ final concept maps
- Behaviors – sequence of activities
- Key student actions are logged
-  Edit map (**EM**)
- Ask query (**AQ**)
- Request quiz (**RQ**)
- Access resources (**RA**)
- Request explanations (**RE**/**CE**)
- Betty could sometimes take (**QT**) or refuse (**QD**) the quiz

## Results – Learning Performance

- a T-SRL > ICS, *p* < .05; b T-SRL > LBT, *p *< .05; c LBT > ICS, *p* < .05.

- Study 1: ICS, LBT, and T-SRL (56 students)

| Condition | Mean (SD) Map Scores |  |
| --- | --- | --- |
|  | Main Phase | Transfer Phase |
| ICS | 22.83 (5.3) | 22.65 (13.7) |
| LBT | 25.65 (6.5)c | 31.81 (12.0) |
| T-SRL | 31.58 (6.6)a,b | 32.56 (9.9)a |

- Study 2: ICS, M-SRL, and T-SRL (83 students)

| Condition | Mean (SD) Map Scores |  |
| --- | --- | --- |
|  | Main Phase | Transfer Phase |
| ICS | 35.80 (10.5) | 36.56 (13.61) |
| M-SRL | 38.41 (8.55) | 39.66(16.41) |
| T-SRL | 41.79 (7.37)a | 42.97(15.83) |

## Behavior Analysis

- Roscoe, et al. (2008): ICS, LBT, and T-SRL in main study
- Map quality was associated with AQ and RE/CE activities
- AQ & RE/CE may indicate students’ attempts to regulate their own knowledge

## Behavior Analysis using HMMs

- Jeong, et al. (2008): ICS, LBT, and T-SRL in main and transfer
- Used hidden Markov models (HMMs) to model learning patterns
- States hidden, output observer
-  Three patterns related to SRL differed by condition
- _Map Building_: EM-RA-RQ
- _Map Probing_: AQ-RA
- _Map Tracing_: AQ-RE-CE
- Interpreted models  on right

**IES 2008: Adaptive Tutoring seminar**

## Behavior Analysis with HMMs

- Stationary probabilities show the likelihood of exhibiting a given state

**IES 2008: Adaptive Tutoring seminar**

| State | ICS | LBT | T-SRL | Transfer
ICS | Transfer LBT | Transfer 
T-SRL |
| --- | --- | --- | --- | --- | --- | --- |
| Map Building | 0.72 | 0.66 | 0.42 | 0.73 | 0.73 | 0.68 |
| Map Probing | 0.24 | 0.30 | 0.47 | 0.25 | 0.25 | 0.27 |
| Map Tracing | 0.04 | 0.04 | 0.11 | 0.02 | 0.02 | 0.05 |

## Pre-Post Test Analysis

- Detailed analyses of students’ written responses to examine learning of five river ecosystem principles
- balance, interdependence, microscopic entities, photosynthesis and cellular respiration, pollution
-  Learning about “microscopic” entities (e.g., oxygen, bacteria, and macroinvertebrates) was strongest
- Perhaps, because concept map representations make normally “invisible” concepts explicit.

**IES 2008: Adaptive Tutoring seminar**

## Current and Future Work

- Adaptive Tutoring through Interactive metacognition
- Betty emulates aspects of self-regulated learner
- Mentor provides additional metacognitive support to remind students of important cognitive learning tasks and to help organize these tasks
- Further study of self vs self-other monitoring
- Mentor SRL versus Betty SRL
- Increased dose of self-other monitoring
- Front-of-class (FOC) Betty
- Moving TA system into classroom – strong links to science curriculum
- Adaptive teaching by the classroom teacher(s)
- Learning science
- From concepts and their relations to causal reasoning about  chain of  events (interdependence)
- Aggregate Processes and Balance

- http://www.teachableagents.org