# Interpretation of semi-structured hand-drawn sketches for the conception of a Pen-Based Intelligent Tutorial System

<figure><img src="image_1.png" /></figure>

**Presented by:** Islam BARCHOUCH, PHD Student at INSA Rennes, Team SHADOC.

<figure><img src="image_2.png" /></figure>

**Supervisors:** Eric ANQUETIL, Prof. INSA Rennes, Team SHADOC.

Nathalie GIRARD, MC. Univ. Rennes, Team SHADOC.

**Contact :**

Islam.barchouch@irisa.fr

## 1. Context & Challenges

* ✓ **Learning by drawing** can improve the learning experiences for students [Fiorella+2015].

* ✓ **Real-time feedback** improves the students' ability to learn [Kulik+1998].

* ✓ Simulation of the pen-and-paper approach is possible on pen-based tablets.

<figure><img src="image_3.png" /><figcaption>Pen-based Tablets</figcaption></figure>

### Spine Exercise Example

<figure><img src="image_4.png" /><figcaption>First Step</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Middle Step</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Last Step</figcaption></figure>

Creating step-by-step anatomy exercises

### ITS for the field of Anatomy

<figure><img src="image_7.png" /></figure>

→ Need to recognize hand-drawn anatomical sketches

<figure><img src="image_8.png" /></figure>

→ Need to generate adaptive feedbacks to students

<figure><img src="image_9.png" /></figure>

## 2. Objectives & Fundamentals

<figure><img src="image_10.png" /></figure>

### IntuiSketch : Intelligent Tutorial System

Recognition Engine

Supervision Engine

Interpret and **recognize** the semantic and structure of **anatomical hand-drawn sketches** dynamically, in real time.

- Supervise and guide the student by providing feedbacks in **Student mode**.

- Enable the teacher to create exercises in **Author mode**.

## 3. IntuiSketch's Flexible Recognition Engine

<figure><img src="image_11.png" /></figure>

Students make **major** and **minor** errors.

<figure><img src="image_12.png" /></figure>

Minor errors **shouldn't** interrupt progress.

<figure><img src="image_13.png" /></figure>

### Primary Constraints (PC):

* ✓ **Recognition:** Essential for basic recognition

* ✓ **Feedback:** Immediate feedbacks

* ✓ **Learning Impact:** Foundational understanding

* ✓ **Examples:** Shape, positioning, basic geometric constraints

* ✓ **Tolerance:** Low

### Secondary Constraints (SC):

* ✓ **Recognition:** Not Essential (Evaluate detail)

* ✓ **Feedback:** Delayed qualitative feedbacks

* ✓ **Learning Impact:** Enhances skills

* ✓ **Examples:** Fine details constraints

* ✓ **Tolerance:** High

<figure><img src="image_14.png" /></figure>

## 4. IntuiSketch's Balanced Supervision Engine

<figure><img src="image_15.png" /></figure>

Categorisation into Immediate Feedbacks and Deferred Feedbacks.

**Immediate feedbacks:** Provided in real-time to correct major errors instantly and related to primary constraints.

**Delayed feedbacks:** Given at key stages to improve overall quality and related to secondary constraints.

<figure><img src="image_16.png" /></figure>

## 5. Experimental work (Test Protocol)

<figure><img src="image_17.png" /></figure>

* ✓ Collaborated with LP3C for year-long tests on a **Static Version IntuiSketch**.

1. 7 test companies on different anatomy exercises under different conditions were launched.

2. Students results were saved in **JSON files**.

3. Students work is tested on a **Test Version of IntuiSketch** after a **cleaning phase** to remove unusable traces.

4. Retrieve results in XML files and create ground truth XML files for each student's work.

5. Compare system results to ground truth to assess **tutor effectiveness**.