# Two-Way Gaze Sharing in Remote Teaching

Oleg Špakov, Diederick Niehorster, Howell Istance, Kari-Jouko Räihä, Harri Siirtola

<figure><img src="image_1.png" /></figure>

## To cite this version:

Oleg Špakov, Diederick Niehorster, Howell Istance, Kari-Jouko Räihä, Harri Siirtola. Two-Way Gaze Sharing in Remote Teaching. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.242-251, 10.1007/978-3-030-29384-0_16. hal-02544621

HAL Id: hal-02544621

https://inria.hal.science/hal-02544621

Submitted on 16 Apr 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Two-way gaze sharing in remote teaching

Oleg Špakov<fnref n="1" />, Diederick Niehorster<fnref n="2" />, Howell Istance<fnref n="1" />, Kari-Jouko Räihä<fnref n="1" />, and
Harri Siirtola<fnref n="1" />

**Abstract.** On-line teaching situations where a tutor and their students are remote from each other mean that contact between them is reduced compared with teaching in a classroom. We report an initial study of two-way gaze sharing between a tutor and a group of students, who were in different locations. A 45-minute class consisted of an introductory lecture followed by an exercise in using two software tools, one for building an experiment and the other for analysis of the data collected. The tutor went through an exercise step by step and the students followed. This was run twice with four students on each run. The tutor had a view of the students' desktops with their gaze markers overlaid and each student had a view of the tutor's desktop and gaze marker. Students found seeing the tutor's gaze marker helpful during the exercise but distracting when reading the text on the lecture slides. The tutor found the view of the students' gaze point helpful as an indicator of their current object of attention when giving assistance to individuals.

**Keywords:** gaze sharing · remote teaching · two-way

## 1 Introduction

A feature of classroom teaching is the ability of the tutor to gauge problems students have and progress made by them while listening to something being explained or when completing exercises. Observing cues from students who have difficulties can enable the tutor to help those students. Some students will ask for help if they need it, but others will not.

The possibility for tutors to observe students will be much reduced in remote internet-mediated teaching situations. This reduction in non-verbal contact can be offset by sharing an indication of a student's current object of visual attention with the tutor. There are also benefits in sharing where the tutor is currently looking with the students, as has been demonstrated in a classroom teaching situation [15]. It is very possible that the value of these benefits would be enhanced in a remote teaching situation.

If a remote eye tracker is to be used to measure where the student is looking, then the area of measurement is typically restricted to a screen in front of them. This screen can display a copy of the tutor's screen as well as the desktop of the student's own computer. Real-time remote teaching is facilitated

<footnote n="1">¹ Tampere University, Kalavantie 4, Tampere, Finland</footnote>

<footnote n="2">² Lund University, Paradisgatan 2, Lund, Sweden</footnote>
by bi-directional voice communication between the tutor and students. We can
characterize the teaching situations using the CSCW classification [9] as 'same
time - same place' (classroom) and 'same time - different place' (internet medi-
ated teaching). We are not considering here 'different place - different time' (as
would be the case with MOOCs for example).

The research question is twofold. First, to what extent does the tutor having access to information about where the student is looking help him or her to be aware of problems the student may have in understanding what is currently being taught? Second, to what extent does the student knowing where the tutor is looking in relation to the teaching material help the student?

This paper presents the outcomes from two remote teaching sessions. In each,
four students sat separated from each other in a digital classroom where all
computers were equipped with eye trackers. The tutor sat in a separate room
with two screens, one was their teaching screen and the other contained views of
each student's screen with the student's gaze point overlaid. Each student had
a window on their desktop containing a view of the tutor's teaching screen with
his gaze point overlaid. The contribution of this paper is that it is the first to
the knowledge of the authors to investigate the value of simultaneous two-way
gaze sharing in a remote teaching situation.

## 2 Background and Previous Work

The value of *gaze sharing* has been studied in several different contexts in addition to education. This means showing someone in real-time where someone else is looking. These contexts include remote expert assistance in problem solving [6, 1], collaborative visual search [2, 11, 13], remote pair programming [3] and games [10].

Educational applications of gaze sharing have used different numbers of shar-
ers of gaze.

*One-to-one.* Sharing the gaze of an expert (tutor) with a novice (student) has been found to be useful in teaching software debugging [14]. The gaze was shared continuously and not intentionally or deliberately (like a mouse pointer). It was found that the expert's gaze functioned as a useful cue and assisted the student in problem solving.

*One-to-many.* Replacing the tutor's pointer with a marker of the tutor's current gaze point has been compared to more traditional pointing methods [15]. The *GazeLaser* system was compared with the PowerPoint pointer and with a conventional laser pointer for manual pointing. GazeLaser performed on a par with the PowerPoint pointer, surpassing manual pointing. The PowerPoint pointer was rated highest by the participants, but GazeLaser was most easily noticed. In a study of gaze sharing in a MOOC [12], it was found that displaying the gaze point of the tutor on video teaching material made understanding what the tutor was referring to in complex situations easier for the student. Also, students stopped and replayed parts of the video less frequently compared with situations where the gaze overlay was absent. Similarly, eye-movement modelling
examples demonstrated the value of using an expert's gaze point to guide the attention of students looking at case videos in medical educational situations [5, 7, 8].

*Many-to-one*. A system for sharing the gaze information from students to a tutor's display during a computer science studio session was studied in [16]. On the teacher's display, each student's gaze was visualized with a colored circle having a 70 pixel radius. In addition, each student's location in the code editor was indicated with a color-coded line in the scroll bar. It was found out that the system was useful for getting confirmation that the students are following along, and monitoring the students while they were working independently. In addition, the teachers did not find the real time gaze visualization distracting as some of the earlier studies have suggested [4].

The study reported in this paper is effectively a combination of the one-to-
many and many-to one situations. The tutor's gaze point is presented to all
participating students and each of the student's displays containing their gaze
points is presented to the tutor.

3 Setting up a teaching situation with two-way gaze sharing

**3.1 Digital classroom**

The experiment took place in the Lund University Humanities Lab's Digital Classroom. In this facility, 25 computer stations are available that were each equipped with an SMI RED-m remote eye tracker and that are connected together through a gigabit Ethernet network. The stations in this setup were arranged in a large O, with all sitting on the outside of the O facing inward.

## **3.2 The gaze sharing solution**

The screen image and gaze marker position were shared separately from the
students' machines to the tutor's machine, and from the tutor's machine to each
of the student machines. The gaze marker was overlaid on the image of the
remote screen at the receiver's end.

*GazeNet*. The tool was built to share gaze data between multiple PCs. It consists of server and client components. The GazeNet Server<fnref n="3" /> runs as a NodeJS application on a local university server that interconnects GazeNet Client instances. All data streams are grouped into named channels, and all clients connected to a same channel exchange gaze data with each other.

The GazeNet Client<fnref n="4" /> is a local .NET application that connects to an eye tracker installed on the local PC (using ETU-Driver <fnref n="5" /> and UDPMultiCast <fnref n="6" />

<footnote n="3">³ Available at https://github.com/uta-gasp/gazenet-server</footnote>

<footnote n="4">⁴ Available at https://github.com/uta-gasp/gazenet-client</footnote>

<footnote n="5">⁵ Available at https://www.sis.uta.fi/ csolsp/downloads.php</footnote>

<footnote n="6">⁶ Available at https://github.com/dcnieho/UDPMulticast</footnote>
as middle-ware) and streams gaze data to the GazeNet Server using a named
channel defined in its settings. The tool can be configured to display the local
and/or remote gaze points on screen as semitransparent colored circles. GazeNet
Client is supplied with several plugins, one of which starts UltraVNC Server
and UltraVNC Viewer instances and scales remote gaze points to fit them into
corresponding UltraVNC Viewer windows.

*UltraVNC*. UltraVNC was chosen as a highly configurable and robust VNC solution to share the images of the desktops. The UltraVNC Server tool captures a PC desktop into a video-stream and delivers it to UltraVNC clients connected to it. At the endpoints of these streams, the UltraVNC Viewer tool was used as the VNC client, which was controlled by the GazeNet Client software.

*Skype*. All students and the tutor were connected to the group chat created in Skype. Only speech was transmitted over Skype, as the pilot tests revealed that video streams over Skype tend to delay or even halt regularly. The chat was the main communication channel between the participants in the session.

*Tutor's workstation*. This had two monitors, one showed the tutor's own desktop and the eye tracker tracked the gaze position on this monitor. The other monitor showed the desktops of the 4 students with their respective gaze positions overlaid. In each of these desktops the student's view of the tutor's desktop is shown in the top right corner. The tutor's gaze point on his or her own desktop is shown to the student in this view. It was not visible when the tutor was looking at the monitor with the students' desktops. The layouts are shown in Figure 1.

3.3 Class teaching exercise

The digital classroom was being used for a course on eye tracking. A lecture and class exercise, lasting 45 minutes and dealing with collecting and analyzing eye tracking data was devised and delivered twice to four course students on each occasion. The lecture lasted about seven minutes and used PowerPoint slides. Students were asked to maximize the window with the view of the tutor's screen during this. Following this, students were asked to return the window to its normal size. There then followed a demonstration of BeGaze (the data analysis tool) to visualize previously collected eye tracking data from a reading experiment. Then the use of Experiment Builder was demonstrated to set up a simple procedure for collecting reading data. Finally, the students collected their own reading data, and then used BeGaze to visualize and analyze this. In the practical exercise the tutor demonstrated and explained each step, and the students followed on their own computer. This was supported by a paper handout that the students could refer to during the exercise.

4 Outcomes and Evaluation

4.1 Student evaluation of shared gaze

Eight students participated in total, five female and three male. There were
two teaching sessions with four students in each. In the second session, two of
<figure><img src="image_3.png" /><figcaption>Fig. 1. Part of tutor’s view of the 4 student’s desktops</figcaption></figure>

the students had used the BeGaze analysis tool previously. Seven of the eight students answered an evaluation questionnaire after their session was over, which was followed by a discussion with all students in the session. One student left shortly before the end of their session by prior agreement and did not complete the questionnaire.

Some questions dealt with the usefulness of seeing the tutor’s gaze marker. Other questions dealt with how students felt about having their own eyes tracked and the tutor seeing their gaze position. The replies are summarized in Table 1.

## 4.2 Tutor evaluation of shared gaze

One tutor gave both lectures. They reported not looking at the students' desktops at all during the lecture part of the session. During the practical exercise, they reported using the screen containing the desktops in two ways. First looking at the desktops showed whether a student was keeping up with the exercise or whether they had encountered a problem and was falling behind. Second, when the tutor was helping a particular student, their gaze point provided a valuable indicator of the location of the student's attention. Each student's name appeared in a text box on the right hand side of their desktop, so the tutor could easily identify the student, and could address them directly.

It was seen quickly whether the student was looking at the object referred to by the tutor, or whether they had difficulties locating this. There was no need
<table><caption>**Table 1.** Summary of responses from student questionnaires and interviews</caption><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>how much attention was paid to the tutor's gaze point</td><td>6 reported following the tutor's gaze indicator 50% to 70% of the time of the session</td></tr><tr><td>situations where the tutor's gaze marker was helpful</td><td>3 said using the gaze marker to find where to find specific elements in the interface that the tutor referred to, especially when the instruction was not heard properly or was unclear.<br>2 used it to confirm they were looking at the same general area as the tutor.<br>2 said it was helpful</td></tr><tr><td>situations where the tutor's gaze marker was distracting</td><td>4 reported this was most of the time during the lecture slides: it was hard to listen, read slides and also see the gaze marker.<br>It was distracting as the gaze marker did not align with the tutor's mouse marker that was also displayed on the slide.<br>When the tutor's desktop window was restored to its normal size and location in the top right corner after the initial lecture, the marker was not distracting</td></tr><tr><td>whether the visual form of the gaze marker was appropriate</td><td>4 thought it was good, although 1 thought it obscured the text on the slides.</td></tr><tr><td>whether the tutor's gaze marker was sufficiently stable</td><td>3 thought it was, 2 thought it unstable part of the time, but 2 thought it was unstable and disturbing</td></tr><tr><td>how to make better use of the tutor's gaze marker</td><td>1 suggested having the option to replay the path of the marker.<br>2 suggested showing it only when the students were supposed to be doing something and not displaying it during the lecture part of the session.<br>1 said that there were times when they wanted more information but the normal size of the tutor's desktop window was too small</td></tr><tr><td>whether having their eyes tracked was disturbing</td><td>4 said no, 2 said only initially, and 1 said yes</td></tr><tr><td>whether having their eyes tracked caused fatigue</td><td>2 said no, but 5 reported some fatigue by the end of the session</td></tr><tr><td>whether having their eyes tracked caused them to behave differently</td><td>3 said no, other than trying to sit in the same position in front of the screen<br>4 said yes in some way they were more conscious of where they were looking. "I sometimes tried to make my gaze be where the teacher wanted at the moment, even if I already knew what to do."</td></tr></tbody></table>

to describe the absolute location if a student had problems, or to indicate the location by moving a mouse pointer to the object. The object could be referred to in relation to the gaze point, for example, 'No.. no.. up from where you're looking .. and right .. yes, there'.
## 4.3 Observations from the teaching sessions

An advantage of a setup where the content of the desktop can be manipulated by the students is that it allows them to work in their preferred fashion. Most students simply made use of the small view of the tutor's window, switching their gaze back and forth between their own work area and the tutor's window when following instructions. One student had problems setting up the experiment and fell a little behind. The tutor was able to get them back on track by individual tutoring a couple of times, but before that they had missed some of the verbal instructions. Therefore they decided to get a more detailed view of the tutor's desktop, maximizing and minimizing it ten times and spending nearly three minutes watching the large view without acting themselves. Their mode of operation was to make a plan and execute it after minimizing the tutor's view, even if it was not in synchrony with the verbal instructions.

# 5 Discussion

This paper has reported an initial study into two-way gaze sharing in a realistic remote teaching situation. A lecture and accompanying exercises to illustrate the collection and visualization of eye tracking data during a course were produced, and the class was repeated twice. One limitation was that only four students took part in each class. This was intentional to enable the views of all students' desktops to be displayed simultaneously to the tutor. The study was not repeated subsequently with larger numbers of students. Another limitation was that no analysis was made of objective measures indicating learning or teaching behaviour on the part of the students or the tutor, and there was no comparison of these indicators with and without shared gaze.

During the initial lecture part of the session, the view of the lecturer's desktop was shown maximized on the students' screens. The slides were shown full screen with the lecturer's gaze marker overlaid. Many students felt that seeing the lecturer's gaze marker was distracting, partly because it was new to them, but mostly because it interfered with their own reading of the text on the slides. Quoting one student, "Very distracting in the beginning! I could not attend to what [the tutor] was saying the first minutes, because I was too focused on following his eye movements and thinking about my own." However as most slides contained both text and illustrations, the gaze marker did work as an indicator for the students when to look at the pictures instead of the text. For this purpose it seemed to work well.

The two-way connection between the teacher and the students allowed the teacher to instruct individual students if they got stuck. For instance, one student was unable to follow the instructions concerning gaze replay, and asked a question using the audio channel. The tutor walked them through step by step and helped them catch up with the others. During this time the other students had to wait, but this is no different from what would happen in a traditional in-class tutoring session.
There are several issues to consider for a larger study of two-way gaze sharing. First, should students be given control of when they see the tutor's gaze marker? This would be in order to switch the marker off to prevent it being a distraction, such as when the tutor is looking at text on slides. It would be possible to detect automatically when the tutor is reading text on slides and suppress display of the tutor's gaze point on the student's view. Second, is the added value of gaze sharing worth the overhead of eye tracking in comparison with using deliberate movements of the mouse pointer to communicate the objects being attended to when the need arises? Third, what are the means of increasing the number of students being taught remotely from four to a more realistic number?

## 5.1 Increasing the number of participating students

The problem with presenting each student's desktop all the time is the amount of screen space required and the bandwidth to stream all desktops to the tutor's workstation. The display in this study (Figure 1) provides more information than the tutor has in a real classroom. One way to increase the number of students that a tutor can monitor concurrently could be to use another tutor. Their role would be to check several pages of tiled screens for students who appear to have difficulties. Another way is to enable students to request help, and then display only one student desktop and gaze marker to the tutor at a time. This is the equivalent of a student putting their hand up in class and the tutor coming to look over their shoulder. During this 1-to-1 help, voice communication could be restricted to the tutor and the individual student. A third way would be to detect automatically the distance between the tutor's marker and the student's marker and alert the tutor if the distance is consistently large.

In a classroom, the tutor may also look at students' faces to judge who might be having problems and ask if they need help. If the number of students is not large, Skype could be used in conference call mode to provide a view of the faces of all students. The tutor could select a student, and talk to them directly with only their desktop and marker being presented.

## 6 Conclusions

This initial study into two-way gaze sharing, in addition to voice, during remote teaching has shown that enabling the tutor to see where a student is looking on their desktop can facilitate helping that student. Little or no negotiation is needed to identify which objects are being referred to. Students reported finding that following the tutor's gaze point was very useful in completing the exercise but distracting when students were reading the text on slides. This work suggests that two-way gaze sharing can be a valuable asset for both tutor and student in a situation where students undertake exercises under tutor supervision. Further research in this area should focus on how these finding can be scaled onto a larger classroom.
# 7 Acknowledgments

This research was funded by the Academy of Finland, project Private and Shared Gaze: Enablers, Applications, Experiences (GaSP, grant number 2501287895)

# References

1. Akkil, D., James, J.M., Isokoski, P., Kangas, J.: GazeTorch: Enabling gaze awareness in collaborative physical tasks. In: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. pp. 1151–1158. ACM (2016)

2. Brennan, S.E., Chen, X., Dickinson, C.A., Neider, M.B., Zelinsky, G.J.: Coordinating cognition: The costs and benefits of shared gaze during collaborative search. Cognition **106**(3), 1465 – 1477 (2008), http://www.sciencedirect.com/science/article/pii/S0010027707001448

3. D'Angelo, S., Begel, A.: Improving communication between pair programmers using shared gaze awareness. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. pp. 6245–6290. ACM (2017)

4. D'Angelo, S., Gergle, D.: Gazed and confused: Understanding and designing shared gaze for remote collaboration. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. pp. 2492–2496. ACM (2016)

5. van Gog, T., Jarodzka, H., Scheiter, K., Gerjets, P., Paas, F.: Attention guidance during example study via the model’s eye movements. Computers in Human Behavior **25**(3), 785 – 791 (2009). https://doi.org/https://doi.org/10.1016/j.chb.2009.02.007

6. Gupta, K., Lee, G.A., Billinghurst, M.: Do you see what I see? The effect of gaze tracking on task space remote collaboration. IEEE Transactions on Visualization and Computer Graphics **22**(11), 2413–2422 (2016)

7. Jarodzka, H., Balslev, T., Holmqvist, K., Nyström, M., Scheiter, K., Gerjets, P., Eika, B.: Conveying clinical reasoning based on visual observation via eye-movement modelling examples. Instructional Science **40**(5), 813–827 (Sep 2012). https://doi.org/10.1007/s11251-012-9218-5

8. Jarodzka, H., van Gog, T., Dorr, M., Scheiter, K., Gerjets, P.: Learning to see: Guiding students’ attention via a model’s eye movements fosters learning. Learning and Instruction **25**, 62 – 70 (2013). https://doi.org/https://doi.org/10.1016/j.learninstruc.2012.11.004

9. Johansen, R.: Groupware: Computer support for business teams. The Free Press (1988)

10. Lankes, M., Maurer, B., Stiglbauer, B.: An eye for an eye: Gaze input in competitive online games and its effects on social presence. In: Proceedings of the 13th International Conference on Advances in Computer Entertainment Technology. pp. 17:1–17:9. ACE ’16, ACM, New York, NY, USA (2016). https://doi.org/10.1145/3001773.3001774, http://doi.acm.org/10.1145/3001773.3001774

11. Niehorster, D.C., Cornelissen, T., Holmqvist, K., Hooge, I.: Searching with and against each other: Spatiotemporal coordination of visual search behavior in collaborative and competitive settings. Attention, Perception, & Psychophysics **81**(3), 666–683 (Apr 2019). https://doi.org/10.3758/s13414-018-01640-0, https://doi.org/10.3758/s13414-018-01640-0
12. Sharma, K., Jermann, P., Dillenbourg, P.: Displaying teacher's gaze in a MOOC: Effects on students' video navigation patterns. In: Design for Teaching and Learning in a Networked World, pp. 325-338. Springer (2015)

13. Siirtola, H., Špakov, O., Istance, H., Räihä, K.J.: Shared gaze in collaborative visual search. International Journal of Human-Computer Interaction (2019), https://www.tandfonline.com/doi/abs/10.1080/10447318.2019.1565746

14. Stein, R., Brennan, S.E.: Another person's eye gaze as a cue in solving programming problems. In: Proceedings of the 6th International Conference on Multimodal Interfaces. pp. 9-15. ICMI '04, ACM, New York, NY, USA (2004), http://doi.acm.org/10.1145/1027933.1027936

15. Špakov, O., Siirtola, H., Istance, H., Räihä, K.J.: GazeLaser: A hands-free highlighting technique for presentations. In: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. pp. 2648-2654. CHI EA '16, ACM, New York, NY, USA (2016). https://doi.org/10.1145/2851581.2892504, http://doi.acm.org/10.1145/2851581.2892504

16. Yao, N., Brewer, J., D'Angelo, S., Horn, M., Gergle, D.: Visualizing gaze information from multiple students to support remote instruction. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. pp. LBW051:1-LBW051:6. CHI EA '18, ACM, New York, NY, USA (2018). https://doi.org/10.1145/3170427.3188453