RGBDGaze: Gaze Tracking on Smartphones
with RGB and Depth Data

Riku Arakawa(adviser: Mayank Goel)

1

Background: gaze tracking technology

1930sPsychological study2

Background: gaze tracking technology

2017
Tobii Eye Tracker
Infrared sensor< 1cm error

3Background: gaze tracking application

Accessible technology

4Background: gaze tracking application

Human behavior analysis
Arakawa and Yakura [CHI’19]

5

Background: gaze tracking application

Attention-based interaction
Arakawa and Yakura [CHI’21]

6

Background: potentials of mobile gaze tracking●Mobile gaze tracking can support more everyday situations.
○

Inability to use touchscreen with encumbered hands○

Social interactions using smartphones○

Novel multimodal interactions between users and smartphones

7

Background: mobile gaze tracking applications in research

Enhance voice assistant with gaze
Mayer et al. [CHI’21]

8

Background: mobile gaze tracking applications in research

Combined use of gaze and hand gestures
Kong et al. [ICMI’21]

9

Background: why is mobile gaze tracking not common?
These mobile gaze tracking works require●

Per-user calibration●Constrained scenarios

10Background: insuﬃcient accuracy in mobile gaze tracking

11

Background: insuﬃcient accuracy in mobile gaze tracking

ARKit prediction range

12What’s needed for mobile gaze tracking?
●High accuracy●Calibration-free●

Capable of working in unconstrained situations

13

Related Work: eﬀectiveness of depth channel

The addition of the depth channel decreased the error by roughly 18%.
Depth provides● Precise head orientation● Distance from the screen to the head

14

Can we leverage depth sensor on recent mobile devices?
Depth cameras on mobiles

iPhone X

Google Pixel 4

15

Can we leverage depth sensor on recent mobile devices?
Depth cameras on mobiles

Desktop-grade depth cameras

iPhone X
Kinect 2

Google Pixel 4

16

Can we leverage depth sensor on recent mobile devices?
Depth cameras on mobiles

Desktop-grade depth cameras

iPhone X
Kinect 2

Google Pixel 4

Our intuition:
Even a coarse depth sensor can provide information of rough head orientationand distance between the screen to the phone in diverse use contexts.

17Data Collection●A dot target moving on the screen●Record synchronized data at 8 Hz○

RGB image○

Depth map○

ARKit prediction (used for evaluation)○(+ motion data from the IMU sensor)

18

Data Collection●7 x 5 locations●

Move from one to another location linearly

19

Data Collection●“Undo” function●

Tap screen to stop the animation and jumpback to the previous dot location

20

Data Collection: attention check●To ensure data reliability (e.g., preventing looking away)●

Participants need to take an action for each dot animation based on its color.

Do nothing

Tap right side

Tap left side21

Data Collection: use contexts

22Data Collection: summary●We did not control the environment.

●Images with blink are automatically detected and removed.

As a result,●●50 participants○

15 diﬀerent iPhones (>= iPhone X)○

14 wore glasses

160,120 samples across 4 use contexts

23

Implementation: model

24

Implementation: model

25

Implementation: model26

Result: all context● Leave-one-participant-out (50 participants)● Compare RGB model and RGB + D model27

Result: all context(p < 0.01)●Depth contributed to the error reduction by 16 %28Result: each context(p < 0.01)●

Use context aﬀects the performance.

●In every case, the depth helps the model.

29

Result: per-context-calibrated model(p < 0.01)●

Context-speciﬁc models perform better than a single “general” model.

●

Existing activity recognition techniques can boost the tracking performance.

30

Result: comparison with prior work
● Our dataset○

ARKit:○

iTracker:○

Our RGB model:○

Our RGBD model:

31

Result: comparison with prior work
iTracker (RGB)● Our dataset○

ARKit: 6.38 cm○

iTracker: 2.77 cm○

Our RGB model: 2.26 cm○

Our RGBD model: 1.89 cm

Ours (RGBD)

32

Result: comparison with prior work
● Our dataset○

ARKit: 6.38 cm○

iTracker: 2.77 cm○

Our RGB model: 2.26 cm○

Our RGBD model: 1.89 cm● GazeCapture dataset○

iTracker: 2.04 cm○

Our RGB model: 2.03 cm

33

Result: comparison with prior work
● Our dataset○

ARKit: 6.38 cm○

iTracker: 2.77 cm○

Our RGB model: 2.26 cm○

Our RGBD model: 1.89 cm● GazeCapture dataset○

iTracker: 2.04 cm○

Our RGB model: 2.03 cm

Our dataset has- Challenging capture scenarios- Larger screen size

34

Result: qualitative error analysis

35On-device Model●iPhone 12 Pro Max●RGBD Model○●

7 FPS (121.3 ms latency)

RGB Model○

10 FPS (85.3 ms latency)

36

Summary● We collected a dataset of mobile RGB + Depth (RGBD) gaze tracking.
○

50 participants○

4 use contexts: standing, walking, sitting, lying● Our RGBD model outperformed existing systems.
○Adding depth channel reduced the error by 16.3% (RGB: 2.26 cm, RGBD: 1.89 cm)● We developed the on-device system to enable real-time interactions.

37Limitation and Future Work
iTracker (RGB)● Further improvement of accuracy○Larger scale data collection○IMU sensor fusion

Ours (RGBD)
External Eye Tracker

38

Limitation and Future Work● Diverse data collection on crowd-sourcing○

More contexts: climbing stairs, biking, etc

GazeCapture dataset

39

RGBDGaze: Gaze Tracking on Smartphones
with RGB and Depth Data

Riku Arakawa
Dataset and source code: https://github.com/FIGLAB/RGBDGaze

40

Appendix41Background: gaze tracking technology

1960s
Chin rest

1990s
Wearable43

Implementation: training protocol●●●PyTorch 1.9.1
Batch Size: 16
SGD Optimizer
○
○
○

●
●
●

Initial learning rate: 5e-4
Momentum: 0.9
Weight decay: 1e-4

Loss function: mean squared error
20 epochs
Leave-one-participant-out evaluation○

12 hours x 50 participants

44

Result: comparison with prior work
● Our dataset○

ARKit: 6.38 cm○

iTracker: 2.77 cm○

Our RGB model: 2.26 cm○

Our RGBD model: 1.89 cm● Other mobile gaze tracking systems (Diﬀerent dataset)○

Tablet Gaze: 3.17 cm○EyeTab: 2.58 cm

45

Related Work46

Result47

Result48