Instructor
Name

Yusuke Sugano,
Associate Professor

Laboratory
Location

Institute ofIndustrial ScienceResearch
Area

Computer
Vision, HCI

In our laboratory, we engage in research activities that span various fields, including computer vision, machinelearning, and human-computer interaction. With the recent advancements in large-scale vision-language models,there is a growing expectation that a single high-performance foundation model could solve many challenges.
However, there remains a significant gap between the actual performance and task diversity and the demands ofthe real world. Aiming for high performance on existing benchmarks alone will not suffice to address and solve
the myriad of challenges people face. We believe it is crucial to approach these issues from a broad perspective,
tackling the diverse problems between people and machines/AI, including user interaction. By designing andevaluating intelligent systems incorporating user interaction, we aim to realize intelligent systems that are more
open to people and society. We welcome students with a wide range of knowledge and curiosity across various
research fields and are eager to tackle new themes with a playful spirit.

Human-in-the-loop Computer Vision and Machine Learning
In the real-world application of systems founded on image recognition and machine learning, it often proves
inadequate to meet users' needs solely through pre-trained models and tailored prompts. The development of
mechanisms for naturally acquiring training data to adapt to environments and people, interaction methods tocompensate for incomplete recognition results, and user interfaces for users to train their own recognition
models becomes pivotal in designing the interaction between users and machine learning models. We are actively
addressing these challenges through the creation of systems and user evaluation, including the design of adaptive
machine learning algorithms, visualization techniques, and interfaces to realize interactive machine learning
environments.

Interactive Machine Learning

Gamified System for Training Data Collection

Vision-based User Understanding and Gaze Estimation

Training Data Synthesis via Face Reconstruction

"#$%

"#$%

…

%#$%

$) %!"!

Fuser

# !"!

# #$%

(,)

(,)

%!"!

(,)

%#$%

(')

# #$%

…

Gaze
Estimator

(,-')

(,-')

%#$%

(,-')
$%#$%

Gaze
Estimator

(')

(,-')

%!"!

Fuser

…

(')

%!"!

RotGaze
Estimator

Fuser(')# !"!

Rot$) %!"*"!"!

…

Gaze
Estimator(+)(+)%#$%(+)
$%#$%

Fuser

Rot(+)%!"!

Rot

Rotatable Feature
Extractor"!"!

Rotatable Feature
Extractor

Backbone
Extractor!#$%

Rotation-constrained
Feature Fusion Block 'Rotation-constrained
Feature Fusion Block 1"!"!

Shared Weights!!"!

Backbone
Extractor

Recognizing human states, such as gaze and behavior, has emerged as a key application area of computer
vision. We are actively engaged in human sensing and user understanding using computer vision and machinelearning technologies. For instance, by inferring where a person is looking within an environment, we canfacilitate various applications such as estimating human internal states related to attention and providing
adaptable information presentation based on human attention. Traditional gaze estimation methods, which
typically necessitate specialized hardware, have been constrained in their application scope. However, we arepioneering gaze estimation methods that only require camera images as input, leveraging large-scale training
datasets and machine learning.

(,)

"#$%

Multi-view Appearance-based Gaze Estimation

Please feel free to contact Yusuke Sugano <sugano@iis.u-tokyo.ac.jp> for questions and inquiries.
http://ivi.iis.u-tokyo.ac.jp/