ACM CHI 2023 参加レポート
Doctoral Consortium / Case Study 編
筑波大学大学院 博士課程3年
矢倉 大夢 (Hiromu Yakura)
2023.05.22

自己紹介
•

所属: 筑波大学大学院 博士後期課程3年•

連携大学院制度で、研究は産業技術総合研究所
メディアインタラクション研究グループで実施•

2

興味分野: Human-Computer Interaction、機械学習•

Google / Microsoft Research の Ph.D Fellow に選出される•

ACM CHI '19 @ Glasgow より参加

目次

1. Doctoral Consortium / Case Study とは
2. 発表内容について
3. 採択までの流れ
4. 参加の様子

3Doctoral Consortium とは•博士課程の学生が研究発表を行い、メンタリングを受ける場•

ACM CHI に限らず幅広い国際会議で実施されており
SIGCHI 系でも C&C, CSCW, DIS, IUI, UIST などがある•

参加する学生に学びの機会を与えながらコミュニティを育て、拡げていくことが目的にある

4

Case Study とは•フルペーパーと別の、HCI の応用に主眼をおいたトラック•

新規性というよりも、実世界での応用やそこからの HCI コミュニティに共有すべき学びについて5目次

1. Doctoral Consortium / Case Study とは
2. 発表内容について
3. 採択までの流れ
4. 参加の様子

6A Generative Framework for Designing Interactions toOvercome the Gaps between Humans and Imperfect AIs
Instead of Improving the Accuracy of the AIs

Hiromu Yakura
University of Tsukuba, Japan
Google / Microsoft Research Ph.D. Fellow

Research context•

Research interest: ML + HCI (human-computer interaction)•How to apply machine learning in a human-centric manner?

ML techniques

TrustworthyInterpretable

Interaction
Creativity

Robustness
AI-basedsystems

7Humans
Productivity

Generate (non-software) Bugs
to Fool Classi ers
Hiromu Yakura*†, Youhei Akimoto*†, Jun Sakuma*†* University of Tsukuba, Japan† RIKEN Center for Articial Intelligence Project, Japanfifi

AAAI 2020

Background: Risk of abusing AI using adversarial example•

It is known that we can mislead ML models byintentionally adding a small noise to the inputs.
•In this case, the right image is classi ed as gibbon.

Using this adversarial example, we can abuse
AI-based systems without being noticed by humans.
Goodfellow, I. J., Shlens, J., & Szegedy, C.: Explaining and harnessing adversarial examples. In Proc. of ICLR. (2015)
fi9

Proposal: Attacking self-driving cars with moth-like stickers•This mechanism is also applicable to deceive self-driving cars.
•What if the cars recognize a STOP sign as Speed 80?
•This example can cause such a mistake
but looks too suspicious not to be noticed by humans.
[Chen+, '18]

Proposed

We showed that these moth-like stickers can mislead
ML models without making humans feel suspicious.
10S. Chen, et al. Shapeshifter: Robust physical adversarial attack on faster RCNN object detector. ECML PKDD 2018.

Failure-Resistant Intelligent Interaction

Basic idea: How to overcome the imperfection of AI models

No matter how much technical improvements we make,AI-based systems will make mistakes.

Design an interaction in which humans and AI-based systems
can collaborate effectively even when AIs make a mistake12

Mindless Attractor: A False-Positive ResistantIntervention for Drawing Attention
Using Auditory Perturbation
Riku Arakawa† and Hiromu Yakura‡(equal contribution)† The University of Tokyo, Japan‡ University of Tsukuba, JapanACM CHI 2021

Background: Limitation of alerting intervention•

ML models can detect the moment
when people are not engaging.
•14[Gupta+, '16]

It is possible to alert distracted students in video lectures.

•

But, misinformed alerts caused by false positives
disrupt the students, which leads them to distrust the system.

•What is the best way to intervene in distracted students
when we have the risk of false positive alerts?

A. Adams, et al. Mindless computing: designing technologies to subtly influence behavior. UbiComp (2015).

Demo: Mindless Attractor

15

Proposal: Mindless Attractor

Humans often intentionally or unconsciously change
the volume or pitch of speech to draw listeners' attention.

•Our system changes the volume or pitch for a moment todraw attention without consuming conscious awareness.
••

This is machine learning-friendly because it won't be
frustrating even when activated by false-positive detection.

In our study, we con rmed its e ectiveness to help refocus
even when the students are unconscious of the changes.

fffi16CatAlyst: Domain-Extensible Intervention
for Preventing Task Procrastination
Using Large Generative Models
Riku Arakawa†*, Hiromu Yakura‡*, and Masataka Goto¶(*equal contribution)† The University of Tokyo, Japan‡ University of Tsukuba, Japan¶ National Institute of Advanced Industrial Science and Technology, JapanACM CHI 2023

Background: Use generative models to avoid procrastination
Large language models not always perfect to alternate ourintellectual tasks (e.g., writing domain-speci c documents).
••

Is it possible to bene t from them in such cases?

We assumed that they can help people avoid procrastination.
•Even imperfect content generated can be used toguide users' interests to their tasks.

18fi

A. Adams, et al. Mindless computing: designing technologies to subtly influence behavior. UbiComp (2015).
fi•

Demo: CatAlyst

19

Proposal: CatAlyst
Generated contents that are not sufficient to alternate our tasks
can serve as a context-aware intervention to draw our interests.
•In addition, they sometimes provide inspirational ideas.

Hurdle forresumption
with conventional intervention(e.g., encouraging message)20

Evaluation: CatAlyst•

We evaluated its e ectiveness from various aspects:•

Ignorance rate: a rate of noti cations ignored•

Interest retrieval time: duration passed before resumption•Progress after resumption: progress made within 45 s afterresumption•Total time: time spent on completing the assigned task•

Subjective quality: product quality rated by crowdworkers•

Cognitive load: NASA-TLX score responded by participants
CatAlyst exhibited better performance thanconventional intervention, except for total timefiff21

Summary: Failure-Resistant Intelligent Interaction•

My research approach is:
1. Clarify the limitation of the current ML models
2. Find a situation in which even such models are bene cial•

For this purpose, I also conducted some qualitative research
focusing on new grassroots situations of using computers.

Speculation-driven new
tweeting behavior forTV drama series

Use of computer-mediatedcommunication by Japanese
idol groups under COVID-19fi22

Toward my thesis•

My thesis will extract design principles from these projects:
Q. What is the meta strategy to nd situations
in which even imperfect AIs are bene cial?
Q. How do the principles relate to other HCI concepts?
⁝•

Happy to hear your comments and questions!
hiromu.yakura@aist.go.jp

fifi

23@hiromu1996

🏅 Honorable Case Study Recognition【Case Study】

AI for human assessment:
What do professional assessors need?

Riku Arakawa†

Hiromu Yakura†

Carnegie Mellon University

University of Tsukuba / AIST

† Equal contribution
In collaboration with ACES Inc.
1

Human assessment

Evaluate candidates regarding their suitability for certain types of employment,
mostly through interviews by professional assessors
2

Human assessment
Human assessment typically consists of two phases:

1. Interview
phase

1. Interview phase• An assessor plays a certain role in an one-on-one interview.
• The conversation is video-recorded.

Employee

Converse

2. Review phase

Review• The assessor playbacks the recorded video.
• The video is used to nd verbal and nonverbal cues forAssessor

evaluating the employee as a manager.

3fi

2. ReviewphaseHuman assessment
Human assessment typically consists of two phases:

1. Interview
phase

1. Interview phase• An assessor plays a certain role in an one-on-one interview.
• The conversation is video-recorded.

EmployeeConverse

2. Review phase

Review• The assessor playbacks the recorded video.
• The video is used to nd verbal and nonverbal cues forAssessor

evaluating the employee as a manager.

4fi

2. Reviewphase

How can we support human assessment with AIs?
[1] Arakawa and Yakura, REsCUE: A framework for REal-time feedback on behavioral CUEs using multimodal
anomaly detection, CHI’19[2] Arakawa and Yakura, INWARD: A Computer-Supported Tool for Video-Re ection Improves E ciency andffiflff

E ectiveness in Executive Coaching, CHI’20

Initial workshop• We conducted a workshop with 2 professional assessors:• Di culties

1. Interview
phase

2. Review
phase

Employee• The review phase is time-consuming ⏳• Assessors’ subjectivity can lead to a wrong decision 🧐

Converse
Review

Assessorffi

6Initial workshop• We conducted a workshop with 2 professional assessors:• Di culties

1. Interview
phase
Employee
Analyze• The review phase is time-consuming ⏳• Assessors’ subjectivity can lead to a wrong decision 🧐

Converse↓

SupportReview

Q: How can AI systems support professional
assessors’ decision-making in review-phase?

7ffi

2. ReviewphaseAssessor

System requirements• The assessors were skeptical about AI-based end-to-end decision making
because human assessment should consider various factors speci c to each employee.
• They are highly human-contextual and di cult to be captured by computers.

fiffi

8

System requirements• The assessors were skeptical about AI-based end-to-end decision making
because human assessment should consider various factors speci c to each employee.
• They are highly human-contextual and di cult to be captured by computers.
• The assessors expected AI systems to help them not miss important behavior cues
due to their subjectivity or mental demands.
• Then, the assessors can revise their judgment by taking the contextual meaning ofsuch AI-detected cues into consideration.

Hypothesis: Separating observation (by AI) and judgment (by professionals)

fi

ffi

9

Feasibility study• We adopted nonverbal behavior analysis algorithm, REsCUE [1] used in executive coaching.
• It can extract anomalous cues of people in conversation.
• It provides clear visualization of the cues based on GMM.

Normal status

Exceptional responses

10Feasibility study• 20 interview videos• Two assessors annotated important scenes manually• Our algorithm also extracts anomaly scenes

1. Interview
phase

2. Review
phase

Employee
Analyze

Converse

Support
Review

Assessor

11

Findings• We examined the agreement between the algorithm and the assessors
and found that the algorithm does not completely replicate their annotation.

• The discrepancy was attributed to both false-positive detection and assessors' subjectivity.

12Findings• We examined the agreement between the algorithm and the assessors
and found that the algorithm does not completely replicate their annotation.

• The discrepancy was attributed to both false-positive detection and assessors' subjectivity.
• However, the assessors found that the algorithm would facilitate their assessment.
• The interpretable output of the anomaly-detection-based algorithm guided them toinfer the reason behind the detection, questioning their decisions.

• It helped maintain the assessors’ trust in the case of false-positives 🤝

A: The separation contributed to the trust in this highly contextual domain.
13

Usability study — Prototype
Browser-Based Prototype

Interview Video

Top 3 outliers

4th - 6th outliers
14

Usability study — Procedure• 6 professional assessors who had not participated in our rst study:• 2 junior assessors, 4 senior assessors• Each assessor reviewed randomly chosen four videos with the prototype.
• We conducted semi-structured interviews after they reviewed all videos
to ask about usability of the prototype.

fi15

Usability study — Result• Deepened quality of assessment• enhanced objectivity (← false-positive)• gain con dence (← true-positive)• not lose con dence (← false-negative)“rethought such cases but could easily resolve the con ict by referring to other signals”

flfi

fi

16

Usability study — Result• Deepened quality of assessment• enhanced objectivity (← false-positive)• gain con dence (← true-positive)• not lose con dence (← false-negative)“rethought such cases but could easily resolve the con ict by referring to other signals”

flfi

fi

17

Usability study — Result• Deepened quality of assessment• enhanced objectivity (← false-positive)• gain con dence (← true-positive)• not lose con dence (← false-negative)“rethought such cases but could easily resolve the con ict by referring to other signals”

flfi

fi

18

Usability study — Result• Room for improvement of the prototype• Potential use scenarios

Please refer to the paper!

19

Lessons learned• It is neither recommended nor feasible to train an AI model that replicates assessors’
decision-making process.

• Inevitable inconsistency among their processes(= di erent assessors look at di erent cues while having the same assessment result)• Lack of interpretability and validity in its output.
• Our design of separating observation and judgment is a promising approach in suchhighly contextual domains.

• Importantly, our goal is not replacing human decision, but helping them.

ff

ff

20

Lessons learned• It is neither recommended nor feasible to train an AI model that replicates assessors’
decision-making process.

• Inevitable inconsistency among their processes(= di erent assessors look at di erent cues while having the same assessment result)• Lack of interpretability and validity in its output.
• Our design of separating observation and judgment is a promising approach in suchhighly contextual domains.

• Importantly, our goal is not replacing human decision, but helping them.

ffff

21

目次

1. Doctoral Consortium / Case Study とは
2. 発表内容について
3. 採択までの流れ
4. 参加の様子

24Doctoral Consortium に採択されるまで••

以下の4つの書類を用意して、PCS で投稿•6ページの Research Statement•

DC で何を得たいか、どう DC に貢献できるかのエッセイ•

指導教員からの推薦状•

Curriculum Vitae (履歴書)

Research Statement については過去の参加者分がACM Digital Library で公開されているので参考にするとよい•

複数の研究をまとめている人から、1つの研究についてあるいはその計画のみの人までフェーズに応じて様々

25

Doctoral Consortium に採択されるまで•2人の査読があり、それを元にして参加可否が決まる•

内容のクオリティに加えて博士課程として有効活用できそうか参加タイミングの適切性も見られる•

CHI '23 では115件の提出があり20件がオンライン開催枠に10件がオフライン開催枠に採択された

26

Case Study に採択されるまで•

フルペーパーと似たような形で査読がある•

シングルカラムで 4-10p と短めなのと査読者が2名であることが大きな違い•

共著者の荒川と一緒に、1年ほど前から戦略的に投稿を準備•

ACM CHI '19 で発表した手法の実応用を進める中でせっかくであればそこでの発見を共有できればと考えた•

ただ、技術的新規性が大きい訳では無いということもありどうしようかと考える中で Case Study に思い至る

27

Case Study に採択されるまで•

ACM CHI '22 のワークショップで In progress 版を発表•

Case Study のスコープに合わせて改訂の上で投稿•

Lessons Learned というセクションを追加してHCI コミュニティへの知見という点の接続性を追加•
28

HCI の応用領域としての可能性をより強調目次

1. Doctoral Consortium / Case Study とは
2. 発表内容について
3. 採択までの流れ
4. 参加の様子

29Doctoral Consortium に参加するまで•

事前にメンバーリストが届き、ピアメンタリングがスタート•

アメリカ 7名, イギリス 4名, フランス・ドイツ 2名に加えカナダ・イタリア・日本・南アフリカ・ルクセンブルク•

提出の内容だけでなく、地理的多様性を考慮して選んだ（らしい）

30

Doctoral Consortium に参加するまで•

事前にメンバーリストが届き、ピアメンタリングがスタート•

1人にメンターになってもらい、別の1人のメンターになる•••プレゼンをシェアして Zoom で練習＆フィードバック

当日時点で知っている人が2人いるというのがありがたい

また、Committee の先生がメンターとしてアサインされて事前コミュニケーションできる•

University College London の Prof. Brumby が担当してくれることに

31Doctoral Consortium の参加の様子•

前日に参加者ディナーがあり、そこからスタート•

当日は朝から晩まで
インテンシブなスケジュール•

各自が10分プレゼン＋10分質疑•

そのあとはメンターの先生を囲みながら3人くらいでディスカッションタイム

•32

ランチ中も Ask Us Anything

Doctoral Consortium の参加の様子•

質疑＆フィードバックもかなりしっかりしたやりとり••

Prof. Burnett: 研究者の採用面接だと思ってやります

プレゼンのフォントサイズからストーリー構成まで研究室内の発表練習かのようにコメントをもらえる•

Q. あなたの HCI に対する貢献を1文でいうとなんですか？
（参加者の間では賛否が分かれていたが個人的には芯を食ったいい質問だと思う）＋•
33

期間中もポスターセッションでの発表の機会を貰えるDoctoral Consortium に参加してよかったこと•

なによりもいろんな人からフィードバックをもらえる••

改めて HCI の分野の幅広さを体感できる••

VR や AI から色覚異常の支援や自画撮り防止なども

CHI コミュニティの中に近い世代の知り合いが増える•

34

Neurodiversity とつなげるといいと思う、など

DC 後もご飯に行ったり、パーティーで何時間も話したりCase Study の参加の様子•

フルペーパーと同じようにパラレルセッションの1つとして3-4本の発表が行われる•

発表したセッションではGoogle と MicrosoftからのCase Study があった•

Google による UX Designer がAI ツールをどう活用できるか調べた発表の注釈が面白かった

35

まとめ
•

Doctoral Consortium はいいぞ！
•

実績よりも教育やコミュニティ形成に焦点をおいており日本からの投稿は通りやすそう（？）•

いろんな研究アプローチの人と交流し、仲良くなることで学べることや発見も大きい•Case Study もおすすめ•

フルペーパーと毛色は違うが、Lessons Learned があればしっかり評価してもらえる•36

聞きたいことや見たい資料などあればhiromu.yakura@aist.go.jp

気軽に連絡ください！

@hiromu1996