Yao-Hung Hubert Tsaim https://yaohungt.github.io

H (412) 736-2674

k yaohungt@cs.cmu.edu

Research Summary
My research focuses on building intelligent agents, which leverage massive heterogeneous, noisy and imperfect information
to learn from, reason about and interact with the real world. Towards this goal, I work on topics in Deep Learning
and Statistical Machine Learning, particularly Multimodal Learning, Machine Reasoning, Transfer Learning and Lesssupervised Learning. Applications of my research span Core Machine Learning (CoreML), Computer Vision (CV),Natural Language Processing (NLP), and Speech Processing (SP).

Education
Carnegie Mellon University, Pittsburgh, PA, USA

Aug 2016 - Aug 2021 (Expected)• Ph.D. in Machine Learning Department within School of Computer Science (GPA: 4.08/4.3)
National Taiwan University, Taipei, Taiwan

Aug 2010 - Jun 2014

• B.S. in Electrical Engineering (graduated with Department Honors) (GPA: 4.17/4.3)• Presidential Awards in 2011/2012/2014, Undergraduate Ceremony Representative (top 10 students out of 230)

Research Experiences
Graduate Research Assistant, Carnegie Mellon University

Aug 2016 - PresentCoreML

Providing an information-theoretical explanation for the effectiveness of the self-supervised learned representations [ICLR’21A]. The explanation inspires new loss function designs [ICLR’21A, ICLR’21B].

CoreML

Tractable and scalable mutual information estimation [ICLR’21B, NeurIPS’20].

CV/NLP/SP

Analyzing human language from spoken text, facial attributes, and tones of voice [EMNLP’20, ACL’19, ICLR’19].

CV/NLP

Learning modality-invariant feature space across vision and text [ICCV’17, N-LLD’17].

CoreML

Deep neural networks regularization [NeurIPS’19] and Transformer architecture explanation [EMNLP’19].

•Advisor: Dr. Ruslan Salakhutdinov & Dr. Louis-Philippe Morency

Graduate Research Intern, Facebook AI ResearchSP•

Summer 2020

Improving and providing a better understanding of self-supervised speech representation learning. Worksinclude 1) incorporating temporal structures within representations, 2) unifying different loss function designs,and 3) introducing weak supervision from fast and cheap clustering [ICASSP’21].
Host: Dr. Abdelrahman Mohamed

Graduate Research Intern, Apple Inc.

Summer 2019

CV3D object recognition from multi-sensory data, spanning LiDAR, Radar, and RGB sensors.

CVImproving the scalability and stability of routing mechanism in Capsule networks [ICLR’20].

•

Host: Dr. Nitish Srivastava & Dr. Ruslan Salakhutdinov

Graduate Research Intern, Allen Institute for Artificial Intelligence
CV/NLP•Summer 2018

Video spatiotemporal relationships modeling for common sense retrieval [CVPR’19].
Host: Dr. Santosh Kumar Divvala & Dr. Ali Farhadi

Graduate Research Intern, Microsoft Research
CoreML•

Temporal order discovery in unordered dataset [N-TSW’17].
Host: Dr. Nebojsa Jojic

Visiting Scholar, RIKEN AIP/ Kyoto University
CoreML•

Summer 2017

Winter 2017, Winter 2018, Winter 2019

Optimal transportation, high-dimensional feature selection, semi-supervised mutual information estimation,and linear-time distribution divergence measurement [see https://yaohungt.github.io/research.html].
Host: Dr. Makoto Yamada & Dr. Masashi Sugiyama

Research Assistant, CITI, Academia Sinica
CV•

Domain Adaptation for Visual Object Recognition [CVPR’16, AAAI’16].
Host: Dr. Yu-Chiang Frank Wang

Aug 2015 - Aug 2016

Selected Publications (complete list at https://yaohungt.github.io/research.html)
ICLR’21A Y.-H. H. Tsai, Y. Wu, R. Salakhutdinov, L.-P. Morency. “Self-supervised Learning from a Multi-view Perspec-tive”, International Conference on Learning Representations, 2021.
ICLR’21B Y.-H. H. Tsai*, M. Q. Ma*, M. Yang, H. Zhao, L.-P. Morency, R. Salakhutdinov. “Self-supervised Represen-tation Learning with Relative Predictive Coding”, International Conference on Learning Representations, 2021.
ICASSP’21 W.-N. Hsu, Y.-H. H. Tsai, B. Bolte, R. Salakhutdinov, A. Mohamed. “HUBERT: How much can a bad teacher

benefit ASR pre-training?”, NeurIPS Self-Supervised Learning for Speech and Audio Processing Workshop, 2020.
NeurIPS’20 Y.-H.

H. Tsai, H. Zhao, M. Yamada, L.-P. Morency, R. Salakhutdinov. “Neural Methods for Point-wise
Dependency Estimation”, Neural Information Processing Systems, 2020. (Spotlight Presentation)

EMNLP’20 Y.-H. H. Tsai*, M. Q. Ma*, M. Yang*, R. Salakhutdinov, L.-P. Morency.

“Multimodal Routing: Improving
Local and Global Interpretability of Multimodal Language Analysis”, Empirical Methods in Natural Language
Processing, 2020.

ICLR’20 Y.-H. H. Tsai, N. Srivastava, H. Goh, R. Salakhutdinov.

“Capsules with Inverted Dot-Product Attention
Routing”, International Conference on Learning Representations, 2020.

NeurIPS’19 H. Zhao*, Y.-H. H. Tsai*, R. Salakhutdinov, G. Gordon. “Learning Neural Networks with Adaptive Regular-ization”, Neural Information Processing Systems, 2019.
EMNLP’19 Y.-H. H. Tsai, S. Bai, M. Yamada, L.-P. Morency, R. Salakhutdinov.

“Transformer Dissection: A Unified
Understanding of Transformer’s Attention via the Lens of Kernel”, Empirical Methods in Natural Language
Processing, 2019.

ACL’19 Y.-H. H. Tsai*, S. Bai*, P. P. Liang, J. Z. Kolter, L.-P. Morency, R. Salakhutdinov. “Multimodal Transformer

for Unaligned Multimodal Language Sequences”, Association for Computational Linguistics, 2019.
ICLR’19 Y.-H. H. Tsai*, P. P. Liang*, A. Zadeh, L.-P. Morency, R. Salakhutdinov.

“Learning Factorized Multimodal
Representations”, International Conference on Learning Representations, 2019.

CVPR’19 Y.-H. H. Tsai, S. K. Divvala, L.-P. Morency, R. Salakhutdinov, A. Farhadi.

“Video Relationship Reasoning
using Gated Spatio-Temporal Energy Graph”, Conference on Computer Vision and Pattern Recognition, 2019.

N-TSW’17 Y.-H. H. Tsai, H. Zhao, R. Salakhutdinov, N. Jojic. “Learning Markov Chain in Unordered Dataset”, NeurIPSTime Series Workshop, 2017. (Oral Presentation)
ICCV’17 Y.-H. H. Tsai, L.-K. Huang, R. Salakhutdinov. “Learning Robust Visual-Semantic Embeddings”, International

Conference on Computer Vision, 2017.
N-LLD’17 Y.-H. H. Tsai, R. Salakhutdinov. “Improving One-Shot Learning through Fusing Side Information”, NeurIPS

Learning with Limited Labeled Data: Weak Supervision and Beyond, 2017.
CVPR’16 Y.-H. H. Tsai, Y.-R. Yeh, Y.-C. F. Wang.

“Learning Cross-Domain Landmarks for Heterogeneous Domain
Adaptation”, Computer Vision and Pattern Recognition, 2016.

AAAI’16 Y.-H. H. Tsai, C.-A. Hou, W.-Y. Chen, Y.-R. Yeh, Y.-C. F. Wang. “Domain-Constraint Transfer Coding for

Imbalanced Unsupervised Domain Adaptation”, Association for the Advancement of Artificial Intelligence, 2016.

Selected Honors & Awards
Facebook Fellowship, Facebook
AI2 Fellowship, Allen Institute for Artificial Intelligence
Government Scholarship to Study Abroad (GSSA), Taiwan Ministry of Education
National Representative Honorable Mention, International Physics Olympiad Selection Camp
1st Runner-Up/Third Prize, Regional/National Physics Olympiad for Senior High School
Honorable Mention, International Junior Science Olympiad Selection Camp2020
2018
2016
2009/2010
2009
2008

Teaching Experiences & Professional Services
Teaching Assistant, Carnegie Mellon University

Fall 2017 and Spring 2019

• 10-707 Topics in Deep Learning (Instructor: Dr. Ruslan Salakhutdinov)• 10-716 Advanced Machine Learning / New Statistical Machine Learning (Instructor: Dr. Pradeep Ravikumar)
CMU Machine Learning Department: PhD Admission, Master Admission, Speaking Skills.
Conference and Journal Reviewer: ICML, NeurIPS, ICLR, ICCV, CVPR, AISTATS, ACL, EMNLP, TPAMI, TIP.
Student Advising: Denny Wu (BSc, CMU, 2018 → PhD, Toronto), Muqiao Yang (MSc, CMU, 2020 → PhD, CMU),
Martin Q. Ma (MSc, CMU, 2020 → PhD, CMU), Yue Wu (BSc, CMU, 2020 → PhD, CMU)