Concept discovery from an image dataset
Toward image representation with an emergent language
Yuta Nakashima
Institute for Datability Science
Osaka University, Japan
June 28, 2023

Question

🤔

Does a model really see an image/video?

In visual question answering?

From: [Antol et al., “VQA: Visual question answering,” ICCV 2015]

Text and image

Text-only

In visual question answering?

🥺

From: [Agrawal et al., “Don’t Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering,” CVPR 2018]

In video question answering?

[Garcial et al., “KnowIT VQA: Answering Knowledge-Based Questions about Videos,” AAAI 2020]

Text and image

Text-only

In video question answering?

🥺
[Garcial et al., “KnowIT VQA: Answering Knowledge-Based Questions about Videos,” AAAI 2020]

In video moment retrieval?

[Otani et al., “Uncovering hidden challenges in query-based video moment retrieval,” BMVC 2020]

In video moment retrieval?

🥺
[Otani et al., “Uncovering hidden challenges in query-based video moment retrieval,” BMVC 2020]

Spurious correlation matters
Visual question answering

Video moment retrieval

From: [Agrawal et al., “Don’t Just Assume; Look and Answer:
Overcoming Priors for Visual Question Answering,” CVPR 2018][Otani et al., “Uncovering hidden challenges inquery-based video moment retrieval,” BMVC 2020]

A similar happens in visual-input-only tasks

From: [Beery et al., “Recognition in Terra Incognita” ECCV 2018]

Mitigating the spurious correlation problem• Considering certain attributes; for example:• [Burns et al., “Women also snowboard: Overcoming bias in
captioning models,” ECCV 2018]• [Agarwal et al., “Does data repair lead to fair models? Curatingcontextually fair data to reduce model bias,” WACV 2022]• Background matters; for example:• [Sagawa et al., “Distributionally robust neural networks for group
shifts: On the importance of regularization for worst-case
generalization,” ICLR 2020]• [Taghanaki et al., “Robust representation learning via perceptual
similarity metrics,” NeurIPS 2022]• Identifying confounders; for example:• [Liu et al., “Show, deconfound and tell: Image captioning with causal
inference,” CVPR 2022]

Yet another (perhaps crazy) idea?
• Representing an image/video with a set or a sequence of
discrete labels, but not with a continuous vector
Continuous rep.

Discrete rep. (concepts ＋combinations)h = [0.13, 0.54,
h = [0.13,0.0, 1.12,0.54,.…. . ] 0.0, 1.12, . .
<latexit sha1_base64="849jMcDQNfXgGy0bkb9wdxkWN8Y=">AAACknicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvoxuQmlmQkpVVn1CrYKkQb6Bka6yjoGuiZmoApAx0FQz1DIx2FmJT8kuLYeAFloBgYKGAyDKEMZQYoCMgXWM4Qw5DCkM+QzFDKkMuQypDHUAJk5zAkMhQDYTSDIYMBQwFQLJahGihWBGRlguVTGWoZuIB6S4GqUoEqEoGi2UAyHciLhormAfkgM4vBupOBtuQAcRFQpwKDqsFVg5UGnw1OGKw2eGnwB6dZ1WAzQG6pBNJJEL2pBfH8XRLB3wnqygXSJQwZCF143VzCkMZgAXZrJtDtBWARkC+SIfrLqqZ/DrYKUq1WM1hk8Bro/oUGNw0OA32QV/YleWlgatBsBi5gBBiiBzcmI8xIz9BMzyTQRNnBDBoVHAzSDEoMGsDwNmdwYPBgCGAIBdo7mWEnwyGGw0wiTFZMjkzOEKVMjFA9wgwogMkHAKwslkM=</latexit><latexit sha1_base64="849jMcDQNfXgGy0bkb9wdxkWN8Y=">AAACknicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvoxuQmlmQkpVVn1CrYKkQb6Bka6yjoGuiZmoApAx0FQz1DIx2FmJT8kuLYeAFloBgYKGAyDKEMZQYoCMgXWM4Qw5DCkM+QzFDKkMuQypDHUAJk5zAkMhQDYTSDIYMBQwFQLJahGihWBGRlguVTGWoZuIB6S4GqUoEqEoGi2UAyHciLhormAfkgM4vBupOBtuQAcRFQpwKDqsFVg5UGnw1OGKw2eGnwB6dZ1WAzQG6pBNJJEL2pBfH8XRLB3wnqygXSJQwZCF143VzCkMZgAXZrJtDtBWARkC+SIfrLqqZ/DrYKUq1WM1hk8Bro/oUGNw0OA32QV/YleWlgatBsBi5gBBiiBzcmI8xIz9BMzyTQRNnBDBoVHAzSDEoMGsDwNmdwYPBgCGAIBdo7mWEnwyGGw0wiTFZMjkzOEKVMjFA9wgwogMkHAKwslkM=</latexit>

h = [0.13, 0.54, 0.0, 1.12, . . . ]A skateboarder
jumping into the air<latexit sha1_base64="849jMcDQNfXgGy0bkb9wdxkWN8Y=">AAACknicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvoxuQmlmQkpVVn1CrYKkQb6Bka6yjoGuiZmoApAx0FQz1DIx2FmJT8kuLYeAFloBgYKGAyDKEMZQYoCMgXWM4Qw5DCkM+QzFDKkMuQypDHUAJk5zAkMhQDYTSDIYMBQwFQLJahGihWBGRlguVTGWoZuIB6S4GqUoEqEoGi2UAyHciLhormAfkgM4vBupOBtuQAcRFQpwKDqsFVg5UGnw1OGKw2eGnwB6dZ1WAzQG6pBNJJEL2pBfH8XRLB3wnqygXSJQwZCF143VzCkMZgAXZrJtDtBWARkC+SIfrLqqZ/DrYKUq1WM1hk8Bro/oUGNw0OA32QV/YleWlgatBsBi5gBBiiBzcmI8xIz9BMzyTQRNnBDBoVHAzSDEoMGsDwNmdwYPBgCGAIBdo7mWEnwyGGw0wiTFZMjkzOEKVMjFA9wgwogMkHAKwslkM=</latexit>a manskateboard

jumping

But not natural languageDiscovering concepts in an image dataset• SCOUTER [Li et al., ICCV 2021]• Better explainability with a SINGLE distinctive visual feature per class• BotCL [Wang et al., CVPR 2023]• Discovering concepts that describe image for a given classification
task

SCOUTER 🧐 [Li et al. ICCV 2021]•a01

2

SCOUTER

Backbone

Input

9

SCOUTER 🧐 [Li et al. ICCV 2021]

BotCL [Wang et al., CVPR 2023]

BotCL: Discovered concepts

BotCL: Evaluation of discovered concepts

BotCL: Interpretation of classifier<latexit sha1_base64="wCd8ZeSglq8UeRyPzTGBgn7O1eQ=">AAACe3ichVHLSgMxFD0dX7U+WhVEcDNYFFEpqVgtglBw49JXVdA6zIypBqczw0xaqdUf8AdcuFIQEf0LN/6ACz9BXCq4UfB2WhAt6g1JTk7uuTlJDNcSvmTsMaQ0Nbe0toXbIx2dXd3RWE/vmu8UPZNnTcdyvA1D97klbJ6VQlp8w/W4XjAsvm7sz1f310vc84Vjr8qyy3MFfdcWeWHqkigt1l/WhDqnHmhie0s6rirVcdXQhBaLswQLQm0EyTqIZ8IIYtGJXWELO3BgoogCOGxIwhZ0+NQ2kQSDS1wOFeI8QiLY5zhGhLRFyuKUoRO7T+MurTbrrE3rak0/UJt0ikXdI6WKYfbArtkLu2c37Im9/1qrEtSoeinTbNS03NWiJwMrb/+qCjRL7H2p/vQskUc68CrIuxsw1VuYNX3p8PRlZXZ5uDLCLtgz+T9nj+yObmCXXs3LJb58hgh9QPLnczeCtclEcjqRWpqKZ9K1n0AYgxjCKL33DDJYwCKydO4RLnCD29CHElfGlIlaqhKqa/rwLZTUJ9oJkf4=</latexit>• Our classifier is a single FC layer yi = wi> t + bi

• wi learns the correlations between each class and concept<latexit sha1_base64="5iQtup0nucWs8f6Bysksyc4ER0c=">AAACZnichVG7SgNBFD1ZXzE+EhVRsAkGxSrclfjAKmBjqYmJgkrYXcc4ZLO77G4iGvwBwVYLKwUR8TNs/AGL/IFiGcHGwptNQFTUO8zMmTP33Dkzozum9Hyiekjp6Ozq7gn3Rvr6BwajsaHhvGdXXEPkDNu03U1d84QpLZHzpW+KTccVWlk3xYZeWm7ub1SF60nbWvcPHbFT1oqW3JOG5jOVPSjIQixBSQoi/hOobZBIhxHEqh27wTZ2YcNABWUIWPAZm9DgcduCCoLD3A5qzLmMZLAvcIwIayucJThDY7bEY5FXW23W4nWzpheoDT7F5O6yMo4peqRbatAD3dEzvf9aqxbUaHo55FlvaYVTiJ6MZ9/+VZV59rH/qfrTs489LAZeJXt3AqZ5C6Olrx6dN7JLmanaNF3RC/u/pDrd8w2s6qtxvSYyF4jwB6jfn/snyM8m1fnk3FoqkU61fgJhTGASM/zeC0hjBavI8blFnOIM56EnZVAZVcZaqUqorRnBl1DiH1RKixc=</latexit><latexit sha1_base64="zqGtiap1argNypVovcA90di/Nv4=">AAACZHichVHLSgMxFD0dX7U+Wi2CIEixKK5KRuoDVwU3Llu1D6hFZsa0Dp0XM2mhFn9At4oLVwoi4me48Qdc+AOCuFRw48I704KoqDckOTm55+YkUR1D9wRjDyGpp7evfyA8GBkaHhmNxsbGC57dcDWe12zDdkuq4nFDt3he6MLgJcfliqkavKjW1/z9YpO7nm5bW6Ll8Iqp1Cy9qmuKICondmJJlmJBJH4CuQuSmTCCyNqxK2xjFzY0NGCCw4IgbECBR60MGQwOcRW0iXMJ6cE+xwEipG1QFqcMhdg6jTValbusRWu/pheoNTrFoO6SMoFZds+u2Qu7Yzfsib3/Wqsd1PC9tGhWO1ru7EQPJzff/lWZNAvsfar+9CxQxUrgVSfvTsD4t9A6+ub+6cvm6sZse45dsGfyf84e2C3dwGq+apc5vnGGCH2A/P25f4LCQkpeSi3m0slMuvMTCGMKM5in915GBuvIIk/nchzhGCehR2lYiksTnVQp1NXE8SWk6Q9yHYo4</latexit><latexit sha1_base64="5iQtup0nucWs8f6Bysksyc4ER0c=">AAACZnichVG7SgNBFD1ZXzE+EhVRsAkGxSrclfjAKmBjqYmJgkrYXcc4ZLO77G4iGvwBwVYLKwUR8TNs/AGL/IFiGcHGwptNQFTUO8zMmTP33Dkzozum9Hyiekjp6Ozq7gn3Rvr6BwajsaHhvGdXXEPkDNu03U1d84QpLZHzpW+KTccVWlk3xYZeWm7ub1SF60nbWvcPHbFT1oqW3JOG5jOVPSjIQixBSQoi/hOobZBIhxHEqh27wTZ2YcNABWUIWPAZm9DgcduCCoLD3A5qzLmMZLAvcIwIayucJThDY7bEY5FXW23W4nWzpheoDT7F5O6yMo4peqRbatAD3dEzvf9aqxbUaHo55FlvaYVTiJ6MZ9/+VZV59rH/qfrTs489LAZeJXt3AqZ5C6Olrx6dN7JLmanaNF3RC/u/pDrd8w2s6qtxvSYyF4jwB6jfn/snyM8m1fnk3FoqkU61fgJhTGASM/zeC0hjBavI8blFnOIM56EnZVAZVcZaqUqorRnBl1DiH1RKixc=</latexit>wit

Next steps• Efficient representation with a smaller set of concepts• More structured• Perhaps with grammar• Without target task
• Unsupervised (self-supervised) training for concept
discovery• Exploring some ways to identify spurious correlations• For vision and language tasks• For vision tasks