Learning Vision without Visual Data
Phillip Isola, MIT
LIMIT Workshop at CVPR
June 2024
Work with Manel Baradad, Jonas Wulff, Tongzhou Wang, Chun-Fu Chen, Rogerio Feris, Pratyusha Sharma,
Tamar Rott Shaham, Stephanie Fu, Adrián Rodríguez-Muñoz, Shivam Duggal, Antonio Torralba

N=0
Phillip Isola, MIT
LIMIT Workshop at CVPR
June 2024
Work with Manel Baradad, Jonas Wulff, Tongzhou Wang, Chun-Fu Chen, Rogerio Feris, Pratyusha Sharma,
Tamar Rott Shaham, Stephanie Fu, Adrián Rodríguez-Muñoz, Shivam Duggal, Antonio Torralba

Learning to See
Training datann<latexit sha1_base64="02ggfyILRRLkqGm5VO3XB99zGJA=">AAACFnicjVC7SgNBFL3rM8ZX1NJmMAhWYVcELYM2lgrmgckSZid3kyGzs8vMrBCW/IWFjb9iI2Irdv6Ns8kWmlh4YOBwzr3cOSdIBNfGdb+cpeWV1bX10kZ5c2t7Z7eyt9/UcaoYNlgsYtUOqEbBJTYMNwLbiUIaBQJbwegq91sPqDSP5Z0ZJ+hHdCB5yBk1VrrvRtQMgzAbT3qVqltzpyCLxCtIFQr8b7xX+ez2Y5ZGKA0TVOuO5ybGz6gynAmclLupxoSyER1gx1JJI9R+No01IcdW6ZMwVvZJQ6bqz42MRlqPo8BO5jH0vJeLf3md1IQXfsZlkhqUbHYoTAUxMck7In2ukBkxtoQyxe1fCRtSRZmxTZZtdG8+6CJpntY8t+bdnlXrl0VnJTiEIzgBD86hDtdwAw1gIOERnuHVeXJenDfnfTa65BQ7B/ALzsc38pqYlg==</latexit>
sha1_base64="w2SeowxjSmKQZ+WPP4dmP+ztLQA=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhNp0NqjW37s5BVolXkBoUaA6qX/1hzNKIK2SSGtPz3AT9jGoUTPJZpZ8anlA2oSPes1TRiBs/myeekTOrDEkYa/sUkrn6eyOjkTHTKLCTeUKz7OXif14vxfDaz4RKUuSKLT4KU0kwJvn5ZCg0ZyinllCmhc1K2JhqytCWVLEleMsnr5L2Rd1z6979Za1xU9RRhhM4hXPw4AoacAdNaAEDBc/wCm+OcV6cd+djMVpyip1j+APn8wf+vpEd</latexit>

o

,

Fish

,

Grizzly,

Chameleon

…

n

y

x

o
o

!
<latexit sha1_base64="sVBkjs/c+hJlwPgmxP0/MoyXMvk=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPSUDbbTbt0kw27E6WE/gwvHhTx6q/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdkBouRcJbKFDybqo5jUPJO+H4duZ3Hrk2QiUPOEl5ENNhIiLBKFrJ72kxHCHVWj31qzW37s5BVolXkBoUaParX72BYlnME2SSGuN7bopBTjUKJvm00ssMTykb0yH3LU1ozE2Qz0+ekjOrDEiktK0EyVz9PZHT2JhJHNrOmOLILHsz8T/PzzC6DnKRpBnyhC0WRZkkqMjsfzIQmjOUE0so08LeStiIasrQplSxIXjLL6+S9kXdc+ve/WWtcVPEUYYTOIVz8OAKGnAHTWgBAwXP8ApvDjovzrvzsWgtOcXMMfyB8/kDwruRjQ==</latexit><latexit

Learner! f :X!Y<latexit sha1_base64="sVBkjs/c+hJlwPgmxP0/MoyXMvk=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPSUDbbTbt0kw27E6WE/gwvHhTx6q/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdkBouRcJbKFDybqo5jUPJO+H4duZ3Hrk2QiUPOEl5ENNhIiLBKFrJ72kxHCHVWj31qzW37s5BVolXkBoUaParX72BYlnME2SSGuN7bopBTjUKJvm00ssMTykb0yH3LU1ozE2Qz0+ekjOrDEiktK0EyVz9PZHT2JhJHNrOmOLILHsz8T/PzzC6DnKRpBnyhC0WRZkkqMjsfzIQmjOUE0so08LeStiIasrQplSxIXjLL6+S9kXdc+ve/WWtcVPEUYYTOIVz8OAKGnAHTWgBAwXP8ApvDjovzrvzsWgtOcXMMfyB8/kDwruRjQ==</latexit><latexit<latexit sha1_base64="HYPSq+DctLdg9hrnUlynFDqraoE=">AAACTnicfZDfShtBFMZn0z/GaGu0l70ZjIKUEnZFULyS1oveFBWMRrIhnJ2c3QzOziwzZ1vDklfprX2c3vZFeic6iSm0Kh4Y+PGdb2bO+ZJCSUdh+DuovXj56vVCfbGxtPzm7Upzde3MmdIK7AijjO0m4FBJjR2SpLBbWIQ8UXieXH6e9s+/oXXS6FMaF9jPIdMylQLIS4PmWrrPuzy2MhsRWGu+84tBsxW2w1nxxxDNocXmdTxYDTbioRFljpqEAud6UVhQvwJLUiicNOLSYQHiEjLsedSQo+tXs+EnfNMrQ54a648mPlP/vVFB7tw4T7wzBxq5h72p+FSvV1K616+kLkpCLe4/SkvFyfBpEnwoLQpSYw8grPSzcjECC4J8Xo1GfIh+GYtf/cNHBVogYz9UMdgsh6uJXy6LP07pOaPUf42efK7RwxQfw9l2Owrb0clO6+DTPOE6e8/W2RaL2C47YF/YMeswwa7YD3bNfga/gj/BTXB7b60F8zvv2H9Vq98BlhezeQ==</latexit>

Learning to See without Data?
Training datann<latexit sha1_base64="02ggfyILRRLkqGm5VO3XB99zGJA=">AAACFnicjVC7SgNBFL3rM8ZX1NJmMAhWYVcELYM2lgrmgckSZid3kyGzs8vMrBCW/IWFjb9iI2Irdv6Ns8kWmlh4YOBwzr3cOSdIBNfGdb+cpeWV1bX10kZ5c2t7Z7eyt9/UcaoYNlgsYtUOqEbBJTYMNwLbiUIaBQJbwegq91sPqDSP5Z0ZJ+hHdCB5yBk1VrrvRtQMgzAbT3qVqltzpyCLxCtIFQr8b7xX+ez2Y5ZGKA0TVOuO5ybGz6gynAmclLupxoSyER1gx1JJI9R+No01IcdW6ZMwVvZJQ6bqz42MRlqPo8BO5jH0vJeLf3md1IQXfsZlkhqUbHYoTAUxMck7In2ukBkxtoQyxe1fCRtSRZmxTZZtdG8+6CJpntY8t+bdnlXrl0VnJTiEIzgBD86hDtdwAw1gIOERnuHVeXJenDfnfTa65BQ7B/ALzsc38pqYlg==</latexit>
sha1_base64="w2SeowxjSmKQZ+WPP4dmP+ztLQA=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhNp0NqjW37s5BVolXkBoUaA6qX/1hzNKIK2SSGtPz3AT9jGoUTPJZpZ8anlA2oSPes1TRiBs/myeekTOrDEkYa/sUkrn6eyOjkTHTKLCTeUKz7OXif14vxfDaz4RKUuSKLT4KU0kwJvn5ZCg0ZyinllCmhc1K2JhqytCWVLEleMsnr5L2Rd1z6979Za1xU9RRhhM4hXPw4AoacAdNaAEDBc/wCm+OcV6cd+djMVpyip1j+APn8wf+vpEd</latexit>

o

,

Fish

,

Grizzly,

Chameleon

…

n

y

x

o
o

!
<latexit sha1_base64="sVBkjs/c+hJlwPgmxP0/MoyXMvk=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPSUDbbTbt0kw27E6WE/gwvHhTx6q/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdkBouRcJbKFDybqo5jUPJO+H4duZ3Hrk2QiUPOEl5ENNhIiLBKFrJ72kxHCHVWj31qzW37s5BVolXkBoUaParX72BYlnME2SSGuN7bopBTjUKJvm00ssMTykb0yH3LU1ozE2Qz0+ekjOrDEiktK0EyVz9PZHT2JhJHNrOmOLILHsz8T/PzzC6DnKRpBnyhC0WRZkkqMjsfzIQmjOUE0so08LeStiIasrQplSxIXjLL6+S9kXdc+ve/WWtcVPEUYYTOIVz8OAKGnAHTWgBAwXP8ApvDjovzrvzsWgtOcXMMfyB8/kDwruRjQ==</latexit><latexit

Learner! f :X!Y<latexit sha1_base64="sVBkjs/c+hJlwPgmxP0/MoyXMvk=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPSUDbbTbt0kw27E6WE/gwvHhTx6q/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdkBouRcJbKFDybqo5jUPJO+H4duZ3Hrk2QiUPOEl5ENNhIiLBKFrJ72kxHCHVWj31qzW37s5BVolXkBoUaParX72BYlnME2SSGuN7bopBTjUKJvm00ssMTykb0yH3LU1ozE2Qz0+ekjOrDEiktK0EyVz9PZHT2JhJHNrOmOLILHsz8T/PzzC6DnKRpBnyhC0WRZkkqMjsfzIQmjOUE0so08LeStiIasrQplSxIXjLL6+S9kXdc+ve/WWtcVPEUYYTOIVz8OAKGnAHTWgBAwXP8ApvDjovzrvzsWgtOcXMMfyB8/kDwruRjQ==</latexit><latexit<latexit sha1_base64="HYPSq+DctLdg9hrnUlynFDqraoE=">AAACTnicfZDfShtBFMZn0z/GaGu0l70ZjIKUEnZFULyS1oveFBWMRrIhnJ2c3QzOziwzZ1vDklfprX2c3vZFeic6iSm0Kh4Y+PGdb2bO+ZJCSUdh+DuovXj56vVCfbGxtPzm7Upzde3MmdIK7AijjO0m4FBJjR2SpLBbWIQ8UXieXH6e9s+/oXXS6FMaF9jPIdMylQLIS4PmWrrPuzy2MhsRWGu+84tBsxW2w1nxxxDNocXmdTxYDTbioRFljpqEAud6UVhQvwJLUiicNOLSYQHiEjLsedSQo+tXs+EnfNMrQ54a648mPlP/vVFB7tw4T7wzBxq5h72p+FSvV1K616+kLkpCLe4/SkvFyfBpEnwoLQpSYw8grPSzcjECC4J8Xo1GfIh+GYtf/cNHBVogYz9UMdgsh6uJXy6LP07pOaPUf42efK7RwxQfw9l2Owrb0clO6+DTPOE6e8/W2RaL2C47YF/YMeswwa7YD3bNfga/gj/BTXB7b60F8zvv2H9Vq98BlhezeQ==</latexit>

What counts as visual data?
Not visual data

Visual data

She sent out for one of those short, plump little cakes called'petites madeleines,' which look as though they had beenmoulded in the fluted scallop of a pilgrim's shell. And soon,mechanically, weary after a dull day with the prospect of adepressing morrow, I raised to my lips a spoonful of the tea inwhich I had soaked a morsel of the cake.

PhotosLanguagex ~ N(0,1)
Stable Diffusion

CGI

Noise
Code

Learning Vision from “Noise”
x ~ N(0,1)

Retinal Waves
Spontaneous activity inthe prenatal retina

May help self-organize
developing vision system
Willshaw & Von Der Malsburg, 1976
Ackman, Burbridge, Crair, 2012
Retinal waves prime visual motion detection by simulating future optic flow
Ge, Zhang, Gribizis, et al. Science 2021

Learning to See by Looking at Noise

Manel Baradad*, Jonas Wulff*, Tongzhou Wang,
Phillip Isola, Antonio Torralba
NeurIPS 2021

See also: [“Pre-training without Natural Images” Kataoka et al., ACCV 2020]

Evaluation: How well does the representation linearly classify ImageNet100?

40
30
20
10

Natural Photos (Places)<latexit sha1_base64="6Ti+/2yet3GYo8fKSUbiWGmAzwM=">AAACDXicjVC7SgNBFL0bXzG+opY2g0GwCrsixDJoY2nAPCBZwuzkbjJkdnaZmRXCki+wsPFXbERs7e38GyfJFppYeGDgcM653LknSATXxnW/nMLa+sbmVnG7tLO7t39QPjxq6ThVDJssFrHqBFSj4BKbhhuBnUQhjQKB7WB8M/PbD6g0j+W9mSToR3QoecgZNVZqhP1yxa26c5BV4uWkAjn+F++XP3uDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtVTSCLWfza+ZkjOrDEgYK/ukIXP150RGI60nUWCTETUjvezNxL+8bmrCKz/jMkkNSrZYFKaCmJjMqiEDrpAZMbGEMsXtXwkbUUWZsQWW7One8qGrpHVR9dyq17is1K/zzopwAqdwDh7UoA63cAdNYIDwCM/w6jw5L86b876IFpx85hh+wfn4Bi8jlGM=</latexit>
sha1_base64="b4HLEbhr7TEtaehBb4ygFYyuiV8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipGQ7KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8AyvGM6g==</latexit>?

50

Rand initfc

How good isClassification accuracy (%)

60

Nearest
neighbors
CNN-RandNearest
neighbors
CNN-RandAccuracy = 4.28%

Let’s design some noise!

Remarkable property of natural images

Fourier
Transform

A simple statistical image model

1/(fxa + fyb)

FFT

IFFT
a,b ~ Uniform[0.5, 3.5]cf<latexit sha1_base64="6Ti+/2yet3GYo8fKSUbiWGmAzwM=">AAACDXicjVC7SgNBFL0bXzG+opY2g0GwCrsixDJoY2nAPCBZwuzkbjJkdnaZmRXCki+wsPFXbERs7e38GyfJFppYeGDgcM653LknSATXxnW/nMLa+sbmVnG7tLO7t39QPjxq6ThVDJssFrHqBFSj4BKbhhuBnUQhjQKB7WB8M/PbD6g0j+W9mSToR3QoecgZNVZqhP1yxa26c5BV4uWkAjn+F++XP3uDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtVTSCLWfza+ZkjOrDEgYK/ukIXP150RGI60nUWCTETUjvezNxL+8bmrCKz/jMkkNSrZYFKaCmJjMqiEDrpAZMbGEMsXtXwkbUUWZsQWW7One8qGrpHVR9dyq17is1K/zzopwAqdwDh7UoA63cAdNYIDwCM/w6jw5L86b876IFpx85hh+wfn4Bi8jlGM=</latexit>
sha1_base64="b4HLEbhr7TEtaehBb4ygFYyuiV8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipGQ7KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8AyvGM6g==</latexit>

Same
image?
cf<latexit sha1_base64="6Ti+/2yet3GYo8fKSUbiWGmAzwM=">AAACDXicjVC7SgNBFL0bXzG+opY2g0GwCrsixDJoY2nAPCBZwuzkbjJkdnaZmRXCki+wsPFXbERs7e38GyfJFppYeGDgcM653LknSATXxnW/nMLa+sbmVnG7tLO7t39QPjxq6ThVDJssFrHqBFSj4BKbhhuBnUQhjQKB7WB8M/PbD6g0j+W9mSToR3QoecgZNVZqhP1yxa26c5BV4uWkAjn+F++XP3uDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtVTSCLWfza+ZkjOrDEgYK/ukIXP150RGI60nUWCTETUjvezNxL+8bmrCKz/jMkkNSrZYFKaCmJjMqiEDrpAZMbGEMsXtXwkbUUWZsQWW7One8qGrpHVR9dyq17is1K/zzopwAqdwDh7UoA63cAdNYIDwCM/w6jw5L86b876IFpx85hh+wfn4Bi8jlGM=</latexit>
sha1_base64="b4HLEbhr7TEtaehBb4ygFYyuiV8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipGQ7KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8AyvGM6g==</latexit>

Classification accuracy (%)
10

30

20

Natural Photos (Places)

Spectrum

Rand init

60

50
Baselines

40
Statistical

Classification accuracy (%)
10

20

30

Natural Photos (Places)

Spectrum + Color

Spectrum

Rand init

60

50
Baselines

40
Statistical

Dead leaves
Sample position ~ uniform
Sample color ~ uniform
Sample size ~ exponential(λ)
Position on top of the image
Repeat until all image covered

Daniel L. Ruderman. Origins of scaling in natural images. Vision Research, 1997.

Classification accuracy (%)
10

30

20

Natural Photos (Places)

Dead leaves - Shapes

Dead leaves - Oriented

Spectrum + Color

Dead leaves - Squares

Spectrum

Rand init

60

50
Baselines

40
Statistical
Dead leaves

StyleGAN v2
For each sample, re-init the network withrandom weights[StyleGANv2: Karras et al. 2019]

Classification accuracy (%)
10

20

30

Natural Photos (Places)

StyleGAN - Oriented

Dead leaves - Shapes

Dead leaves - Oriented

Spectrum + Color

Dead leaves - Squares

StyleGAN - Rand

Spectrum

Rand init

60

50
Baselines

40
Statistical

Dead leaves

Untrained StyleGAN

Nearest
neighbors
Random StyleGAN

Nearest
neighbors
Random StyleGAN
Accuracy = 33%

Learning Vision from Code

What’s the shortest program that yields “vision”

Procedural Image Programs for Representation Learning
Manel Baradad, Chun-Fu Chen, Jonas Wulff, Tongzhou
Wang, Rogerio Feris, Antonio Torralba, Phillip Isola
NeurIPS 2022

• Downloaded ~21k shaders from TwiGL and Shadertoy• Human artists made each• 100s to 1000s of characters of code eachClassification accuracy (%)
10

20

30

Natural Photos (Places)

Shaders

StyleGAN - Oriented

Dead leaves - Shapes

Dead leaves - Oriented

Spectrum + Color

Dead leaves - Squares

StyleGAN - Rand

Spectrum

Rand init

60

50
Baselines

40
Statistical

Dead leaves

Untrained StyleGAN
Shaders

Nearest
neighbors
Shaders-21k
MixUp

Accuracy = 43%

Nearest
neighbors
Shaders-21k
MixUp

Accuracy = 43%

What’s the shortest program that yields “vision”!
<latexit sha1_base64="sVBkjs/c+hJlwPgmxP0/MoyXMvk=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPSUDbbTbt0kw27E6WE/gwvHhTx6q/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdkBouRcJbKFDybqo5jUPJO+H4duZ3Hrk2QiUPOEl5ENNhIiLBKFrJ72kxHCHVWj31qzW37s5BVolXkBoUaParX72BYlnME2SSGuN7bopBTjUKJvm00ssMTykb0yH3LU1ozE2Qz0+ekjOrDEiktK0EyVz9PZHT2JhJHNrOmOLILHsz8T/PzzC6DnKRpBnyhC0WRZkkqMjsfzIQmjOUE0so08LeStiIasrQplSxIXjLL6+S9kXdc+ve/WWtcVPEUYYTOIVz8OAKGnAHTWgBAwXP8ApvDjovzrvzsWgtOcXMMfyB8/kDwruRjQ==</latexit><latexit

Code: 2140 characters
https://www.shadertoy.com/view/ltVfWG

36% top-1 acc on
ImageNet100

Learning Vision from Language
She sent out for one of those short, plump little cakes called'petites madeleines,' which look as though they had beenmoulded in the fluted scallop of a pilgrim's shell. And soon,mechanically, weary after a dull day with the prospect of adepressing morrow, I raised to my lips a spoonful of the tea
in which I had soaked a morsel of the cake.

A Vision Check-up for Language Models

Pratyusha Sharma*, Tamar Rott Shaham*, Manel Baradad, Stephanie Fu,
Adrián Rodríguez-Muñoz, Shivam Duggal, Phillip Isola and Antonio Torralba

Can language models draw?

Can language models see?

Large
Language
ModelCan language models draw?
“Return code inprocessing to
draw men herding
sheep.”

fill(0);

LLM

circle(150,150,100,100);
rectangle(250,150,100,100);. . .

RenderCan language models draw?

Can language models see?

Large
LanguageModelHuman drawing dataset

fill(0);circle(150,150,100,100);
rectangle(250,150,100,100);. . .

Human drawing dataset

Can language models see?
“What does the following
code represent?:
fill(0);ellipse(150,150,100,100);
ellipse(250,150,100,100);. . .

Human Drawing“”

A lake, sky, terrace, traffic
lights or a sandy beach?

Lake
Lake
LLM
Sky
Sky
Terrace
Terrace
TrafficLights
LightsTraffic
SandyBeach
Beach
Sandy

Objects

ScenesDavinci

0.136

0.221

GPT-3.5

0.136

0.380

GPT-4

0.234

0.212

Chance0.2

0.2

Models correctly classify human drawings over the chance baseline (although barely!)
Models fail to recognize concept categories that they can contrarily render well!

What do the models get wrong?

True: Terrace
Predicted: Staircase

True: Cake
Predicted: Beach

Can we train a vision model from language models?

Large
Language
Model

VisionModelCan we train a vision model from language models?
(a) Synthetic pre-training: Unsupervised
Contrastive Representation Learning(b) Evaluate: Real Images}

ResNet 50

ResNet 50
k-Nearest Neighbors“Anemone Fish”

Nearest
neighbors
LLM trained
Accuracy = 27.4%

the ways by which different neural networks repWe argue
that representations
inacrossAImodels,par-are converging. First, we
ticularly
deep
networks,
demonstrate
convergencedata
modalities:
Neural networks, trainedwith different objectives
possibleselective pressures
toward it.
Finally,
The
Platonic
Representation
Hypothesis
resentdata
arebecoming
morealigned.
Next, weticularly
deep
networks,
areconverging.
First,
wesurveymanymodels
examplesof convergenceinthelitNeural
networks,
trainedwithdifferent
objectives
as vision models andlanguagegetlarger,on different data and modalities, will tend to conAbstract of these
Abstract
we discuss the implications
trends,theirdemonstrate
convergencedata
modalities:*
1*Representation
1acrossThe*1*1
TheCheung
Platonic
Platonic
Hypothesis
Representation
Hypothesis
surveythey
many
examplesoferature:
convergenceindatapoints
thelit-acrossMinyoung
Huh
Brian
Tongzhou
Wang
Phillip
Isolaover timeandmultipledomains,measurethe distance
betweeninondifferent
dataandmodalities,
will
tend
to
converge
to andaparcommon
statistical
modelof reality.
We argue thatand
representations
We arguein that
AI models,representations
parin AImodels,limitations,counterexamples
to ouranalysis.
asvisionmodelslanguagemodelsgetlarger,over andtime
acrossmultiple
The
Platonic
Hypothesis* more1 and alike* 1domains,*1
the Representation
ways
by whichdifferent
neuralnetworks
ticularly
deep
networks,
ticularlyareconverging.
deepFirst, wearestatisticalconverging. NeuralFirst, we
more
way.
We
hypothesize
that
verge
torepanetworks,
commonmodelof reality.
Minyoung Huh * 1 erature:
Briana Cheung
Tongzhou
Wang
Phillip
Isolanetworks,
trainedNeuralwith
networks,
different objectives
trained with different objectives
theymeasurethe
distance
betweendatapoints
inthe wayswhich different
neuralnetworks
rep-many
surveyof convergence
manyweexamplesin the litof convergence onin thelit- data and modalities,
data
are becoming
morealigned.
Next,
thisbyconvergenceisresent
driving
towarda survey
sharedsta-examples
different
on differentwill
datatend
and tomodalities,
conwill tend to con*Abstract
1 domains,*domains,
1 alike way. We hypothesize
*1
*1
a
more
and
more
that
erature:
over
time
and
erature:acrossmultiple
over
time
and
acrossmultiple
Minyoung
Huh
Brian Cheung vergeTongzhou
Isola
Introduction
resent tistical
data aremodel
becoming
more akinaligned.
Next,
weto a commonWang
statisticalverge
tomodelaPhillip
commonofRepresentation
reality.
statistical modelHypothesisof reality.
demonstrate
convergenceacrossdata modalities:of reality,
to1.
Plato’sconcept
The
Platonic
the ways by which
different
the waysneuralbywhich
networks
different
repneural
networksparrep-driving toward a shared stathis inconvergence
is
We
argue
that
representations
AI models,demonstrate
convergenceacrossdata
modalities:
asvisionmodelsandlanguagemodelsgetlarger,of
an
ideal
reality.
We
term
such
a
representation
Abstract
resent data
becoming
resentmore
dataaligned.
are becoming
Next,
wemore aligned.
Next,
we
AI
systems
are arerapidly
evolving
intohighlymultifunctional
ticularly
deep
networks,
areconverging.
First,
wetisticalmodelofreality,akin toNeural
Plato’snetworks,
concept trained with different objectives
*
1
*
1
*
1
*
1
The
Platonic
Representation
Hypothesis
asvisionandlanguagemodelsgetlarger,theymeasurethe
distance
betweendatapoints
inthe models
platonic
representation
andentities.
discuss
severaldemonstrate
convergencedemonstrateacross data
convergencemodalities:across data modalities:
Minyoung
Huh
Brian
CheungTongzhou
Wang
Phillip
Isolasurveymany examples
of ofconvergencein the litFor example,
whereasin the past
weanhadspecialWe argue that representations in AI models,par- the distance between datapoints
ideal
reality.
We
term1.such
aPlatonic
representationondifferent
data and modalities,
will tend
to conAbstractasvisionmodelsandlanguageasvisionmodelsmodelsgetandlarger,languagemodelsget larger,
they measure
inFigureThe
Representation
Hypothesis:
Images(X)
possible selective apressures
it.
Finally,
more andtoward
more
alike
way.
We
hypothesize
that
erature:
over
time
and
acrossmultipledomains,
purposesolutionsfordifferent
languageprocessing
tasks
The
Platonic
Representation
Hypothesis
ticularly deep networks, are converging.
First,and
wemore alike way.
verge
to a several
commonstatisticalmodel of reality.
theplatonic
representation
anddiscusstheymeasurethe different
distance
theybetween
measuredatapoints
the distancein between
datapoints
inand
Neuralnetworks,
trainedwith
objectives
text
(Y
)
are
projectionsofacommonunderlying
reality (Z).
a moreWe
hypothesize
that
we discuss
the implicationsof
thesetrends,their
this convergenceisdriving
towardashared
staWe
argue
that
representationsinAI
models,parthe
ways
by
which
different
neuralnetworks
rep(e.g.,
sentiment
analysis,
parsing,
dialogue),modernlarge thataand
moremodalities,
and morealikewill
away.
more
We
and
hypothesize
morealikethatway.
We hypothesize
survey many examples of convergenceinconvergencethe litpossible
selective
pressures
towardFinally, learning algorithms will conWe conjecture
thatit.representationondifferent
data
tend
to
conthisisdriving
towardasharedstalimitations, and counterexamples
toouranalysis.
Abstract
tistical modelofreality,akin
toPlato’sconcept
ticularly
deep
networks,
areconverging.
First,
we
resentdata
arebecoming
morealigned.
Next,
wethis models
convergenceis driving
thistoward a shared
isatdriving
sta-these
towardathe
sharedsta- Neuralofnetworks,
traineddifferent
objectives
language(LLMs)
areconvergencecompetent
all
tasks
uswe
discuss
implications
these trends,
theirwitherature: over time and across multiple
domains,verge
onasharedrepresentation
of
Z,
and
scalingmodel size, as
The
Platonic
Representation
Hypothesis
verge
to
a
commonstatisticalmodel
ofreality.
tisticalmodelofreality,akin
toPlato’sconcept
demonstrate
convergenceacrossdata
modalities:
of
an
ideal
reality.
We
term
such
a
representation
tisticalmodelofreality,tistical
akin
model
to
Plato’sofreality,conceptakin
toPlato’sconcept
surveymany
examplesofconvergenceinthelitWe argueneuralthat representationsin AI models, paring a single set of weights (Srivastava et al.,limitations,2022). Unified
on as
different
and modalities,
tend to conandlarger,
counterexamples
to data
our analysis.
the ways by which different
networks
repwelldata andtask
diversity,
drives thiswill
convergence.
asvisionmodelsandlanguagemodels
get
of
an
ideal
reality.
We
of
term
an
such
ideal
a
reality.
representation
We
term
such
a
representation
of
an
ideal
reality.
We
term
such
a
representation
the weplatonicsystems
representation
andanddiscuss
severalerature:overtimeacrossmultipledomains, instead
ticularly
deep
networks,
arewe
converging. First,
are
also
being
built
acrossdata
modalities:
verge to a common statistical model of reality.
1.Next,Introduction
resent data are becoming
morealigned.
Neuralnetworks,
trainedwithdifferent
objectives
theplatonic
representation
theplatonic
anddiscuss
representation
severalanddiscuss
severaltheymeasurethe
distance
betweendatapoints
inthe platonic
representation
anddiscuss
severalpossibleselective
pressures
towardit. toward
Finally,
theusing
ways
which
different
neuralnetworks
repsurvey
many
examplesofconvergence
in
the
litof
a by
differentarchitecture
foralike
processing
imagesverpossible
selective
pressures
possibleselective
it. Finally,pressures
towardit.
Finally,demonstrate convergence across datapossible
modalities:ondifferent
dataandmodalities,
will
tend
to
cona
more
and
more
way.
We
hypothesizethat
1.
Introduction
selective
pressures
towardit.
Finally,AI
systemsarerapidly
evolving
intohighlymultifunctional
wediscuss
theimplicationsof
thesetrends,their
resentdata
arebecoming
morealigned.
Next,
we
we discuss
theimplications
wediscuss
of as
these
the
implications
theirof thesetrends,sus text,
recentmodels,such
GPT4-V(Achiamal., theirovertime andacross multiple domains,
this convergenceistrends,
driving
towarda etsharedsta-This paper explores one aspect of this trend: representational
as vision models anderature:
languagemodelsgetlarger,verge
toacommonstatisticalmodel
ofreality.
wediscussFor
theexample,
implicationsof thesetrends,theirentities.
whereasin thepast
we
had convergencespeciallimitations,andcounterexamples
limitations,
toand
our
counterexamples
analysis.
to ouranalysis. convergence. We argue that there is a growing similarity
limitations,andcounterexamples
toouranalysis.
demonstrateacrossdata
modalities:2023),
Gemini
(Anil
et
al.,
2023),
and
LLaVA
(Liu
et
al.,
the
ways
by
which
different
neuralnetworks
reptisticalmodel
ofreality,akin
toPlato’sconcept
Figure
1.
The
Platonic
Representation
Hypothesis:
Imagesmultifunctional(X)
AI systems are rapidly evolving
into highly
they measure the distance between datapoints
inand counterexamples
purposesolutionsfordifferent
languageprocessing
tasks
limitations,toouranalysis.
as visionmodelsgetlarger,inunderlying
howindatapoints
arehadrepresentedof anlanguage
ideal
term
such
a representation2023),
handleboth andmodalities
withaWe
combinedarchitecture.
resentdata
becoming morealigned. Next, we
andreality.
textmodels(Y)entities.
areprojectionsof a common(Z).
For
example,whereasthe reality
past we
special- in different neural neta more and more alike
way.
Weare
hypothesize
that(e.g., sentiment
analysis,
parsing, dialogue),modernlarge
Figuredifferent
1. The Platonic
1. Introduction
1. Introduction
theymeasurethe
distance
betweendatapoints
intheplatonic
representation
anddiscuss forseveralwork
models.
This
similaritytasksspans acrossmodel Representati
More
and
more
systems
are
built
off
of
general-purpose
Weconjecturethat
representation
learning
algorithms
willcondemonstrate
convergenceacrossdata
modalities:
purposesolutionsdifferent
languageprocessing
this convergence is driving toward a sharedsta1. Introduction
andtext(Y modalities.
) are projections of a com
languagemodels(LLMs)are competentatsystems
all thesetasks
uspossible
selective
pressures
towardit.modFinally,
a
more
and
more
alike
way.
We
hypothesizethatarchitectures,training
objectives,
and
even
data
verge
on
a
sharedrepresentation
of
Z,
and
scaling
model
size,
as
pretrained
backbones,
sometimes
calledfoundationAIarerapidlyAI
evolving
systemsintoarehighlyrapidlymultifunctional
evolving
intohighlymultifunctionalasvisionmodelsandlanguagemodelsgetlarger,(e.g., sentiment analysis, parsing, dialogue), modern large
Introduction
tistical model of reality, akin to1.Plato’sWe
conjecture
thattorepresentation lewediscuss
theimplicationsoftrends,their
ingconcept
a single set of weights(Srivastava
etal.,2022).
UnifiedA redsphere nextwellasthatdatasupportandtask
diversity,
drivesthisconvergence.
entities.
For example,
entities.
whereasFor
in theexample,past
wewhereashad
specialinthese
the past
we hadspecialthis
convergenceishighlydriving
towardasharedstaels(Bommasanietal.,2021),
a
large
range
AI
systemsarerapidly
evolving
into
multifunctionaltheymeasurethe
distance
betweendatapoints
inlanguage models(LLMs)
competent
all
these
tasks
us- (X)Will
Figure1. Theare
Platonic
Representation
Figure1.atTheHypothesis:
Representation
ImagesHypothesis:
a bluecone.
of an ideal reality. We term such a representationSo,
what
has ledto Platonic
thisconvergence?
iton
continue?
Andverge
a Imagesshared(X)representationof Z
limitations,andcounterexamples
toouranalysis.
systems
are
also
being
built
acrossdata
modalities:
insteadpurposesolutionsfordifferent
purposesolutionslanguageforprocessing
differenttasks
languageprocessing
tasksAIsystemsarerapidly
evolving
intohighlymultifunctional
tisticalmodelofreality,akin
toPlato’sconceptoftasks
includingrobotics(Driess
et
al.,
2023;
Brohanentities.
For
example,whereas
in
the
past
we
had
specialand
text
(Y
)
are
projections
and
of
text
a
common(Y)
areprojections
underlyingofrealityacommon(Z).
underlying
reality(Z).
a more andalikeseveral
way. We hypothesize thating
a single
set ofweights(Srivastava
et al.,does2022).
Unified Images
ultimately,
where
it
end?
the platonic representation
andmorediscusswell(X)
as data and task diversity, drives(e.g., sentiment
analysis,(e.g.,
parsing,sentiment
dialogue),analysis,modernparsing,
large
dialogue),modernlarge
Figure
1.
The
Platonic
Representation
Hypothesis:
of
using
a
differentarchitectureforprocessing
imagesverWe
conjecturethat
representationWe
conjecturelearningthat
algorithms
representation
will
learning
conalgorithms
will conentities.
For
example,whereas
in
the
past
we
had
specialofanidealreality.
We
term
such
a
representationetal.,2023),
bioinformatics
(Ma
et
al.,
2024),
andhealthpurpose
solutionsfordifferent
languageprocessing
tasks
this
convergenceisdriving
towardasharedstasystems
are
also
being
built
acrossdata
modalities:
insteadlanguagemodels(LLMs)
languagearecompetent
models(LLMs)
at
all
these
are
tasks
competent
usat
all
these
tasks
uspossible selective pressures toward it.
Finally,
Figure1. ThePlatonic
Representation
Imagesandaspecttext(Yof) Hypothesis:
areprojections
ofon
aofcommon
reality
(Z).
verge
on
a shared
representation
verge
a(X)
Zshared(i.e. frepresentation(X) ftext
of (Y
Z
(i.e.
)), fimg(X) ftext (Y )),
This
paper
exploresone
thistrend:
representational
imgunderlying
sussolutions
text, recent
models,suchas
GPT4-V(Achiametal.,1.
Introduction
Our
centralhypothesis,
statedabove
inFigure1, is that
there
purposefordifferent
languageprocessing
tasks
care
(Steinbergetal.,2021).
In
short,
AI
systemsarebecomtheplatonic
representation
anddiscuss
several(e.g.,
sentiment
analysis,
parsing,
dialogue),modernlarge
ingasinglesetofweightsinga(Srivastava
single
set
of
et
al.,
weights2022).
(Srivastava
Unifiedetal.,2022).
Unifiedtisticalmodelofreality,akin
toPlato’sconceptofusingadifferentarchitectureforprocessing
imagesverandscaling
model
size,
as
and
well
scaling
as
data
model
and
size,
task
diversity,
as
well
as
drives
data
and
task
diversity,
drives
and
text
(Y
)
are
projectionsofacommonunderlying
reality(Z).
We
conjecturethat
representation
learning
algorithms
willconwe discuss the implications of these trends,
convergence.
We
argue
that
there instead
isisaindeed
growingsimilarity
2023),their
Gemini(Anil
et al., 2023),
and
LLaVA
(Liu
et
al.,
an
endpoint
to thisconvergence
and aexplores
principle
systems
are
also
being
systems
built
across
are
also
data
being
modalities:
built
acrossinsteaddata
modalities:
ing
increasingly
homogeneous
in
both
their
architectures(e.g.,
sentiment
analysis,
parsing,
dialogue),modernlarge
possibleselective
pressures
towardit.
Finally,
thisconvergence.
thisconvergence.
languagemodels(LLMs)
arecompetent
at
all
these
tasks
usThispaper
one aspect of t
of
an
ideal
reality.
We
term
such
a
representation
sus
text,
recentmodels,suchas
GPT4-V(Achiametal.,AIsystemsarerapidly
evolving
intohighlymultifunctionalWe
conjecturethat
representation
learning
algorithms
willconvergeonasharedrepresentation
of
Z
(i.e.
f
(X)
f
(Y
)),
img
text
limitations, and counterexamples to our2023),
analysis.
inhowdatapoints
arerepresentedindifferent
neuralnethandle
both
modalities
withacombinedarchitecture.
oftheseusing
acapabilities.
differentarchitectureof using afor
different
processing
architecture
images theirforver-processing images
ver-drives it: different models are all trying to arrive at a
that
their
models and(LLMs)
are acompetent
all
tasks
we
discuss
theusimplicationsof
thesetrends,ing
single set atofandweights(Srivastava
etal.,example,2022).
Unifiedconvergence.
We argue that ther
entities.
For
whereasinGemini
the
past
we
had
specialthe platoniclanguage
representation
discuss
several2023),
(Anil
et
al.,
2023),
and
LLaVA
(Liu
etdiversity,
al.,
verge
on
a
sharedrepresentation
of
Z,
and
scalingmodel
size,
as andandscaling
model
size,
as
well
as
data
task
drivessus
text,
recentmodels,
sus
such
text,
as
recent
GPT4-Vmodels,(Achiam
such
et
as
al.,
GPT4-V(Achiam
et
al.,
work
models.
This
similarity
spans
acrossdifferent
model
More set
and moresystemsare built
offlimitations,of general-purpose
representationofreality,meaninga representationof theing a single
weights
et al.,2022).
Unifiedandcounterexamples
toouranalysis.
purposesolutionsfordifferent
languageprocessing
tasks
in
how
datapoints
aresystems
are also
being
built
acrossdata
modalities:
instead2023),
handleboth
modalities
withacombinedarchitecture.
possible selective
pressuresoftoward
it.(Srivastava
Finally,
well
as
data
and
task
diversity,
drivesthisconvergence.
*
1al.,Figure
1.
The
Platonic
Representation
Hypothesis:
Images(X)represente
thisconvergence.
This
paper
exploresone
This
aspect
paper
of
this
explorestrend:
one
representational
aspectof
thistrend:
representational
2023),
Gemini(Anilet2023),2023),
Gemini
and
(Anil
LLaVA
et
al.,
(Liu
2023),
et
al.,
and
LLaVA
(Liu
et
al.,
Equal
contribution
MIT.architectures,
Correspondence
to: <FIXME>.
training
objectives,
and
even
data
modalities.
pretrained
backbones,
sometimes
calledfoundation
modjoint
distribution
over
events
in
the
world
that
generatethe(e.g.,
sentiment
parsing,
dialogue),modernlarge
1. Introduction
systems
are alsoofbeing
acrossmodalities:
insteadandtext(Ygeneral-purpose
) is
are
of
a commonunderlying
reality(Z).
convergence.
Webuilt
argue
convergence.
that
there
We
a projections
growingargue thatsimilarity
there
isworka growing
similarity
models.
This
similarity
spa2023),
handleboth
modalities
2023), handle
withanalysis,
aimagescombinedboth modalities
architecture.
with
amore
combinedarchitecture.
of using
a datadifferentarchitectureforprocessing
verMoreandsystemsareoffof
we discuss theimplications
thesebuilt
trends,their
els(Bommasanietal.,2021),
that
support
a
large
range
languagemodels(LLMs)
arecompetent
at
all
these
tasks
us- wedata
observe.
Figure1represented
conveysthishypothesis:
there
st areWe
conjecture
representation
learning
algorithms
will coningeneral-purpose
how datapoints
areinrepresented
how
datapoints
in thatdifferent
areneural netin
different
neuraltraining
netMoreimages
andmoresystems
and
built
more
off ofsystems
general-purpose
aretobuilt
offofof using
a different architecture
processing
So,
what
has
led
this
convergence?
Will
it continue?
Andarchitectures,
objectives,
susanalysis.
text,forrecent
models,suchas
GPT4-V(Achiametal.,pretrained
sometimes
calledfoundation
modProceedings
ofverthe41MoreInternational
Conference
onbackbones,
Machinelimitations,andcounterexamples
to
our
1.
Introductionoftasks including robotics (DriessLearning,
etpretrainedal., 2023;
BrohanAI systems are rapidly evolving into highly
multifunctionalvergeworldona shared
representation
ofZ),
Z,
and
scalingwork
Thissimilarity
work
models.
spans
acrossThis
similarity
different
model
spans
acrossdifferent
modelsize, asing
a single
setof
weights(Srivastava
etital.,2022).
Unifiedbackbones,
pretrained
sometimes
backbones,
calledfoundation
sometimes
modcalled
foundation
modexists
athat
real
out
there
(labeledwhich
wemodelmeasureThis
paper
exploresonedoes
aspectof
thistrend:
representational
Vienna,Austria.
PMLR
235,
2024.
Copyright
2024
bymodels.
ultimately,
where
end?
sus text,
recent models, 2023),
such asGemini
GPT4-V(Achiam
et al.,
This
paper
exploresone
aspectof
thistrend:
representationalels(Bommasanietal.,2021),
support
a
large
range
(Anil
et
al.,
2023),
and
LLaVA
(Liu
et
al.,
architectures,training objectives,
architectures,training
even
data
objectives,modalities.
andSo,
even
data hasmodalities.
well assensors,
dataandandtask
diversity,
drivesthisconvergence.
elsauthor(s).
(Bommasani
et al.,els2021),(Bommasani
thatbeing
supportetbuiltal.,a 2021),
large range
thata large
range
what
led
toleftthis convergenarealsoacrossdatasupport
modalities:
insteadal.,2023),
bioinformatics (Ma et theal.,2024),
andsystems
healthentities. For example, whereas in the2023),
past etwe
hadspecialwith
varioussuchasthecamerashown
totheconvergence.
We
argue
that
there
is
a
growingsimilarity
Gemini (Anil et al.,2023),
and
LLaVA
(Liu
et
al.,
convergence.
We
argue
that
there
is
a
growingsimilarityoftasks
robotics(Driessetthatal.,there
2023; Brohan2023),
bothmodalities
withaevolvingcombinedarchitecture.
Figurehandle
Platonic
Representation
Images(X)
AI
systemsarerapidlyintohighlymultifunctionaltasks
includingrobotics
ofHypothesis:(Driess
including
etal.,robotics
2023;
Brohan(Driess
et al., above
2023;
Brohanofusingatasksdifferentarchitecture
for including
processing
imagesverOurcentralhypothesis,
statedinled
Figure1,iswherecare (Steinberg
et al.,2021).1.InThe
short,
AIof
systemsarebecompurpose solutions forlanguageprocessing
tasks
So,
what
has
to
this
So,
convergence?
what
has
led
Will
to thisit continue?
convergence?
And ultimately,Will it continue?
Anddoes it end?
1. different
Introductioninhowdatapoints
arerepresentedindifferent
neuralnet2023), handle both modalities
amore
combinedarchitecture.
and with
text)entities.
are projectionsof bioinformatics
abuiltcommonunderlying
(Z).
in
how
datapoints
arerepresented
inand
different
neural netet endpointal.,2023),
bioinformatics
etal.,2024),
healthet al.,2023),
et al.,2023),(Maet bioinformatics
al., reality2024),
andhealthetspecialal.,2024),and healthMore
and(Ysystems
areoffofgeneral-purpose
For
example,whereas
the
past
we
had
sus
text,
recentmodels,such
as(MaGPT4-V(Achiam
et al.,(Maultimately,
where
does
ultimately,
it
end?
where
does
it
end?
is in
indeedantothisconvergenceandaprinciple
ingmodernincreasingly
homogeneousin boththeirarchitectures(e.g., sentiment analysis, parsing, dialogue),large
1.short,The Platonic
Representation
Hypothesis:
Images(X) stated abov
1Figurecare(Steinbergetal.,2021).
care(Steinberg
Inalgorithms
short,
AIet systems
al.,will2021).
areInbecomshort,AIsystemsarebecomwork
models.
This
similarity
spans
acrossdifferentmodel
Our
centralhypothesis,We
conjecturethat
representation
learning
conMoreandmoresystems
are
built
off
of
general-purpose
care(Steinbergetal.,2021).
In
AI
systemsarebecomwork
models.
This
similarity
spans
acrossdifferent
model
This
paper
exploresone
aspectof
thistrend:
representational
pretrained
backbones,
sometimes
calledfoundation
modpurpose
solutionsfordifferent
languageprocessing
tasks2023),
Gemini
(Anil
et
al.,
2023),
and
LLaVA
(Liu
et
al.,
AI systems
are rapidly
evolving
intohighlymultifunctional ing increasingly homogeneousthat
drivesit:different
modelsareandalltryingtoarriveata of a1,statedandtheirtasks
capabilities.
text(Y modalities.
)Our
arecentral
projections
common(Z). to this con
Our central
hypothesis,
statedabovehypothesis,
in Figureis thatabove
thereunderlying
in Figure 1, isrealitythat there
language models (LLMs)
are competent
at
all
these
using
increasingly
in bothhomogeneous
theirarchitecturesin boththeirarchitectures
verge
on
a
sharedrepresentation
of
Z,
and
scaling
model
size,
as
is
indeedan
endpointarchitectures,training
objectives,
and
even
data
convergence.
We
argue
that
there
is
a
growingsimilarity
ing
increasingly
homogeneous
in
both
their
architectures
pretrained
backbones,
sometimes
calledfoundation
modarchitectures,training
objectives,
and
even
data
modalities.
2023),
handleboth
modalities
withacombinedarchitecture.
els(Bommasanietal.,2021),
that
support
a
large
range
(e.g., sentiment
analysis,andparsing,dialogue), ofmodernlarge
entities.(Srivastava
For example,
whereasin the past we had specialreality,
meaning
representationofthe andis indeedWe
anaAendpointisto indeed
thisconvergencean endpoint
to athisprinciple
convergence
and a principle
and their capabilities.
theirrepresentation
capabilities.
conjecturethat
representation
learning
algorithms
will
con-neting a single set of weightsetal.,2022).
Unified*1
redspherenext
Figure
1.
The
Platonic
Representation
Hypothesis:
Images
(X)
well
as
data
and
task
diversity,
drivesthisconvergence.
inhowdatapoints
arerepresentedindifferent
neuralthat
drivesit:different
models a
More
and
more
systems
are
built
off
of
general-purpose
and
their
capabilities.
els(Bommasanietal.,2021),
that
support
a
large
range
Equal
contribution
MIT.
Correspondence
to:<FIXME>.
of tasks including
robotics
etare
al.,
2023;
Brohanlanguagemodels(Driess(LLMs)
competent
at1 allthesetasks
usthatdrivesit:todifferentthatmodelsdrivesareit:alldifferent
trying tomodelsarriveareat aalltryingmodel
to arriveat aaspurposesolutionsfordifferent
languageprocessing
tasks
a
blue
cone.
joint
distribution
over
events
in
the
world
that
generate
the
verge
on
a
sharedrepresentation
of
Z,
and
scaling
size,
So,
what
has
led
to
this
convergence?
Will
it
continue?
And*
1*
So, what
ledmodto this convergence?
Willsimilarity
it continue?
Andsystems are also being built across data
modalities:
insteadEqualtext
contribution
MIT.Equal
Correspondence
contribution
to:MIT.
<FIXME>.
Correspondence
to: has<FIXME>.
and(Y pretrained) are projectionsof a common
underlying
reality(Z).
work
models.
This
spansacross different
backbones,
sometimes
calledfoundation
representation
reality, meanin
representationofreality,representation
meaningarepresentationofreality,meaningofthearepresentation
of ofthemodeloftasks
includingrobotics(Driess
et
al.,
2023;
Brohanetal.,2023),
bioinformatics
(Ma
et
al.,
2024),
andhealthingasinglesetofweights(Srivastava
etal.,2022).
Unified*1(e.g.,
sentiment
analysis,
parsing,
dialogue),modern
large
data
we
observe.
Figure1conveysthishypothesis:
there
st
well
as
data
and
task
diversity,
drivesthisconvergence.
ultimately,
where
does
it
end?
Equal
contribution
MIT.
Correspondence
to:world<FIXME>.
ultimately,
where
does
end?
of using a different architecture for processing
imagesverWe conjecturethat
representation
learning
algorithms
conarchitectures,training
objectives,
and thateven
data modalities.
Proceedingsofthe 41 (MaInternational
Conference
on Machine
els41(Bommasani
etConferenceal.,412021),
that
support
awill
large
range
joint
distribution
over iteventsjoint
distribution
in theoverthatevents
generatein thetheworld
generate
theover events in th
st
st
joint
distributionetal.,2023),
bioinformatics
etal.,2024),
andhealthProceedingsoftheProceedingsInternationaloftheInternational
on
MachineConferenceonMachinecare(Steinbergetal.,2021).
In
short,
AI
systemsarebecomsystems
are
also
being
built
acrossdata
modalities:
insteadlanguagemodels(LLMs)
arecompetent
at
all
these
tasks
usexists
a
real
world
out
there
(labeled
Z),
which
we
measureLearning,
Vienna,Austria.
2024.
Copyright
2024
ftext
ThisPMLR
paper 235,
exploresone
aspectofthisbytrend:
representationalverge
on a shared
representationofCopyright
Z (i.e.
f2024
(X)
⇡data
f2023;)), by Figurewest(Yobserve.
data
1 we
conveysobserve.
this Figure
hypothesis:
1 conveys
there data
this hypothesis:
there Figure 1 conveoftasks
includingrobotics(Driess
et2024.
al.,Brohansus text, recent models, such as GPT4-V(Achiam
et
al.,
img
text
Learning,
Vienna,Austria.
Learning,
PMLR
235,
Vienna,2024.
Austria.
PMLR
235,
by
Copyright
2024
we
observe.
Ourcentralhypothesis,
statedabove
inFigure
1,
is
that
there
care
(Steinbergetal.,2021).
In
short,
AI
systemsarebecomOur
centralhypothesis,
statedabove
inFigure
1,
is
that
there
ing
increasingly
homogeneous
in
both
their
architecturesof using
a different
architectureforprocessing
verProceedingsof2024),theInternational
Conference
ingetaal.,
singleset ofweights
et al.,2022).
Unifiedthe (Srivastava
author(s).
withvarioussensors,
such
as41
the
camerashown
totheworld
left
theargueauthor(s).
theauthor(s).
existsadrivesrealworld out
there
exists(labeled
a realZ),onwhich
outMachine
there
we (labeledmeasure Z), which we measureandscaling
as
well
as
data
and
task
diversity,
etmodelal.,2023),
bioinformatics(Maimages
et al.,
andhealthconvergence.
We
that there
issize,a
growingsimilarity
2023), Gemini (Anil2023),
and
LLaVA
(Liu
et
al.,
existsa realworld out
there (label
So,
what
has
led
to
this
convergence?
Will
it continue?
And
is indeed
an2021).
endpoint
toVienna,
thisconvergenceand235,
aexplores
principle
Learning,
Austria.
PMLR
2024.convergence
Copyright
2024
bya trend:ing increasingly
homogeneous
incapabilities.
both
their
architecturesisindeedan
endpoint
to
this
and
principle
This
paper
one
aspectof
thisrepresentational
and
their
sus
text,
recentmodels,suchas
GPT4-V(Achiametal.,systems
are
also
being
built
acrossdata
modalities:
insteadcare(Steinberg
et
al.,
In
short,
AI
systemsarebecomthis represented
convergence. in different neural netin how datapoints are2023), handle both modalities with a combinedarchitecture.
ultimately,
where
does it end?
the author(s).
withvarioussensors, such as thethatdrivesit:different
modelsarealltryingtoarriveata11andarchitecture
their capabilities.
thatdrivesit:different
modelsarealltryingtoarriveataconvergence.
We
argue
that
there
is
a
growingsimilarity
2023),
Gemini
(Anil
et
al.,
2023),
and
LLaVA
(Liu
et
al.,
ing
increasingly
homogeneous
in
both
their
architectures
of
using
a
differentforprocessing
imagesver1 different model*
work
This 1similarity
spans acrossMore and more systems are built off of general-purpose
Equalmodels.
contribution
MIT. Correspondence
to: <FIXME>.
representationof reality,
meaning a representationof theOurmeaning
centralhypothesis,
statedinabove
Figure1, is that
there
representation
reality,
arepresented
representation
ofinthein ofhowdatapoints
aredifferent
neuralnetandtheircapabilities.
2023),
handleboth
modalities
withacombinedarchitecture.
sus
text,
recentmodels,suchas
GPT4-V(Achiametal.,*1
training
objectives, andjoint
evendistribution
data modalities.
contribution
Correspondence
to: <FIXME>.
pretrained backbones, sometimes calledEqual
foundation
mod-MIT.architectures,
1convergenceis indeedanendpoint
to thisand modela principle
overeventsinthe worldthatgenerate
joint
distribution
over
eventsinthe
theworld thatgenerate
the
st
work
models.
This
similarity
spans
acrossdifferent
More
and
more
systems
are
built
off
of
general-purpose
This
paper
exploresone
aspectof
thistrend:
representational*
1 Machine2023),
Gemini
(Anil
et
al.,
2023),
and
LLaVA
(Liu
et
al.,
Proceedings
of
the
41
International
Conferenceon
Equal contribution MIT. Correspondence to: <FIXME>.
els (Bommasani et al., 2021), that support a large rangest
that drives
data we observe. Figure 1 conveys this hypothesis:
thereit: different models are all trying to arrive at a

Conclusion• N=0 is possible• You can learn to see from noise, code, and/or language• Can you learn to see from anything??

→ Platonic Representation Hypothesis

Thanks!
Manel Baradad
Jonas Wulff
Tongzhou Wang
Pratyusha Sharma
Tamar Rott Shaham
Stephanie Fu
Adrián Rodríguez-Muñoz
Shivam Duggal
Antonio Torralba