Localized Learning Through the Lens ofGlobal Workspace TheoryYuwei Sun
University of Tokyo

The state of deep learning
Large language modelGo

Text to video

Self-driving carProtein folding

CoRN 2023

2

Building systems with consciousness?

CoRN 2023

3

Making sense of information processing•

Selection of information for global broadcasting, thus making it flexibly
available for computation and report (C1)

•Self-monitoring of those computations, leading to a subjective sense of certainty
or error (C2)
Dehaene et al., What is consciousness, and could machines have it, Science 2017

Ø Our goal: architecture that resembles theC1 functionality in terms of information
reusability

CoRN 2023

4

System 1 and System 2 AI
System 1System 2• Intuitive and fast

• Explicit and slow

• Without explanation• Logical reasoning and planning

Monolithic neural networks
CNNs, RNNs, Transformers …

Neural coordination
Localized Learning, Meta Learning…

Thousand Brains Theory, Jeff Hawkins 2018
CoRN 2023

5

System 1 and System 2 AI
System 1

How to bridge the gap?

System 2• Intuitive and fast

• Explicit and slow

• Without explanation• Logical reasoning and planning

Monolithic neural networks
CNNs, RNNs, Transformers …

Neural coordination
Localized Learning, Meta Learning…

Thousand Brains Theory, Jeff Hawkins 2018
CoRN 2023

6

Grounding of Global Workspace Theory for C1 functionality• A collection of specialized modules• Guided attention in a limited capacity workspacewith a communication bottleneck• States are conscious when they are broadcast to
many modules through the workspace• Modules compete to share info for better efficiency[Kanai, 2021]
CoRN 2023[Baars, 1988; Dehaene, 1998]

7

How to implement GW in artificial systems
Global Learning

Localized Learning••

Learns patterns and relationships over theentire dataset••Learns patterns and features in a restricted
context of data for specialized tasks

Ideal for capturing general trends and•Faster convergence on local patterns

insights•1) Modules are trained on independent tasks

Slow and less suitable for generalizationor 2) jointly trained end-to-end, from which
module specialization naturally emerges

Ø A collection of specialized modules which can perform tasks in parallel
CoRN 2023

8Localized Learning

Coordination in Shared Space

Global Latent Workspace

Bengio et al., ICLR’22Kanai et al., Trends in Neurosciences 2021

Capsule Networks

Graph-structured causality

Hinton et al., NeurIPS’17

Goyal and Bengio, arXiv’22
CoRN 2023

9

“Are the boys sitting
at a table?”“no”“Is there a giraffein the picture?”“yes”“How many cats?”
transfer“3”“How many people
are on the right?”“2”

Tractable toy problems of generalization
Ø Could implementing global workspace improve information alignment
among modules, mitigating information loss in knowledge transfer?
transfer

𝑃! (𝑥)𝑃" (𝑥)

transfer

𝑃! (𝑦)𝑃" (𝑦)

𝑃! (𝑥|𝑦) 𝑃" (𝑥|𝑦)“What color isthe frisbee?”

Covariate shift

Prior probability shift

Concept shift

𝑃! 𝑦 𝑥 = 𝑃" 𝑦 𝑥 ,but 𝑃! (𝑥) ≠ 𝑃" (𝑥)𝑃! 𝑦 𝑥 = 𝑃" 𝑦 𝑥 , but 𝑃! (𝑦) ≠ 𝑃" (𝑦)𝑃! 𝑥 𝑦 ≠the𝑃"pizzas?”
𝑥𝑦 ,but 𝑃! 𝑦 = 𝑃" (𝑦)“purple”“What shape are“square”

Meta Learning in Decentralized Neural Networks: Towards More General AI, AAAI 23
CoRN 2023

10GW Case 1: modules are trained on independent tasks
Ø Generalization through shared workspace by reusing localized
Discrepancy
knowledge from various modules
FeatureExtractor

DS(1)Observation 1

Embedding
Matching

Classifier
DS(1)

DT

DS(1)

DS(2)

Agent

Observation 2

Global Workspace

Sharedworkspace

Observation 3

Local modules
trainedLocal models
on independent tasks

...

DS(K)

AgentsNovel task

DS(2)

...

DS(K)

Feature Representation

GlobalmodelDT

Agents Novel task
Localmodels

CoRN 2023

model task
New Global
generalizationDT

Novel task

11

Global features disentangler
Feature Representation

Target/Source
Target/Source

FeatureExtractor
DS(1)SourceAgentClassifier

Features
Disentangler

Sharedfeatures
DT

Global Workspace"𝑓#!" = arg max 𝐽$%&'()*(+,'- (𝑓#/ , 𝑓!" , 𝑓#!0 )#!"TargetNoveltask•

Local feature extractors are trained to learn shared features•𝑓# distinguishes between the source 𝑓$% and the target 𝑓$&
CoRN 202312

Embedding matching
Feature Representation

Discrepancy
Discrepancy

FeatureExtractor

Classifier

StatisticalEmbedding
Matching

DT

DS(1)

Global Workspace
5

1
1 MMD6!# (𝑓!" (𝑋7" ), 𝑓!0 (𝑋8 ))5234""
𝑓#! = arg min 𝑗11$
(𝑓!" , 𝑓#!0 )"𝐽11$=AgentSourcedomainSourceNoveltask
TargetTargetdomain#!"•Common features could be either background noise or objects•

Embedding matching aligns features across multiple observations, which enables the extractionof common objects among these observations, based on a discrepancy loss between 𝑓$% and 𝑓$&
CoRN 2023

13

Module aggregation• Alignment process is carried out
sequentially for each module

Feature Representation
Discrepancy

FeatureExtractor

Embedding
Matching

Classifier
DS(1)DTDS(1)

Global Workspace• Aggregate modules to obtain theD
global model through weight sharingAgent

Novel task

(2)
S...

DS(K)

GlobalmodelDT• Broadcast the global model toreplace each module

AgentsModules
Localmodels

CoRN 2023

Novel task

14

Module aggregation• Alignment process is carried out
sequentially for each module

Feature Representation

FeatureExtractor

Discrepancy
Better-refined
representations for
Embedding
subsequentalignment processes
Matching

Classifier
DS(1)

DT

DS(1)

Global Workspace• Aggregate modules to obtain theD
global model through weight sharingAgent

Novel task

(2)
S...

DS(K)

GlobalmodelDT• Broadcast the global model toreplace each module

AgentsModules
Localmodels

CoRN 2023

Novel task

15

GW Case 1: modules are trained on independent tasks
Ø Generalization through shared workspace by reusing localized
knowledge from various modules
Local ModelDS(1)Target/Source

Discrepancy

Feature Representation

Features
Disentangler

Embedding
Matching

Fine-tuning

Classifier
FeatureExtractors
Label

DS(2)

Federated
Voting

...

DS(K)

DT

Global Workspace

AgentsNovel task

Feature Distribution Matching for Federated Domain Generalization, ACML 22
CoRN 2023

16

Transfer learning in vision and language tasks

TransferDigit-Five [Ganin, 2015]

Office-Caltech10 [Gong, 2012]

Amazon review [Blitzer, 2007]
Task

DVD: This is a great DVD for all collections (Positive)
Book: This book turns the entire concept of intelligence inside out (Negative)
Electronics: This is perfect for my iPod and keeps it totally secure (Positive)
Kitchen: Simple, straight forward to use, very easy to clean, and durable (Positive)Ø Classification tasks with an unsupervised approach based on theshared workspace
CoRN 2023

17

Measuring information loss in the shared workspace• Target Task Accuracy (TTA)•Group Effect (GE)Lower is better

W/O shared workspace
Ours W/O voting
Ours

0.0

Converged

CoRN 2023

18

Measuring information loss in the shared workspace• Target Task Accuracy (TTA)•Group Effect (GE)Lower is better

W/O shared workspace
Ours W/O voting(Our method)
Ours

0.0

Reduced information loss
in the workspace(improved module infotransferability)

Converged

Late convergence
CoRN 2023

19

Improved performance and better aligned representations
Module representation distribution

Digit Five

Add shared workspace
Office-Caltech

CoRN 2023

20

If tasks of modules are unknown
Ideally: adaptively prioritizing someof the modules to enter the space

Reality: the task or knowledge
distribution of modules is unknown

modules★

modules?

★

★

★

Homogeneous Learning: Self-Attention Decentralized Deep Learning. IEEE Access 2022
CoRN 2023

21

Coordination with a Markov decision process
Select a module based ona learnable policy 𝜋★

★
★

Module2Start

𝐺'

1. 𝛥𝐿6 = 𝜂 ∇𝐽(𝑌 : , 𝐿6 (𝑋 : ))2. 𝐺4 = 𝐺9 + ∆𝐿6

Ø Action: reuse and allow the information sharing from a specific module
CoRN 2023

22

Coordination with a Markov decision process
Select a module based ona learnable policy 𝜋★

Optimizationobjectives★
★

Module2Start

𝐺'Evaluation Reward𝑟(1. 𝛥𝐿6 = 𝜂 ∇𝐽(𝑌 : , 𝐿6 (𝑋 : ))Module1

2. 𝐺4 = 𝐺9 + ∆𝐿6

Ø Action: reuse and allow the information sharing from a specific module
CoRN 2023

23

Coordination with a Markov decision processØ Learn a-10 module selection policy that-10 maximizes
the accumulative
reward-15-12-6-20

Reward𝜋)∗ = -14
arg max(𝑟) (𝐺) ) + 𝛾 ⋅ 𝑟),(̂ (𝐺),())
-25
-16+!

-30

-18
-20
-22-35

Reward 𝑟4

Reward 𝑟;-40

Module3-45

Module2Round𝐺'
Module1
Reward 𝑟6
CoRN 2023

Fashi-10-10-20-14-30-18-40Round
round
Trainingroundround(outerloop)

Round

…

Start

MNIST
100Clientsmodules
MNISTwith
with 100

Fashion-MNIST with 10 Clients

Episodic reward

MNIST with 10 Clientsrou𝐺"

Termination state:when the global model achieves acertain level of performance or themaximum selection steps are reached24

GW Case 2: naturally emerging module specialization

Ø Pairwise interactions such as the self-attention in Transformers
become expensive with scaleØ There is an absence of communication bottleneckØ Competition results in naturally emerging module specialization
CoRN 2023[Dosovitskiy, 2021]
25

Inducing global workspace for emerging module specialization(1) Collecting modules(2) Computing the bottleneckattention and selecting modules(3) Information broadcast
from the global workspacepriorssamples

Upscalemodules

XCoRN 2023

27

Inducing global workspace for emerging module specialization(1) Collecting modules(2) Computing the bottleneckattention and selecting modules(3) Information broadcast
from the global workspacepriorssamples

Upscalemodules

XØ Bottleneck allows a few patterns to enter the workspace inducing competition among modules

CoRN 2023

28

Inducing global workspace for emerging module specialization(1) Collecting modules(2) Computing the bottleneckattention and selecting modules(3) Information broadcast
from the global workspacepriorssamples

UpscalemodulesXØ Bottleneck allows a few patterns to enter the workspace inducing competition among modulesØ Hopfield network uses the learned memory from the bottleneck to reconstruct information that
achieves globally lower energy (any neural trajectory that enters an attractor’s basin of
attraction will converge to that attractor)
CoRN 2023

29

Learned bottleneck attention maps in CIFAR10
Image 1

Image 2

Slot 1-16

Image 3

Image 4
Slot 17-32
Slot 1

Slot 2

Slot 3

Slot 4(b) The attention maps for all 32 slots in the memory bank,(a) Aof the attention
maps at to
Slot
1 to Slotapplied region
to four distinct
inputinimages.
Each memory slotØglimpse
Each memory
slot learns
attendto 4a differentof pixels
input imagesof four distinct input images.
learned to attend to different regions of pixels in input images.
CoRN 2023

30

Increased bottleneck attention distribution sparsity
Ø Selected image patches (modules) by the bottleneck attention

CoRN 2023

31

Information retrieval from GW with continuous Hopfield networks
of attraction Hopfield networks
Basin of attractionBasinin continuous[Ramsauer, 2021]

fixed points

metastable states

global fixed point

UpscaleXØ Input module states converge to fixed attractor points in the memory of GW
Ø Emerging module specialization and enhanced performance in small datasets
CoRN 2023

32

Conclusions•Selecting information for global broadcasting and making it flexibly available•

Building an architecture that resembles the C1 functionality by inducing global workspacein conventional ML models•

Modules can be trained on independent tasks or jointly trained end-to-end•

Global workspace helps tackle generalization problems by improving informationtransferability and encouraging the competition among localized modules•

Inducing global workspace based on the bottleneck attention and Hopfield networks for
emerging module specialization

CoRN 2023

33

Localized Learning Through the Lens ofGlobal Workspace TheoryYuwei Sun
University of Tokyo