Dynamic Boltzmann Machine
IBM Research – Tokyo
Takayuki Osogami
osogami@jp.ibm.com

How can we make effective use of spike-timing
dependent plasticity (STDP) in artificial neural networks?
Hebb’s rule (’49)

Cells that fire together,wire together

Today’s artificial neural networks

STDP (’90s)?

[Nessler et al. 2013,
Bengio et al. 2016,
Scellier & Bengio 2016]

Amount of changes depends
on timing of spikes

Bi & Poo (1998)
Dan & Poo (2006)

Takayuki Osogami and Makoto Otsuka, “Seven neurons
memorizing sequences of alphabetical images via spike-timing
dependent plasticity,” Scientific Reports, 5, 14149 (2015).
www.nature.com/articles/srep14149

This talk• Boltzmann machine, Hebb’s rule, STDP
• Dynamic Boltzmann machine [O & Otsuka (2015)]• Experiments [O & Otsuka (2015), Dasgupta & O (2017)]

Boltzmann machine
wijxi

xj

Energy of being x:
𝐸 𝒙 = −𝒙⊤ 𝑾𝒙

Probability of being x:
1
𝑃 𝒙 = exp −𝐸 𝒙
𝑍Parameters: 𝑾 = (𝑤𝑖𝑗 )Variables: x = (x1,x2,…)

𝑍≡exp(−𝐸 𝒙 )𝒙

Learning rule of Boltzmann machine, maximizing loglikelihood [Hinton et al. ’83]
Log likelihood of training data D:
𝐿𝐿 𝐷 =log(𝑃 𝒙 )Neuron ixiSynapse
wij

Neuron j𝒙∈𝐷𝜕
𝐿𝐿 𝐷 =𝜕𝑤𝑖𝑗
Stochastic gradient𝑥𝑖 𝑥𝑗 − 𝑋𝑖 𝑋𝑗𝒙∈𝐷𝑤𝑖𝑗 ← 𝑤𝑖𝑗 + 𝜂 𝑥𝑖 𝑥𝑗 − 𝑋𝑖 𝑋𝑗
Expected value:cf. Hebb’s rule𝑋𝑖 𝑋𝑗 ≡𝑃 𝒙 𝑥𝑖 𝑥𝑗𝒙xjPost-synaptic neuron

Pre-synaptic neuron
Do not distribute

Image courtesy of dream designs at FreeDigitalPhotos.net

Spike-timing dependent plasticity (STDP): Amount ofchanges depends on timing of spikes
Bi & Poo (1998)
Dan & Poo (2006)

Pre-synaptic
neuron ixi

Synapse weakened(Long Term Depression)

Synapse strengthened(Long Term Potentiation)

Synapsewij

Post-synaptic
neuron jxj

We will construct a dynamic Boltzmann machine as alimit of a sequence of Boltzmann machines
Boltzmann machine for a
T-th order Markov model

𝑇→∞
Time

Particular parametricform of weight

x[-T]

x[-1]

x[0]

Historical values Next value

Dynamic Boltzmann machineThis talk• Boltzmann machine, Hebb’s rule, STDP• Dynamic Boltzmann machine (DyBM) [O & Otsuka (2015)]• DyBM with LTP• DyBM with LTP and LTD• Experiments [O & Otsuka (2015), Dasgupta & O (2017)]

LTP: Weight is strengthened when the post-synaptic
neuron fires shortly after the pre-synaptic neuron
Bi & Poo (1998)
Dan & Poo (2006)

Pre-synaptic
neuron ixi

Synapse strengthened(Long Term Potentiation)

Synapsewij

Post-synaptic
neuron jxj

Dynamic Boltzmann machine as a limit of a sequence ofBoltzmann machines
Boltzmann machine for aT-th order Markov model𝑇→∞

Dynamic Boltzmann machine

TimeWeight from neuron i at time tto neuron j at time t + 𝛿𝛿

𝑤𝑖𝑗 = 𝑢𝑖𝑗 𝜆𝛿−𝑑𝑖𝑗
We learn 𝑢𝑖𝑗
x[-T]

x[-1]

x[0]

Historical values Next value𝑑𝑖𝑗𝛿Dynamic Boltzmann machine (LTP only)
Conduction delay, dijxi[t]xi[t-1] xi[t-2] xi[t-3]Probability for neuron j to fire at time t:

xj[t]

𝑋𝑗

Synaptic eligibility trace:
[𝑡−1]

𝛼𝑖𝑗

𝜆𝑡−𝑠−𝑑𝑖𝑗 𝑥𝑖≡𝑠<𝑡−𝑑𝑖𝑗[𝑡]
𝛼𝑖𝑗 ← 𝜆𝑡−𝑑𝑖𝑗
𝑡−1
𝛼𝑖𝑗+ 𝑥𝑖𝑠𝑡𝑡≡ 𝑃 𝑥𝑗 = 1|𝑥 :𝑡−1
1=𝑡−1
1 + exp − 𝑖 𝑢𝑖𝑗 𝛼𝑖𝑗Learning rule of DyBM, maximizing log-likelihood
𝜕𝑡−1𝑡𝑡log 𝑃 𝑥 𝑡 𝑥 [:𝑡−1] = 𝛼𝑖𝑗
𝑥𝑗 − 𝑋𝑗𝜕𝑢𝑖𝑗

Conduction delay, dijxi[t]xi[t-1] xi[t-2] xi[t-3]xj[t]Stochastic gradient for LTP weight:

Spike-timing dependent𝑡−1

𝑢𝑖𝑗 ← 𝑢𝑖𝑗 + 𝜂 𝛼𝑖𝑗[𝑡−1]

𝜆𝑡−𝑠−𝑑𝑖𝑗 𝑥𝑖≡𝑠<𝑡−𝑑𝑖𝑗[𝑡]
𝛼𝑖𝑗 ← 𝜆𝑡−𝑑𝑖𝑗
𝑡−1
𝛼𝑖𝑗+ 𝑥𝑖𝑡

How recently/often
spikes reachedfrom neuron i

Synaptic eligibility trace:
𝛼𝑖𝑗

𝑡𝑥𝑗 − 𝑋𝑗𝑠cf. Boltzmann machine𝑤𝑖𝑗 ← 𝑤𝑖𝑗 + 𝜂(𝑥𝑖 𝑥𝑗 − 𝑋𝑖 𝑋𝑗 )This talk• Boltzmann machine, Hebb’s rule, STDP
• Dynamic Boltzmann machine (DyBM) [O & Otsuka (2015)]• DyBM with LTP• DyBM with LTP and LTD• Experiments [O & Otsuka (2015), Dasgupta & O (2017)]

LTP: Weight is strengthened when the pre-synaptic
neuron fires shortly after the post-synaptic neuron
Bi & Poo (1998)
Dan & Poo (2006)

Pre-synaptic
neuron ixi

Synapse weakened(Long Term Depression)

Synapsewij

Post-synaptic
neuron jxj

Interpreting LTD as negative LTP for a connection frompost-synaptic neuron to pre-synaptic neuron
Bi & Poo (1998)
Dan & Poo (2006)

Time from pre-spike to post-spike

Time from post-spike to pre-spikePost-synaptic neuron

Pre-synaptic neuron
Image courtesy of dream designs at FreeDigitalPhotos.net

Dynamic Boltzmann machine as a limit of a sequence ofBoltzmann machines
Boltzmann machine for aT-th order Markov model𝑇→∞

Dynamic Boltzmann machine

Time

Weight from neuron i at time tto neuron j at time t + 𝛿𝛿

𝑤𝑖𝑗 = 𝑢𝑖𝑗 𝜆𝛿−𝑑𝑖𝑗
𝑑𝑖𝑗x[-T]

x[-1]

𝛿x[0]

Historical values Next value𝛿

𝑤𝑖𝑗 = −𝑣𝑖𝑗 𝜇−𝛿Dynamic Boltzmann machine as a limit of a sequence ofBoltzmann machines
Boltzmann machine for a
T-th order Markov model𝑇→∞

Dynamic Boltzmann machine

TimeWeight from neuron i at time tto neuron j at time t + 𝛿𝛿

𝑤𝑖𝑗 = 𝑢𝑖𝑗 𝜆𝛿−𝑑𝑖𝑗
𝑑𝑖𝑗
x[-T]

x[-1]

x[0]

Historical values Next value𝛿weight from post to pre𝛿

𝑤𝑖𝑗 = −𝑣𝑖𝑗 𝜇−𝛿Dynamic Boltzmann machine (LTP & LTD)
Conduction delay, dij

Neural eligibility trace:[𝑡]𝛾𝑖 ← 𝜇𝛾𝑖𝑡−1+ 𝑥𝑖

Synaptic eligibility trace:[𝑡]𝛼𝑖𝑗 ← 𝜆𝑡xi[t]xi[t-1] xi[t-2] xi[t-3]

xj[t]

𝑡−1
𝛼𝑖𝑗

+ 𝑥𝑖

𝑡−𝑑𝑖𝑗

−𝟏
𝑡−1𝛽𝑖𝑗≡𝑥𝑖

𝑡+𝑠𝜇𝑠

𝑠=−𝑑𝑖𝑗 +1

Probability for neuron j to fire at time t:
𝑋𝑗𝑡

≡𝑃[𝑡]
𝑥𝑗 = 1|𝑥 [:𝑡−1]=1 + exp1
𝑡−1

𝑖 −𝑢𝑖𝑗 𝛼𝑖𝑗

𝑡−1+ 𝑣𝑖𝑗 𝛽𝑖𝑗+ 𝑣𝑗𝑖 𝛾𝑖𝑡−1

Learning rule for Dynamic Boltzmann machine (LTP & LTD)
Conduction delay, dij

Neural eligibility trace:[𝑡]𝛾𝑖 ← 𝜇𝛾𝑖𝑡−1+ 𝑥𝑖

Synaptic eligibility trace:[𝑡]𝛼𝑖𝑗 ← 𝜆𝑡

xi[t-1] xi[t-2] xi[t-3]

xi[t]

xj[t]

−𝟏
𝑡−1

𝛽𝑖𝑗

≡𝑥𝑖

𝑡+𝑠𝜇𝑠

𝑠=−𝑑𝑖𝑗 +1

Stochastic gradient for LTD weight:
𝑡−1𝑣𝑖𝑗 ← 𝑣𝑖𝑗 + 𝜂 𝛽𝑖𝑗

𝑋𝑗

𝑡− 𝑥𝑗

𝑡

Stochastic gradient for LTP weight:𝑡−1

𝑢𝑖𝑗 ← 𝑢𝑖𝑗 + 𝜂 𝛼𝑖𝑗

𝑡𝑥𝑗 − 𝑋𝑗𝑡+ 𝜂 𝛾𝑗

𝑡−1

𝑋𝑖

𝑡− 𝑥𝑖

𝑡

𝑡−1
𝛼𝑖𝑗

+ 𝑥𝑖

𝑡−𝑑𝑖𝑗

We can use multiple eligibility traces with varying decay
rates for long term memory
Neural eligibility trace:[𝑡]𝛾𝑖 ← 𝜇𝛾𝑖

𝑡−1+ 𝑥𝑖

Conduction delay, dij

Synaptic eligibility trace:𝑡xi[t]

xi[t-1] xi[t-2] xi[t-3]

xj[t]

[𝑡]
𝛼𝑖𝑗 ← 𝜆−𝟏𝑡−1𝛽𝑖𝑗ℓ≡𝑥𝑖𝑡+𝑠𝜇ℓ𝑠𝑠=−𝑑𝑖𝑗 +1ℓ = 1,2, …

𝑘 = 1,2, …

𝑡−1
𝛼𝑖𝑗+ 𝑥𝑖𝑡−𝑑𝑖𝑗This talk• Boltzmann machine, Hebb’s rule, STDP
• Dynamic Boltzmann machine [O & Otsuka (2015)]• Experiments [O & Otsuka (2015), Dasgupta & O (2017)]

Presenting sequences of 7-bit patterns to a dynamic
Boltzmann machine

DyBM as associative memory for sequential patterns[O & Otsuka (2015)]

Giving DyBM a partialsequence as a cue

DyBM completes theremaining sequence

Energy landscape of DyBM evolves
depending on the past patterns
Images from www.nature.com/articles/srep14149

DyBM detecting anomaly [O & Otsuka (2015)]
Image from www.nature.com/articles/srep14149

Negative log-likelihood

Beginning of anomalous sequence

DyBM learning a generative model of human evolution[O & Otsuka (2015)]
20 neurons

Time

Images from www.nature.com/articles/srep14149

DyBM learning a generative model of Ich bin ein Musikante[O & Otsuka (2015)]
Images from www.nature.com/articles/srep14149• 12 neurons• Sounds prepared by Shohei Ohsawa& Yachiko ObaraComparison between DyBM and LSTM
[Dasgupta & O (AAAI-17)]
Root mean squared error after learning for 20 epochs
Model

Retail price of gasoline & diesel• 8 dimensions• 20 hidden units

Sunspot number• 1 dimension• 50 hidden units

LSTM

0.067

0.073

DyBM (delay=2)

0.058

0.082

DyBM (delay=3)

0.056

0.077

DyBM (delay=4)

0.060

0.077

Systematic approaches to tuning
DyBM’s hyperparameters[Dasgupta, Yoshizumi, O (ICPR 2016)][O & Dasgupta (IBM R&D J. 2017)]

DyBM and LSTM have comparable structures

Comparison between DyBM and LSTM
[Dasgupta & O (AAAI-17)]
LearningPrediction

DyBM
Time per epoch = 0.7 sec.

RMSE = 0.077

DyBM learns 16 times faster

Comparable accuracy
LSTM (Keras library)

Time per epoch = 11.2 sec.

RMSE = 0.073

Learning and predicting sunspot number

Images from IBM Research Report RT0975

Summary: Dynamic Boltzmann machine
Spike-timing dependent plasticity

Dynamic Boltzmann machine

Bi & Poo (1998)
Dan & Poo (2006)

MLE

Formal limit

RefineBoltzmann machinexi

wijxj

Hebb’s rule

MLE

Cells that fire together, wire togetherWe provide theoretical underpinnings for STDP
Hebb’s rule (’49)

STDP (’90s)
Theoretical underpinnings- Hopfield network (’82)- Boltzmann machine (’83)

Theoretical underpinnings- Dynamic Boltzmann machine[This talk]

Motivated artificial
neural networks- Perceptron (’58)
Failure

1950

1960

1970

Success- Deep learning
1980

1990

2000

2010

Successfulapplications

Dynamic Boltzmann machineReferences• Takayuki Osogami and Makoto Otsuka, “Seven neurons memorizing sequences ofalphabetical images via spike-timing dependent plasticity,” Scientific Reports, 5, 14149 (2015).
www.nature.com/articles/srep14149• Takayuki Osogami and Makoto Otsuka, Learning dynamic Boltzmann machines with spiketiming dependent plasticity, IBM Research Report, RT0967 (2015). arxiv.org/abs/1509.08634• Sakyasingha Dasgupta, Takayuki Yoshizumi, and Takayuki Osogami, “Regularized dynamicBoltzmann machine with delay pruning for unsupervised learning of temporal sequences,”
ICPR 2016. arxiv.org/abs/1610.01989• Takayuki Osogami, “Learning binary or real-valued time-series via spike-timing dependent
plasticity,” Computing with Spikes NIPS 2016 Workshop, to appear.
• Sakyasingha Dasgupta and Takayuki Osogami, “Nonlinear dynamic Boltzmann machines fortime-series prediction,” AAAI-17. Extended research report available at goo.gl/Vd0wna• Takayuki Osogami and Sakyasingha Dasgupta, “Learning the values of the hyperparametersof a dynamic Boltzmann machine,” IBM Journal of Research and Development, 61(4/5), 2017,to appear.

This research was supported by CREST, JST