A Biologically Plausible Learning Algorithm
for Neural Networks

Dmitry Krotov
MIT-IBM Watson AI Lab
IBM ResearchDoes brain do deep learning?
• Architectural similarity between CNNs and visual cortex. ZeilerandR.
FergusSimilarity of the receptive fields of CNNs and direction selective cells•(a)Zeiler and Fergus, 2014(b)Hubel Wiesel, 1959(c)

Does brain do backpropagation?
output<latexit sha1_base64="FcS3rQ7MVQhyaOxeQmpYJ7mG1fg=">AAAB9HicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbbSbt0s4m7k2IJ/R1ePCji1R/jzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmjjVHBo8lrFuB8yAFAoaKFBCO9HAokBCKxjdzvzWGLQRsXrASQJ+xAZKhIIztJLfRXjCLE4xSXHaK1fcqjsHXSVeTiokR71X/ur2Y55GoJBLZkzHcxP0M6ZRcAnTUjc1kDA+YgPoWKpYBMbP5kdP6ZlV+jSMtS2FdK7+nshYZMwkCmxnxHBolr2Z+J/XSTG89jOh7Eug+GJRmEqKMZ0lQPtCA0c5sYRxLeytlA+ZZhxtTiUbgrf88ippXlQ9t+rdX1ZqN3kcRXJCTsk58cgVqZE7UicNwskjeSav5M0ZOy/Ou/OxaC04+cwx+QPn8wfUH5LG</latexit>

L=
hAhA<latexit sha1_base64="CEfz7hUFKNhtGfHqjpF5cbSDK4Y=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeqF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHM0nQj+hQ8pAzaqz0MOpf98sVt+rOQVaJl5MK5Gj0y1+9QczSCKVhgmrd9dzE+BlVhjOB01Iv1ZhQNqZD7FoqaYTaz+anTsmZVQYkjJUtachc/T2R0UjrSRTYzoiakV72ZuJ/Xjc14ZWfcZmkBiVbLApTQUxMZn+TAVfIjJhYQpni9lbCRlRRZmw6JRuCt/zyKmldVD236t1fVuo3eRxFOIFTOAcPalCHO2hAExgM4Rle4c0Rzovz7nwsWgtOPnMMf+B8/gALkI2g</latexit><latexit sha1_base64="+M338Vj2HHmdRYnMwU1ShgJTTQA=">AAACJXicbVDLSgNBEJz1bXxFPXoZDIIeDLsi6EFB9OLBg4JRIRtD76QTB2d2l5leMSz7M178FS8eFBE8+StOHgdfBQPVVd30dEWpkpZ8/8MbGR0bn5icmi7NzM7NL5QXly5skhmBNZGoxFxFYFHJGGskSeFVahB0pPAyuj3q+Zd3aKxM4nPqptjQ0IllWwogJzXLeyf7PLSZDpXUkmwzJLynvAUEBV8fFElGaUYF3+SDmsB0kIqN661mueJX/T74XxIMSYUNcdosv4atRGQaYxIKrK0HfkqNHAxJobAohZnFFMQtdLDuaAwabSPvX1nwNae0eDsx7sXE++r3iRy0tV0duU4NdGN/ez3xP6+eUXu3kcvYXYmxGCxqZ4pTwnuR8ZY0KEh1HQFhpPsrFzdgQJALtuRCCH6f/JdcbFUDvxqcbVcODodxTLEVtsrWWcB22AE7ZqesxgR7YE/shb16j96z9+a9D1pHvOHMMvsB7/MLEnGmNQ==</latexit><latexit sha1_base64="CEfz7hUFKNhtGfHqjpF5cbSDK4Y=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeqF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHM0nQj+hQ8pAzaqz0MOpf98sVt+rOQVaJl5MK5Gj0y1+9QczSCKVhgmrd9dzE+BlVhjOB01Iv1ZhQNqZD7FoqaYTaz+anTsmZVQYkjJUtachc/T2R0UjrSRTYzoiakV72ZuJ/Xjc14ZWfcZmkBiVbLApTQUxMZn+TAVfIjJhYQpni9lbCRlRRZmw6JRuCt/zyKmldVD236t1fVuo3eRxFOIFTOAcPalCHO2hAExgM4Rle4c0Rzovz7nwsWgtOPnMMf+B8/gALkI2g</latexit>

X

vi

Wµi =
<latexit sha1_base64="KURt72zyY5Cufm4cik/JfXRgBtM=">AAACVHicbVHBThsxEPUuUCBtaShHLiMiJHpotIsqwQUJ0R44cACpIUhxFM06s8GK17uyvZGiVT4SDkh8CZce8IbQUmBkS8/vzdPYz0mhpHVRdB+ES8srH1bX1hsfP33e+NLc/Hpp89II6ohc5eYqQUtKauo46RRdFYYwSxR1k/HPWu9OyFiZ699uWlA/w5GWqRToPDVojvkvUg6hO6h4VoKcwRF8Bz5BQ4WVKtfAU4Oi4gUaJ1HB2ewf/muqXS88HPiJHO1xaD+v+vxt0GxF7Whe8BbEC9BiizofNG/5MBdlRtoJhdb24qhw/aqeLhTNGry0VKAY44h6HmrMyPareSgz2PXMENLc+K0dzNmXjgoza6dZ4jszdNf2tVaT72m90qWH/UrqonSkxdOgtFTgcqgThqE0JJyaeoDCSH9XENfoQ3T+Hxo+hPj1k9+Cy/12HLXjix+t45NFHGtsm+2wPRazA3bMTtk56zDBbthDwIIguAv+hEvhylNrGCw8W+y/CjceAccfsPw=</latexit>

target)

⇣

⌘

data

hµ

Wµi

(output

2

@L
"
="
@Wµi

The update rule is non-local...

Biological plasticity rules!

!

Wµi = f<latexit sha1_base64="bXaxTlWbyXHazzsJjHMXMS/IMCM=">AAACF3icbZBNS8MwGMfT+TbnW9Wjl+AQJkhpRdCLMKYHvU1wL7CWkmbpFpa0JUmFUfYtvPhVvHhQxKve/DamWw+6+UDIn9//eUief5AwKpVtfxulpeWV1bXyemVjc2t7x9zda8s4FZi0cMxi0Q2QJIxGpKWoYqSbCIJ4wEgnGF3lfueBCEnj6F6NE+JxNIhoSDFSGvmm5V4TphDs+JnLU0gn8BKGboMOai689TU6yW8KXZjDY9+s2pY9LbgonEJUQVFN3/xy+zFOOYkUZkjKnmMnysuQUBQzMqm4qSQJwiM0ID0tI8SJ9LLpXhN4pEkfhrHQJ1JwSn9PZIhLOeaB7uRIDeW8l8P/vF6qwgsvo1GSKhLh2UNhyqCKYR4S7FNBsGJjLRAWVP8V4iESCCsdZUWH4MyvvCjap5ZjW87dWbXeKOIogwNwCGrAAeegDm5AE7QABo/gGbyCN+PJeDHejY9Za8koZvbBnzI+fwC2TJ0Z</latexit>

Wµi

⇣

⌘

Wµi = f Iµ , Ii

Iµ , Ii<latexit sha1_base64="e7FH5DlbW/z+9TEILaCMyg65YYU=">AAACFHicjZDLSgMxFIYz9VbrbdSlm2ARKkqZEUE3QqkudFfBXqAzDJk004YmmSHJCGXoQ7jxVdy4UMStC3e+jZm2CxUFD4T8fP85nOQPE0aVdpwPqzA3v7C4VFwurayurW/Ym1stFacSkyaOWSw7IVKEUUGammpGOokkiIeMtMPhee63b4lUNBY3epQQn6O+oBHFSBsU2AfeBWEawXaQeTyFdAzPYOTVab8CrwJDDj1zU5iT/cAuu1VnUvBvUQazagT2u9eLccqJ0Jghpbquk2g/Q1JTzMi45KWKJAgPUZ90jRSIE+Vnk0+N4Z4hPRjF0hyh4YR+ncgQV2rEQ9PJkR6on14Of/O6qY5O/YyKJNVE4OmiKGVQxzBPCPaoJFizkREIS2reCvEASYS1ybH0vxBaR1XXqbrXx+VafRZHEeyAXVABLjgBNXAJGqAJMLgDD+AJPFv31qP1Yr1OWwvWbGYbfCvr7RPYdpwm</latexit>

Iµ

information available to cell µIµIi

information available to cell i<latexit sha1_base64="nz39xb4PEkuJlPwHDmHeJxN3cz4=">AAACG3icbVDLSgMxFM3UV62vqks3wVZwY5kpgi6LbnRXwT6gLUMmvdOGJjNDkimWof/hxl9x40IRV4IL/8ZMOwttvRA4nHPvzT3HizhT2ra/rdzK6tr6Rn6zsLW9s7tX3D9oqjCWFBo05KFse0QBZwE0NNMc2pEEIjwOLW90neqtMUjFwuBeTyLoCTIImM8o0YZyi9VbtytifIa7Gh50wgI/lGKmYTImjBOzCOsQU+Acl01reeoWS3bFnhVeBk4GSiirulv87PZDGgsINOVEqY5jR7qXEKkZ5TAtdGMFEaEjMoCOgQERoHrJzNsUnximj81V5gUaz9jfEwkRSk2EZzrN3UO1qKXkf1on1v5lzxiOYg0BnX/kxzw1mwaF+0wC1XxiAKGSmVsxHRJJqDZxFkwIzqLlZdCsVhy74tydl2pXWRx5dISO0Sly0AWqoRtURw1E0SN6Rq/ozXqyXqx362PemrOymUP0p6yvHy4IoNs=</latexit><latexit sha1_base64="GikZ8dA08x4Gmivbof3sQpeTk2I=">AAAB8HicjVDLSgNBEOyNrxhfUY9eBoPgKeyKEI9BLx4jmIckS5idzCZDZmaXmV4hLPkKLx4U8ernePNvnDwOKgoWNBRV3XR3RakUFn3/wyusrK6tbxQ3S1vbO7t75f2Dlk0yw3iTJTIxnYhaLoXmTRQoeSc1nKpI8nY0vpr57XturEj0LU5SHio61CIWjKKT7tr9vKcyIqb9ciWo+nOQv0kFlmj0y++9QcIyxTUySa3tBn6KYU4NCib5tNTLLE8pG9Mh7zqqqeI2zOcHT8mJUwYkTowrjWSufp3IqbJ2oiLXqSiO7E9vJv7mdTOML8Jc6DRDrtliUZxJggmZfU8GwnCGcuIIZUa4WwkbUUMZuoxK/wuhdVYN/Gpwc16pXy7jKMIRHMMpBFCDOlxDA5rAQMEDPMGzZ7xH78V7XbQWvOXMIXyD9/YJp3+QTA==</latexit><latexit sha1_base64="IdB9RXM4PqQLTGvygWPlalLjpxQ=">AAACF3icbVDLSsNAFJ34rPUVdelmsBXcGBIRdFl0o7sK9gFtCJPpTTt08mBmUiylf+HGX3HjQhG3uvNvnKRZaOuFgcM599655/gJZ1LZ9rextLyyurZe2ihvbm3v7Jp7+00Zp4JCg8Y8Fm2fSOAsgoZiikM7EUBCn0PLH15nemsEQrI4ulfjBNyQ9CMWMEqUpjzTuvUYPsVdBQ9qwqIgFmGuYDIijBO9BqsYU+AcV1l16pkV27LzwovAKUAFFVX3zK9uL6ZpCJGinEjZcexEuRMiFKMcpuVuKiEhdEj60NEwIiFId5L7muJjzfSwvkm/SOGc/T0xIaGU49DXnfrqgZzXMvI/rZOq4NLVdpNUQURnHwUpz6xmIeEeE0AVH2tAqGD6VkwHRBCqdJRlHYIzb3kRNM8sx7acu/NK7aqIo4QO0RE6QQ66QDV0g+qogSh6RM/oFb0ZT8aL8W58zFqXjGLmAP0p4/MHyN+fCQ==</latexit>

cell µcell i⇣⌘information available to cell µ<latexit sha1_base64="bZp+yNSQ2LfDXXPbopOtPGtKRJY=">AAACG3icjVDLSsNAFJ34rPUVdelmsAhuLEkRdFl0o7sK9gFNCJPpTTt0Jgkzk2Ip/Q83/oobF4q4Elz4N07SLlQUPDBwOefeO+eeMOVMacf5sBYWl5ZXVktr5fWNza1te2e3pZJMUmjShCeyExIFnMXQ1Exz6KQSiAg5tMPhRa63RyAVS+IbPU7BF6Qfs4hRog0V2LWrwBMZPsaehls9YXGUSFFomIwI48QswjrBFDifejjvDeyKW3UK4L+LCpqjEdhvXi+hmYBYU06U6rpOqv0JkZpRDtOylylICR2SPnRNGRMByp8Ut03xoWF62LgyL9a4YL9OTIhQaixC02l8D9RPLSd/07qZjs58c3CaaYjp7KMo4/mxeVC4xyRQzcemIFQy4xXTAZGEahNn+X8htGpV16m61yeV+vk8jhLaRwfoCLnoFNXRJWqgJqLoDj2gJ/Rs3VuP1ov1OmtdsOYze+gbrPdPg6GhEg==</latexit><latexit sha1_base64="GO6htCvnHU4HS/Mgo9aDEhr4Lg4=">AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJMw5STrpU9MoVt+rOQJeJl5MKyVHvlb+6/ZinESjkkhnT8dwE/YxpFFzCpNRNDSSMj9gAOpYqFoHxs9nVE3pilT4NY21LIZ2pvycyFhkzjgLbGTEcmkVvKv7ndVIML/1MqCRFUHy+KEwlxZhOI6B9oYGjHFvCuBb2VsqHTDOONqiSDcFbfHmZNM+qnlv1bs8rtas8jiI5IsfklHjkgtTIDamTBuFEk2fySt6cR+fFeXc+5q0FJ585JH/gfP4AutSSpA==</latexit><latexit sha1_base64="AWC6qH/yDtJmzXlDxhPE4mEwcV4=">AAAB+XicbVBNS8NAEN34WetX1KOXxSJ4KokIeix68VjBfkATymY7bZduNmF3Uiyh/8SLB0W8+k+8+W/ctjlo64OBx3szzMyLUikMet63s7a+sbm1Xdop7+7tHxy6R8dNk2SaQ4MnMtHtiBmQQkEDBUpopxpYHEloRaO7md8agzYiUY84SSGM2UCJvuAMrdR13QDhCXMOUk4DGsRZ1614VW8Oukr8glRIgXrX/Qp6Cc9iUMglM6bjeymGOdMouIRpOcgMpIyP2AA6lioWgwnz+eVTem6VHu0n2pZCOld/T+QsNmYSR7YzZjg0y95M/M/rZNi/CXOh0gxB8cWifiYpJnQWA+0JDRzlxBLGtbC3Uj5kmnG0YZVtCP7yy6ukeVn1var/cFWp3RZxlMgpOSMXxCfXpEbuSZ00CCdj8kxeyZuTOy/Ou/OxaF1zipkT8gfO5w/LJ5O+</latexit><latexit sha1_base64="cn3fwZFdFMfA29NuJDnBLp7F9a0=">AAACF3icjVDLSsNAFJ3UV62vqEs3g0VwY0hE0GXRje4q2Ac0IUymN+3QyYOZSbGE/oUbf8WNC0Xc6s6/cZJ2oaLggYHLOffeuecEKWdS2faHUVlYXFpeqa7W1tY3NrfM7Z22TDJBoUUTnohuQCRwFkNLMcWhmwogUcChE4wuCr0zBiFZEt+oSQpeRAYxCxklSlO+aV35DB9hV8GtylkcJiIqFUzGhHGi12CVYAqcT13MsG/WHcsugf8u6miOpm++u/2EZhHEinIiZc+xU+XlRChGOUxrbiYhJXREBtDTZUwikF5e+priA830sb5Jv1jhkv06kZNIykkU6E599VD+1AryN62XqfDM03bTTEFMZx+FGS+sFiHhPhNAFZ/oglDB9K2YDokgVOkoa/8LoX1sObblXJ/UG+fzOKpoD+2jQ+SgU9RAl6iJWoiiO/SAntCzcW88Gi/G66y1YsxndtE3GG+fHfKfQA==</latexit>

Ii

information available to cell i

The update rule is local

Supervised backpropagation learning versus
biological learning
Cat Wonʼt Stop Meowing? 7 Reasons For All That Cat Meowing - Catster

Join (https://www.catster.com/register/)

9/3/18, 3(10 PM

Log InSearch Catster
SUBSCRIBE TO CATSTER MAGAZINE (HTTPS://SSL.DRGNETWORK.COM/ECOM/BEL/APP/LIVE/SUBSCRIPTIONS?ORG=BEL&PUBL=CS&KEY_CODE=LDACS01&TYPE=S)(https://www.catster.com/)

CAT BEHAVIOR (/CAT-BEHAVIOR/)KITTENS (/TOPIC/KITTENS/)

CAT FOOD (TOPIC/CAT-FOOD/)LIFESTYLE (/LIFESTYLE/)

VIDEOS (/TOPIC/CATSTER-ORIGINAL-VIDEO/)Customer Service for Subscribers(https://ssl.drgnetwork.com/ecom/bel/app/live/subcustserv?
pagemode=start&org=BEL&publ=CS&php=Y)

Free Images : person, boy, cute, child, blue, butterfly, baby, materni…ions, human action 4861x3241 - - 119045

HEALTH & CARE (/TOPIC/HEALTH-CARE/)

CAT BREEDS (/PUREBRED-AND-HYBRID-CAT-BREEDS/)COMMUNITY (HTTPS://COMMUNITY.CATSTER.COM/)

Home (/) » Lifestyle(https://www.catster.com/lifestyle/)
Next on The Scoop: What Are the Signs of Stress in Cats?
(https://www.catster.com/cat-behavior/signs-of-stress-incats)

cat

Also:

CATSTER TIPS

cat
Get Catster in Your Inbox!

Cute Cat Pics - Home | Facebook

9/3/18, 3(11 PM

THE SCOOP
Email or Phone

Stay informed! Get tips and exclusive deals.

Password

Belgium Mandates
Nationwide Spay and
Neuter Policy

Log In( 0

Forgot account?

Enter your email addressGO!

THE SCOOP

What’s Mew at Catster:
September 2018 Cat
EventsCute Cat Pics
Home

Fascinating Facts Aboutthe BEHAVIOR
Savannah Cat - Catster
CAT

About
Photo Gifts

9/3/18, 3(18 PM

Send Message
Cat Love Bites — What Shop Now
Do
They
Mean
and
Why
Join (https://www.catster.com/register/)
Log
In
PhotosSearch for posts on this Page
Do They Happen?
Like

PhotosVideos
Posts

Share

Suggest Edits

WelcomeCommunity

Store

See All

Search Catster
310,304 people like thisSUBSCRIBE TO CATSTER MAGAZINE (HTTPS://SSL.DRGNETWORK.COM/ECOM/BEL/APP/LIVE/SUBSCRIPTIONS?ORG=BEL&PUBL=CS&KEY_CODE=LDACS01&TYPE=S)

Info and Ads

309,356 people follow thisAbout

Cat Won’t Stop Meowing? 7 Reasons
For All That Cat Meowing

See All

Typically replies within a few hours
Contact Cute Cat Pics on Messenger(https://www.catster.com/)www.cats.animalgoss.com

CAT BEHAVIOR (/CAT-BEHAVIOR/)

Community

CAT FOOD (TOPIC/CAT-FOOD/)HEALTH & CARE (/TOPIC/HEALTH-CARE/)

PeopleKITTENS (/TOPIC/KITTENS/)

310,304
likes
LIFESTYLE(/LIFESTYLE/)
Related Pages

VIDEOS (/TOPIC/CATSTER-ORIGINAL-VIDEO/)

cat



cat

We Love Cats

Cat won't stop meowing? There might be more to your catmeowing all the time than nonstop vocalization. Find out if allthat cat meowing signals something serious like a medical
condition.

News & Media Website

See All

CAT BREEDS (/PUREBRED-AND-HYBRID-CAT-BREEDS/)COMMUNITY (HTTPS://COMMUNITY.CATSTER.COM/)

Home (/) » Cats 101

vs.

Customer Service for Subscribers(https://ssl.drgnetwork.com/ecom/bel/app/live/subcustserv?
pagemode=start&org=BEL&publ=CS&php=Y)

Create a Page

(https://www.catster.com/cats-Free for pe
No attributi

101/)

Dakota The Bunnyfaced Lion

Videosfile-20171109-13296-mmbjtn.jpg 1,000×773 pixels& CC0 Pu

Community

Community

Next onPetThe Scoop: Acromegaly in Cats: Learn theSymptoms, Diagnosis and Treatment(https://www.catster.com/cat-health-care/acromegaly-in-cats)
Cute cats
App Page

9/3/18, 3(13 PM

BREEDShttps://www.facebook.com/Cute-Cat-Pics-559375390879965/

Page 1 of 4

Also:

Get Catster in Your Inbox!

THE SCOOP

JaneA Kelley (https://www.catster.com/author/janea-kelley) | Jul
20th 2018

Belgium Mandates
Nationwide Spay and
Neuter Policy

Stay informed! Get tips and exclusive deals.
Enter your email address

GO!

%F

THE SCOOP

What’s Mew at Catster:
September 2018 Cat
Events!
(https://www.facebook.com/sharer/sharer.php?
u=https://www.catster.com/lifestyle/catwont-stop-meowingreasons-for-cat"meowing&t=Cat Won\'t
Stop Meowing? 7
Reasons For All ThatCat
Meowing&redirect_uri=https://www.catster.com?
sharing-thankyou=yes)

Electrical/
Tooling Parts

CAT BEHAVIORMISUMI USA has
Japanese Brand
Components delivered
quickly to reduce yourmachine downtime

Cat Love Bites — What
Do They Mean and Why
Do They Happen?

#(https://plus.google.com/share?
$
url=https://www.catster.com/lifestyle/catwont-stop-meowingreasons-for-catmeowing) Fascinating Facts About the

Popular

The Cat Doctor of
NashuaExceptional care for catsin a feline-centeredenvironment.
catdoctors.com

OPEN

Savannah Cat

4861x3241! unknown camera

How long does a Savannah cat typically live? Will a Savannah
cat get along with kids? Does this breed like to walk on a leash?
Let’s get to know more about the Savannah cat right here.

https://www.catster.com/lifestyle/cat-wont-stop-meowing-reasons-for-cat-meowing
https://images.theconversation.com/files/194020/original/file-20171109-13296-mmbjtn.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip

Page 1 of 1

heavily supervised
greedy on labeled data
Kim Campbell Thornton (https://www.catster.com/author/kimcampbell-thornton) | Jul 12th 2018

mostly unsupervised
very03/18/2017
few labeled examples
unknown lens
" Resolution

Page 1 of 20!

(https://www.facebook.com/sharer/sharer.php?
u=https://www.catster.com/cats101/savannah-catabout-thisbreed&t=Fascinating
Facts About theSavannah
Cat&redirect_uri=https://www.catster.com?
sharing-thankyou=yes)"#(https://plus.google.com/share?
url=https://www.catster.com/cats101/savannah-catabout-this-breed)$

If you’ve ever daydreamed about living with a wild cat but recognize the dangers and drawbacks, theSavannah (http://www.catster.com/catbreed/savannah) cat may be just the kitty for you.

https://www.catster.com/cats-101/savannah-cat-about-this-breed

Page 1 of 6

# Uploaded

Original
Size
1600x900
1280x1024
800x600

Main question:
Given unsupervised aspect of learning and locality ofsynaptic plasticity rules, can we engineer a learningalgorithm that leads to a good generalization
performance?

features of the data. Neither BCM algorithm, nor the plasticity rule (2) address this issue
selectivity. In order to do this, we introduce a neural network with global inhibition
hidden neurons and dynamical equations
X
dhµ
hµ
N
⌧
= Iµ winhr(h⌫ ) hµ , where IµX= hWµ , vi
dt
Wµi
⌫6=µ
Iµ =
Wµi viviIn this equation the activities of the hidden neurons are denoted by hi=1
µ , r(hµ ) = max(corresponding firing rates (we use a ReLU), Iµ is the input current from the visible lparameterthatdefinesthestrengthoftheglobalinhibition,
constant⌧⌧⌧
definestLinput vi
hµ
h3
g(h)
I1
time scale of individual
neurons.
1
I
2
The pipeline of the unsupervised part of the algorithm in shown in Fig.2. Using the wIh3
1
raw input is converted into a set of input currents
Iµ . These currents drive the dynam
h
5
h
h
h
I
3set so th
5
1
4
units, as is shown in Fig.2. The strength parameteroftheglobalinhibition
is
t
h2
h
⇤
h
I
5
state only a small fraction of hidden units have positive activity (in this case h3 , h1 ,h4values of this steady state activities are then used as arguments in the non-linear learn
function g(h) in (2). This function implements temporal competition between the pa
it is positive for the activitiesW
exceedingh⇤ , andnegative for the activitie)vi Hebbianlearningµi ⇠ g(haµthreshold
0 < hµ < h⇤ . Activities that are below zero do not contribute to training. The intuitiv

Learning algorithm<latexit sha1_base64="Lymn/vkqiD2R1qIZGKk2RGPmkFM=">AAACEXicbVDLSsNAFJ3UV62vqEs3g0XoqiQi6KZQdKMbqWAf0MQwmU7aoTOTMDMplNBfcOOvuHGhiFt37vwbp20W2nrgwuGce7n3njBhVGnH+bYKK6tr6xvFzdLW9s7unr1/0FJxKjFp4pjFshMiRRgVpKmpZqSTSIJ4yEg7HF5N/faISEVjca/HCfE56gsaUYy0kQK7chN4PK1BT6XcY5RTrYKM1tzJwy1sB5nxIJ3AUUADu+xUnRngMnFzUgY5GoH95fVinHIiNGZIqa7rJNrPkNQUMzIpeakiCcJD1CddQwXiRPnZ7KMJPDFKD0axNCU0nKm/JzLElRrz0HRypAdq0ZuK/3ndVEcXfkZFkmoi8HxRlDKoYziNB/aoJFizsSEIS2puhXiAJMLahFgyIbiLLy+T1mnVdaru3Vm5fpnHUQRH4BhUgAvOQR1cgwZoAgwewTN4BW/Wk/VivVsf89aClc8cgj+wPn8Asi6c6g==</latexit><latexit sha1_base64="xC0UVIwdebnFI8vROjkz5QIwZ9E=">AAACC3icbVDLSsNAFL2pr1pfUZduhhahbkoigi6LunBZwT6gCWEynbZDZ5IwMymU0L0bf8WNC0Xc+gPu/BunbRbaeuDCmXPuZe49YcKZ0o7zbRXW1jc2t4rbpZ3dvf0D+/CopeJUEtokMY9lJ8SKchbRpmaa004iKRYhp+1wdDPz22MqFYujBz1JqC/wIGJ9RrA2UmCXvVvKNUbtIPNEitgUeYoJNKgOA/M+Q+OABXbFqTlzoFXi5qQCORqB/eX1YpIKGmnCsVJd10m0n2GpGeF0WvJSRRNMRnhAu4ZGWFDlZ/NbpujUKD3Uj6WpSKO5+nsiw0KpiQhNp8B6qJa9mfif1011/8rPWJSkmkZk8VE/5UjHaBYM6jFJieYTQzCRzOyKyBBLTLSJr2RCcJdPXiWt85rr1Nz7i0r9Oo+jCCdQhiq4cAl1uIMGNIHAIzzDK7xZT9aL9W59LFoLVj5zDH9gff4ALzKZ2g==</latexit>

Synaptic plasticity rule
⇣

dWµi
⌧L
= g hµdt<latexit sha1_base64="Gh577FU43Rnrq1XXEWdFYp4qlKU=">AAACYnicbVHPT9swGHUy2CAwWsaRHSwqJDhQJdOkcalUsQsHhJhEKVITIsdxWit2Etmfkaoo/yS3nXbZH4ITOsGAT7L1/N73w35OKsE1+P5vx/2wtv7x08amt7X9eafX3/1yo0ujKJvQUpTqNiGaCV6wCXAQ7LZSjMhEsGmS/2z16T1TmpfFNSwrFkkyL3jGKQFLxf1lCMTEFzjMFKF1iqdxHUqDedPUKTR4hOfhGZ8fLWLLtui43Wb4Pub4BHdSqI0MBZccdFzno6C5u/zXJW9sYt6VPTduj5HneXF/4A/9LvBbEKzAAK3iKu4/hGlJjWQFUEG0ngV+BVFNFHAqWOOFRrOK0JzM2czCgkimo7qzqMGHlklxViq7CsAd+7KiJlLrpUxspiSw0K+1lnxPmxnITqOaF5UBVtCnQZkRGErc+o1TrhgFsbSAUMXtXTFdEGs22F9pTQheP/ktuPk2DPxh8Ov7YHy2smMD7aMDdIQC9AON0Tm6QhNE0R9n3dlxes5f13N33b2nVNdZ1eyh/8L9+gj58bRd</latexit>

N
X<latexit sha1_base64="CMezGb8uWsLbjNdwbBiSmoBLHPE=">AAACD3icbVDLSsNAFJ3UV42vqEs3g0VxVZIi6KZQdONKKtgHNGmYTKft0JlJmJkIJeQP3Pgrblwo4tatO//GpM1CWw9cOJxzL/feE0SMKm3b30ZpZXVtfaO8aW5t7+zuWfsHbRXGEpMWDlkouwFShFFBWppqRrqRJIgHjHSCyXXudx6IVDQU93oaEY+jkaBDipHOJN86dVXMXUY51cpPaN1J+7ew4ycujyFN+zVYh45pmr5Vsav2DHCZOAWpgAJN3/pyByGOOREaM6RUz7Ej7SVIaooZSU03ViRCeIJGpJdRgThRXjL7J4UnmTKAw1BmJTScqb8nEsSVmvIg6+RIj9Wil4v/eb1YDy+9hIoo1kTg+aJhzKAOYR4OHFBJsGbTjCAsaXYrxGMkEdZZhHkIzuLLy6Rdqzp21bk7rzSuijjK4AgcgzPggAvQADegCVoAg0fwDF7Bm/FkvBjvxse8tWQUM4fgD4zPH+G7mpE=</latexit>

i=1

⌘h

vi

2
Wµi = 1

N
⇣X

k=1

⌘

Wµk vk Wµi

i

The weights of each hidden
unit dynamically converge tothe surface of a unit sphereStacking the layers
S<latexit sha1_base64="0INrc2Ub1nZE/anxyraz3C6/Kqw=">AAAB6HicdVDJSgNBEK2JW4xb1KOXxiB4CjMixGPQi8cEzQLJEHo6NUmbnp6hu0cIQ77AiwdFvPpJ3vwbO4sQtwcFj/eqqKoXJIJr47ofTm5ldW19I79Z2Nre2d0r7h80dZwqhg0Wi1i1A6pRcIkNw43AdqKQRoHAVjC6mvqte1Sax/LWjBP0IzqQPOSMGivVb3rFkld2ZyDuL/JllWCBWq/43u3HLI1QGiao1h3PTYyfUWU4EzgpdFONCWUjOsCOpZJGqP1sduiEnFilT8JY2ZKGzNTliYxGWo+jwHZG1Az1T28q/uV1UhNe+BmXSWpQsvmiMBXExGT6NelzhcyIsSWUKW5vJWxIFWXGZlNYDuF/0jwre27Zq5+XqpeLOPJwBMdwCh5UoArXUIMGMEB4gCd4du6cR+fFeZ235pzFzCF8g/P2Ca+ljNg=</latexit><latexit

W(2)<latexit sha1_base64="bQ/FMWUOGdF0rk6ScnJXyJ6SJQE=">AAAB7nicbVBNS8NAEJ34WetX1aOXxSLUS0mKoMeiF48VbFNoa9lsN+3SzSbsToQS+iO8eFDEq7/Hm//GbZuDtj4YeLw3w8y8IJHCoOt+O2vrG5tb24Wd4u7e/sFh6ei4ZeJUM95ksYx1O6CGS6F4EwVK3k40p1EguR+Mb2e+/8S1EbF6wEnCexEdKhEKRtFKvv+YVWoX036p7FbdOcgq8XJShhyNfumrO4hZGnGFTFJjOp6bYC+jGgWTfFrspoYnlI3pkHcsVTTippfNz52Sc6sMSBhrWwrJXP09kdHImEkU2M6I4sgsezPxP6+TYnjdy4RKUuSKLRaFqSQYk9nvZCA0ZygnllCmhb2VsBHVlKFNqGhD8JZfXiWtWtVzq979Zbl+k8dRgFM4gwp4cAV1uIMGNIHBGJ7hFd6cxHlx3p2PReuak8+cwB84nz9lnI7w</latexit>

W<latexit sha1_base64="xkxogPCQIQ12jDQQEXn9T1YRBZE=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh6LXjxWsE2hrWWznbRLN5uwuxFK6I/w4kERr/4eb/4bt20O2vpg4PHeDDPzgkRwbVz32ymsrW9sbhW3Szu7e/sH5cOjlo5TxbDJYhGrdkA1Ci6xabgR2E4U0igQ6Afj25nvP6HSPJYPZpJgL6JDyUPOqLGS7z9mVe982i9X3Jo7B1klXk4qkKPRL391BzFLI5SGCap1x3MT08uoMpwJnJa6qcaEsjEdYsdSSSPUvWx+7pScWWVAwljZkobM1d8TGY20nkSB7YyoGellbyb+53VSE173Mi6T1KBki0VhKoiJyex3MuAKmRETSyhT3N5K2IgqyoxNqGRD8JZfXiWti5rn1rz7y0r9Jo+jCCdwClXw4ArqcAcNaAKDMTzDK7w5ifPivDsfi9aCk88cwx84nz9kFo7v</latexit>(1)W<latexit sha1_base64="xkxogPCQIQ12jDQQEXn9T1YRBZE=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh6LXjxWsE2hrWWznbRLN5uwuxFK6I/w4kERr/4eb/4bt20O2vpg4PHeDDPzgkRwbVz32ymsrW9sbhW3Szu7e/sH5cOjlo5TxbDJYhGrdkA1Ci6xabgR2E4U0igQ6Afj25nvP6HSPJYPZpJgL6JDyUPOqLGS7z9mVe982i9X3Jo7B1klXk4qkKPRL391BzFLI5SGCap1x3MT08uoMpwJnJa6qcaEsjEdYsdSSSPUvWx+7pScWWVAwljZkobM1d8TGY20nkSB7YyoGellbyb+53VSE173Mi6T1KBki0VhKoiJyex3MuAKmRETSyhT3N5K2IgqyoxNqGRD8JZfXiWti5rn1rz7y0r9Jo+jCCdwClXw4ArqcAcNaAKDMTzDK7w5ifPivDsfi9aCk88cwx84nz9kFo7v</latexit>

(1)

W

(2)

W

(1)

<latexit sha1_base64="bQ/FMWUOGdF0rk6ScnJXyJ6SJQE=">AAAB7nicbVBNS8NAEJ34WetX1aOXxSLUS0mKoMeiF48VbFNoa9lsN+3SzSbsToQS+iO8eFDEq7/Hm//GbZuDtj4YeLw3w8y8IJHCoOt+O2vrG5tb24Wd4u7e/sFh6ei4ZeJUM95ksYx1O6CGS6F4EwVK3k40p1EguR+Mb2e+/8S1EbF6wEnCexEdKhEKRtFKvv+YVWoX036p7FbdOcgq8XJShhyNfumrO4hZGnGFTFJjOp6bYC+jGgWTfFrspoYnlI3pkHcsVTTippfNz52Sc6sMSBhrWwrJXP09kdHImEkU2M6I4sgsezPxP6+TYnjdy4RKUuSKLRaFqSQYk9nvZCA0ZygnllCmhb2VsBHVlKFNqGhD8JZfXiWtWtVzq979Zbl+k8dRgFM4gwp4cAV1uIMGNIHBGJ7hFd6cxHlx3p2PReuak8+cwB84nz9lnI7w</latexit><latexit sha1_base64="xkxogPCQIQ12jDQQEXn9T1YRBZE=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh6LXjxWsE2hrWWznbRLN5uwuxFK6I/w4kERr/4eb/4bt20O2vpg4PHeDDPzgkRwbVz32ymsrW9sbhW3Szu7e/sH5cOjlo5TxbDJYhGrdkA1Ci6xabgR2E4U0igQ6Afj25nvP6HSPJYPZpJgL6JDyUPOqLGS7z9mVe982i9X3Jo7B1klXk4qkKPRL391BzFLI5SGCap1x3MT08uoMpwJnJa6qcaEsjEdYsdSSSPUvWx+7pScWWVAwljZkobM1d8TGY20nkSB7YyoGellbyb+53VSE173Mi6T1KBki0VhKoiJyex3MuAKmRETSyhT3N5K2IgqyoxNqGRD8JZfXiWti5rn1rz7y0r9Jo+jCCdwClXw4ArqcAcNaAKDMTzDK7w5ifPivDsfi9aCk88cwx84nz9kFo7v</latexit>

Generalization performanceend-to-end training

error, %“biological training”

number of epochs

Alternative ideas on biologically plausible
learning
Dendritic error backpropagationin deep cortical microcircuits

ORIGINAL RESEARCH
published: 04 May 2017
doi: 10.3389/fncom.2017.00024

João Sacramento1* , Rui Ponte Costa1 , Yoshua Bengio2 , Walter Senn1*
1

Department of Physiology
University of Bern, Switzerland

test error = 2-3%

Montreal Institute for Learning Algorithms
Université de Montréal, Quebec, Canada

Introduction

Test error (%)

We introduce Equilibrium Propagation, a learning framework for energy-based models.
It involves only one kind of neural computation, performed in both the first phase (whenthe prediction is made) and the second phase of training (after the target or prediction
error is revealed). Although this algorithm computes the gradient of an objective function
just like Backpropagation, it does not need a special computation or circuit for thesecond phase, where errors are implicitly propagated. Equilibrium Propagation shares
similarities with Contrastive Hebbian Learning and Contrastive Divergence while solving
the theoretical issues of both algorithms: our algorithm computes the gradient of awell-defined objective function. Because the objective function is defined in terms of
local perturbations, the second phase of Equilibrium Propagation corresponds to only
nudging the prediction (fixed point or stationary distribution) toward a configurationthat reduces prediction error. In the case of a recurrent multi-layer supervised network,
Edited by:
Marcel van Gerven,
the output units are slightly nudged toward their target in the second phase, and theRadboud University Nijmegen,perturbation introduced at the output layer propagates backward in the hidden layers.
Netherlands
We show that the signal “back-propagated” during this second phase corresponds toReviewed by:
the propagation of error derivatives and encodes the gradient of the objective function,
Stefan Frank,
Radboud University Nijmegen,when the synaptic update corresponds to a standard form of spike-timing dependent
Netherlands
plasticity. This work makes it more plausible that a mechanism similar to Backpropagation
Petia D. Koprinkova-Hristova,
Institute of Information andcould be implemented by brains, since leaky integrator neural computation performs bothCommunication Technologies (BAS),
inference and error back-propagation in our model. The only local difference between the
Bulgaria
FIGURE 3 | Training and validation
error for neural networks with one hidden layer of 500 units (top left), two hidden layers of 500 units (top right), andtwo phases is whether synaptic changes are allowed or not. We also show experimentally*Correspondence:
three hidden layers of 500
units (bottom). The training error eventually decreases to 0.00% in all three cases.
that multi-layer recurrently connected networks with 1, 2, and 3 hidden layers can be
Benjamin ScellierB

outputDépartement d’Informatique et de Recherche Opérationnelle, Montreal Institute for Learning Algorithms, Université deMontréal, Montreal, QC, Canada

Animal behaviour depends on learning to associate sensory stimuli with the desired motor command. Understanding how the brain orchestrates the necessary synaptic modifications acrossdifferent brain areas has remained a longstanding puzzle. Here, we introduce a multi-area neuronal network model in which synaptic plasticity continuously adapts the network towards a global
desired output. In this model synaptic learning is driven by a local dendritic prediction error thatarises from a failure to predict the top-down input given the bottom-up activities. Such errors occur at apical dendrites of pyramidal neurons where both long-range excitatory feedback and local
inhibitory predictions are integrated. When local inhibition fails to match excitatory feedback anerror occurs which triggers plasticity at bottom-up synapses at basal dendrites of the same pyra10
midal neurons. We demonstrate the learning capabilities of the modelin a number of tasks and8.4%
show that it approximates the classical error backpropagation algorithm. Finally, complement10
ing this corticalstimulus denoising
9 a disinhibitory mechanism enables attention-like
5
8 circuit withand generation. Our framework makes several experimental predictions on the function of den1.96%
dritic integration and cortical microcircuits, is consistent with recent observations of cross-area
1.53%
0learning, and suggests a biological implementation of deep learning.

Ahidden 2

Benjamin Scellier * and Yoshua Bengio †hidden 1

Equilibrium Propagation

test error = 1.96%

Abstract

0

500

200

Epochssingle-layer
500+500+10 microcircuit

While walking on the street we are constantly bombarded with complex sensory stimuli. Learning

500+300+10 backprop

500
to navigate such complex environments
is of fundamental importance for survival. In the brain,
these forms of learning are believed to rely on the orchestrated wiring of synaptic communication

input

Equilibrium Propagation: Bridging
the Gap between Energy-Based
Models and Backpropagation

arXiv:1801.00062v1 [q-bio.NC] 30 Dec 2017

Scellier and Bengio

2

between different cortical areas, such as visual and motor cortices (Petreanu et al., 2012; Manita
et al., 2015; Makino and Komiyama, 2015; 28x28
Poort et al., 2015; Zmarz and Keller, 2016; Attinger

handwritten
imageset al., 2017). However, how to correctly modify synapses to MNIST
achieve anappropriatedigit
interaction
between brain areas has remained an open question. This fundamental issue in learning anddevelopment is often referred to as the credit assignment problem (Rumelhart et al., 1986; Suttonand Barto, 1998; Roelfsema and van Ooyen, 2005; Friedrich et al., 2011; Bengio, 2014). The brain,
Figure 4: Learning
to classify real-world, structured stimuli with a multi-area network. (A)* Corresponding authors: {sacramento},{senn}@pyl.unibe.ch
A 784-500-500-10 (i.e. with two hidden areas) network of pyramidal neurons learns to recognize
and classify handwritten digits from the MNIST1data set. Only a subset of connections is shown toenhance clarity. (B) Competitive accuracy (< 2%, an empirical signature of backprop-like learning)is achieved on the standard MNIST testing dataset by our network (solid blue). For comparison theperformance of a shallow learner (i.e. a network in which only output weights are adapted, dashedblack) and of a standard artificial neural network trained with backprop (dashed red, see Methods)
are also shown.

reference mark of non-convolutional artificial neural networks optimized with backprop (1.53%) andend-to-end training“bio” learning“bio” learning

backpropagation
backpropagation

error,
error,%%“biological training”

error, %Generalization performance on CIFAR-10

number of epochs

Lebesgue norm p

p=2

p=3

p=4

Leopold Grinberg - Home

p=5

Strength of anti-hebbian learning

Homepage of Leopold Grinberg

Research Staff Member

Work with L.Grinberg
IBM Research

Visiting Scientist

at Division of Applied Mathematics, Brown University

Conclusions
1. Generalization performance of the “biologically plausible” neural network is
close to that of the neural network trained end-to-end.
2. The weights of the intermediate layers W , W , etc., do not haveinformation about the task the network will have to solve eventually. Thus,
they produce a “general” representation of the data.
(1)<latexit sha1_base64="pUIJGjlLPGaddcqZWq2LwsstpiI=">AAAB7nicjVBNS8NAEJ3Ur1q/qh69LBahXkoiBT0WvXisYJtCG8tmO2mXbjZhdyOU0B/hxYMiXv093vw3bj8OKgo+GHi8N8PMvDAVXBvX/XAKK6tr6xvFzdLW9s7uXnn/oK2TTDFssUQkqhNSjYJLbBluBHZShTQOBfrh+Grm+/eoNE/krZmkGMR0KHnEGTVW8v27vOqdTvvlildz5yB/kwos0eyX33uDhGUxSsME1brruakJcqoMZwKnpV6mMaVsTIfYtVTSGHWQz8+dkhOrDEiUKFvSkLn6dSKnsdaTOLSdMTUj/dObib953cxEF0HOZZoZlGyxKMoEMQmZ/U4GXCEzYmIJZYrbWwkbUUWZsQmV/hdC+6zmuTXvpl5pXC7jKMIRHEMVPDiHBlxDE1rAYAwP8ATPTuo8Oi/O66K14CxnDuEbnLdPaHuO8g==</latexit>(2)<latexit sha1_base64="TeaO/Y8yaLp7WQf8ZX1xITfWY+E=">AAAB7nicjVBNS8NAEJ3Ur1q/qh69LBahXkpSBD0WvXisYJtCG8tmu2mXbjZhdyKU0B/hxYMiXv093vw3bj8OKgo+GHi8N8PMvDCVwqDrfjiFldW19Y3iZmlre2d3r7x/0DZJphlvsUQmuhNSw6VQvIUCJe+kmtM4lNwPx1cz37/n2ohE3eIk5UFMh0pEglG0ku/f5dX66bRfrng1dw7yN6nAEs1++b03SFgWc4VMUmO6nptikFONgkk+LfUyw1PKxnTIu5YqGnMT5PNzp+TEKgMSJdqWQjJXv07kNDZmEoe2M6Y4Mj+9mfib180wughyodIMuWKLRVEmCSZk9jsZCM0ZyokllGlhbyVsRDVlaBMq/S+Edr3muTXv5qzSuFzGUYQjOIYqeHAODbiGJrSAwRge4AmendR5dF6c10VrwVnOHMI3OG+fagGO8w==</latexit>

D.Krotov, J.Hopfield: “Unsupervised
learning by competing hidden units”,
arXiv:1806.10181

github.com/DimaKrotov/Biological_Learning

Twitter:@dimakrotov