Building a ML Transformer in a Spreadsheet
Channel: Kevin Lubick
Date: 2022-03-13

We continue the theme of building neural networks in a spreadsheet, as a way to do a worked example of them.
In this video, we build a Transformer (a type of Machine Learning model) which can encipher text using the Atbash cipher and reverse strings. It also introduces the topics of self-attention and positional encoding.

Follow along by making a copy of https://bit.ly/TransformerSheet1

Three press releases about breakthroughs with Transformer models:
 - https://deepmind.com/blog/article/Competitive-programming-with-AlphaCode
 - https://ai.googleblog.com/2022/01/lamda-towards-safe-grounded-and-high.html
 - https://openai.com/blog/gpt-3-apps/
 
Training a Neural Network in a Spreadsheet
 - https://www.youtube.com/watch?v=fjfZZ6S1ad4

Implementing a Convolutional Neural Network in a Spreadsheet
 - https://www.youtube.com/watch?v=8QfFtUcuYCU

Implementing a Recurrent Neural Network in a Spreadsheet
 - https://www.youtube.com/watch?v=a4JLw4Du8YM

Appendix Video with more details about multi-head attention, residual connections and more
 - https://youtu.be/67n0eudtcqI

Attention Is All You Need by Vaswani et al (Original Transformers Paper)
 - https://arxiv.org/pdf/1706.03762.pdf

Three-in-a-row word game inspired by Spit Not So, which I learned about from "Computing Without Computers" by Paul Curzon, who got it from "Winning Ways for your
Mathematical Plays" by Berlekamp, Conway, and Guy
 - https://teachinglondoncomputing.org/resources/inspiring-computing-stories/computingwithoutcomputers

More resources about "embedding"
 - https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/
 - https://medium.com/deeper-learning/glossary-of-deep-learning-word-embedding-f90c3cec34ca

More resources about Attention and Transformer Models
 - https://nlp.seas.harvard.edu/2018/04/03/attention.html
 - https://jalammar.github.io/illustrated-transformer/
 - https://towardsdatascience.com/illustrated-guide-to-transformers-step-by-step-explanation-f74876522bc0
 - The Narrated Transformer Language Model https://youtu.be/-QH8fRhqFHM
 - Attention Model Intuition https://youtu.be/SysgYptB198
 - Attention Mechanisms https://youtu.be/3qdQAtfq1jU
 - LSTM is dead. Long Live Transformers! https://youtu.be/S27pHKBEp30
 - Answer to "What exactly are keys, queries, and values?" https://stats.stackexchange.com/a/424127
 - https://dugas.ch/artificial_curiosity/GPT_architecture.html

Filled out Machine Learning Transformer in a Spreadsheet: https://bit.ly/TransformerSheet2

Narration by James K. Script and visualizations by Kaylee L.

0:00 Introduction
1:49 High Level "Why are Transformers Effective"
2:15 Word Game demonstrating benefits of data conversion
3:33 Data Converter Overview
4:05 The four conversions we will build
5:03 Converting binary to truth gradients
8:35 Converting ABCDEF to ZYXWVU (Atbash cipher)
9:23 Implementing Atbash with Self-attention
12:57 Understanding Attention and Compatibility Scores
16:17 Why are the matrices named Query, Key, and Value?
18:10 Introducing Positional Encoding
19:26 Reversing a string using Self-Attention and Positional Encoding
22:00 Transformer Check for Understanding
23:24 Recap

Special Thanks to
 - Somin Wadhwa for initial research assistance
 - Jorge Betancourt and Pat Berard for formative feedback