Attention is all you need: How Transformer Architecture in NLP started.
Source: tai_blog

Original Paper: Attention is all you need.   This was THE paper that introduced Transformer Architecture to NLP. This transformative concept led to the rise of LLMs and solved the problem of contextualized word embeddings!   Lets take a journey that led up to the statement written above.   I was researching Embedding Models  and some of the material I came across talked about Word Vector Embeddings.   What are Vector Embeddings?Vector embeddings map real-world entities  such as a word  sentence  or image  into vector representations or points in some vector space.   Points that are closer to each other in a vector space have similar semantic meanings  which means that they convey comparable meanings or concepts.   Here  you see sample words and their embedding vector using a word embedding model  such as Word2Vec and GloVe  which gives you the embeddings that capture the semantic meaning of each word.   However  the problem with word embedding models is that they dont really understand the context.   For Example:   The bark of the ancient oak tree was thick and rough  providing shelter for various insects.The dogs bark echoed through the quiet neighborhood  alerting everyone to the approaching mailman.Word embedding models like GloVe wont be able to separate these words by their context. Both models produce static embeddings  which means the same word will have the same vector regardless of its context.   So  we understood the problem  Contexualised Word Embeddings.   Now  lets go back to the original title of this article  Attention is all you need: How Transformer Architecture in NLP started.   In 2017  a new paper  Attention is all you need  was published in Arxiv U+1F9E0. This article introduced the transformer architecture to NLP.   This architecture was what we needed to lead us to large language models  but it also solved the problem we discussed earlier: Contextualized Word Embeddings!   How?The transformer architecture was originally designed for translation  like French to English. So  it makes sense that it only had two components:   The EncoderThe DecoderThe input to the encoder would be a sequence of words or tokens  and the output would be a sequence of continuous representations. Then  the output of the decoder  which would decode  was again words or tokens.   How would the translation work?The encoder would take in a phrase in one language and produce output vectors representing the meaning of the input phrase. To produce these vectors  the encoder could attend to tokens to the left or right of any given token.   On the other hand  the decoder operates one token at a time and considers the predicted tokens along with the encoders outputs.   Hence  the decoder predicts the first word: I.   This is again fed around to the input. Now  the decoder considers the encoder input and the previously generated token and predicts am and so on  one token at a time.   Reread it:   The encoder attends to tokens to the left and right of its output  resulting in encoder output vectors being the contextualized vectors were looking for.   But  the decoder only attends to the inputs to the left.         Is translation the only thing we use this for?Transformers with attention are used for more than simple translation tasks. The most famous ones are the LLMs  like GPT-2  GPT-3  and GPT-4  which were decoder-only architectures. Another well-known example is BERT (Bidirectional Encoder Representations from Transformers)  an encoder-only transformer mode used as a component in sentence embedding models.   Lets talk about BERT!BERT stands for Bidirectional Encoder Representations from Transformers. It is a language model by Google that uses a transformer architecture to understand and generate human-like language. BERT is designed to simultaneously process text in both directions  allowing it to capture context more effectively than traditional unidirectional models  which read text sequentially from left to right or right to left.   Example of Bidirectional CapabilityConsider the sentence:    The bank is situated on the _______ of the river.   In a unidirectional model  understanding the blank would primarily rely on the words before it  potentially leading to ambiguity about whether bank refers to a financial institution or the side of a river.   However  BERTs bidirectional approach allows it to use the entire sentences context  including the words before and after the blank. Thus  the missing word is likely related to the river  resulting in a more accurate prediction  such as bank referring to the riverbank rather than a financial institution.   BERT has two versions:   BERT BASE with    Layers: 12Parameters: 110MAttention Heads: 12Hidden Units: 768BERT LARGE with    Layers: 24Parameters: 340MAttention Heads: 12Hidden Units: 1024DYK?BERT was pre-trained on 3.3 Billion words!   What was it pre-trained on? For what?BERT was pre-trained on two tasks:   Masked Language Modeling (MLM):The inputs are sentences that start with a special token called CLS (Classify Token) and end with a SEP (separator token).   Words  tokens (consider)   Around 15% of the input tokens are masked  and the model is trained to predict those masked tokens.   The model learns to produce contextualized vectors based on the surrounding words at this stage. Read the example above and reread this sentence.   Next Sentence Prediction (NSP):In this one  the model predicts if one sentence is likely to follow another.   Example:   Pair 1: FollowsSentence A: The sun was setting over the horizon.Sentence B: The sky turned a beautiful shade of orange.Sentence B logically follows Sentence A in this pair  so the output prediction is accurate.   Pair 2: Does Not FollowSentence A: She opened the door to find a package on the floor.Sentence B: Cats are known for their playful nature.In this pair  Sentence B does not logically follow Sentence A  so the output prediction isnt accurate; it isnt likely that B follows A.   So  this task basically trains the model to understand the relationship between two sentences.   After youre done with pre-training  you can do transfer learning and fine-tune it with classification  like entity recognition or question answering  to adapt it to specific tasks.   What is a Cross-Encoder?Weve already seen the two types of tokens: Classify Token (CLS) and Separator Token (SEP).   So  a Cross-Encoder is a type of classifier in which the input is two sentences separated by a special SEP token.   Then  it is asked to determine the semantic similarity between those two sentences. In other words  how closely related the meanings of those sentences are.   Fine-Tuning ExamplesText ClassificationIn Text classification  we categorize text into predefined labels. For example  a model can be trained to classify movie reviews as positive or negative.   Fine-Tuning Process   Model: Use the BertForSequenceClassification from the Hugging Face Transformers Library.Data Prep: Input data consists of sentences labeled with categories. For Example:{text: This movie was fantastic!  label: positive}Training: The model is trained on this labeled dataset  adjusting its weights to minimize the classification error. The output will be logits representing the probability of each class.So  if a review is given  The film was boring  the model might output logits that indicate a higher probability for the negative class  which classifies the review as negative.   Named Entity Recognition (NER)In NER  we identify and classify named entities in text  such as people  organizations  and locations.   Fine-Tuning Process   Model: Use BertForTokenClassificationData Prep: Annotated datasets are required; entities are labeled within the text. For Example:{text: Barack Obama was the 44th President of the United States.  labels: {entities: [(0  12  PERSON)  (27  40  TITLE)  (44  57  GPE)]}}Training: The model learns to predict labels for each token based on the context. Each tokens output will indicate its entity type.So  in the sentence Apple Inc. is based in California  the model would identify Apple Inc. as an organization and California as a location.   Question AnsweringPretty obvious  but we generate answers to questions based on a given context in this example.   Fine-Tuning Process:   Model: Use BertForQuestionAnsweringData Preparation: The training data is pairs of questions and context passages. For example:Context: The capital of France is Paris.Question: What is the capital of France?Answer: The model learns to predict the start and end indices of the answer in the context.Training: The model adjusts its parameters to identify the answer span within the context accurately.So  For the context The capital of France is Paris and the question What is the capital of France?  the model would output the indices corresponding to Paris as the answer!   What are some Transformer-Based Language Models?The Transformer architecture has been the foundation for many LLMs  like:   GPT (Generative Pre-trained Transformer): Developed by OpenAI  such as GPT-2  GPT-3  and GPT-4.BERT (Bidirectional Encoder Representations from Transformers): Google developed this Algorithm  which uses a transformer encoder to understand language bidirectionally and helps capture context from both left and right.T5 (Text-to-Text Transfer Transformer): Developed by Google  it can perform various NLP tasks by converting them to a text-to-text format.RoBERTa (Robustly Optimized BERT Approach): An improved version of BERT developed by Facebook AI Research.      So  to summarize    We discussed the evolution of transformer architecture in NLP  starting with the introduction of the Attention is all you need paper. We explored the problem of contextualized word embeddings and how transformer architecture addressed it by introducing the encoder-decoder model for translation.   We also learned the use of transformer architectures in large language models (LLMs) such as GPT-2  GPT-3  GPT-4  and BERT  explaining BERTs bidirectional approach and its two versions: BERT BASE and BERT LARGE.   And finally  we wrapped with some fine-tuning examples  and some Transformer-Based Language Models.   Ive been researching this for about two weeks now  and I tried to condense every piece of research material I reviewed without making it boring.   Thats it for this time; thanks for Reading and Happy Learning!         References: How I learned this concept Attention Is All You Need  Wikipedia Attention Is All You Need is a 2017 landmark research paper in machine learning authored by eight scientists workingen.wikipedia.org   Understanding Googles Attention Is All You Need Paper and Its Groundbreaking ImpactWith all the buzz around Generative AI  ChatGPT  Bard  etc.  it is worthwhile to look at the work that influenced italok-shankar.medium.com