# 5. Training

Transformers are trained to predict the next word in a sequence. This process involves feeding the model with text and teaching it to understand patterns in language through a process called masked language modeling.

## 5.1 Masked Language Modeling

During training, the model learns by trying to predict missing words in sentences. For example:
- Original sentence: "The cat sits on the mat"
- Masked sentence: "The cat [MASK] on the mat"
- Task: Predict the missing word "sits"

This approach helps the model learn:
- How words relate to each other
- Common patterns in sentence structure
- Contextual meaning of words

## 5.2 Training Process

The training process involves several key components:

### 5.2.1 Input Processing
1. Text is converted into tokens
2. Tokens are transformed into vectors (embeddings)
3. Positional information is added to these embeddings

### 5.2.2 Attention Mechanism Training

**Multi-head attention operates in three ways during training:**

1. Decoder positions can focus on all input sequence positions
2. Encoder focuses on all positions in its previous layer
3. Decoder focuses on all positions in its previous layer (while maintaining appropriate masking)

This structure helps the model learn both short and long-range relationships between words in text. **The shorter paths in self-attention make it easier for the model to learn long-term dependencies, which is crucial since relevant information can be spread across different parts of the text** [Source: Bachelorthesis - Thorben Schomacker].

### 5.2.3 Learning Process
1. The model makes predictions for masked words
2. Predictions are compared with actual words
3. The model adjusts its parameters based on prediction accuracy
4. The embedding matrix is gradually refined to capture word relationships better

# 6. Inference

Inference is the process where the trained transformer generates text. This process differs from training as the model now uses what it has learned to predict new sequences of words.

## 6.1 How Inference Works

1. **Initial Input**
   - The process starts with a beginning token
   - The decoder receives this token as its first input

2. **Generation Loop**
   - The decoder predicts the most likely next token
   - Each prediction becomes part of the input for the next prediction
   - This loop continues until an end token is generated

3. **Prediction Process**
   - The decoder uses its trained attention mechanisms to understand context
   - It references the encoder's key (K) and value (V) matrices
   - Generates probability distributions over possible next words
   - Selects the most likely word as its prediction

## 6.2 Step-by-Step Example

Let's say the model is generating text starting with "The cat":

1. Initial input: "The cat"
2. Model processes this through its attention layers
3. Generates probabilities for possible next words
4. Selects the most probable word (e.g., "sits")
5. Adds "sits" to the sequence
6. Repeats the process for the next word

## 6.3 Key Aspects of Inference

1. **Autoregressive Nature**
   - Each prediction depends on all previous predictions
   - The model can only see what it has generated so far

2. **Attention During Inference**
   - The model uses its trained attention mechanisms
   - Focuses on relevant parts of the input to make predictions
   - **Maintains the autoregressive property by preventing access to future information** [Source: Bachelorthesis - Thorben Schomacker]

3. **Generation Process**
   - Continues until a stopping condition is met
   - Can be controlled by parameters like maximum length
   - Each prediction builds upon the context of previous words

in this text the training is explained biut for me it is hard to understand what it is trained on it learns the correlation between words via the attentioin step but the masking and predicting most likely next word seems different from that is it learning completly doifferent thinfgs here or at#re attention mechanism and predicting next word via masking the same. are they run in paralel one after another please explain in great detail