Repository: huggingface/course
Pull request #911: Improve and re-release chapter 2
Author: login: burtenshaw
Created: 2025-05-06T11:12:00Z | Merged: 2025-05-23T07:01:54Z

This is a minor improvement to chapter 2 to do these things:

- remove tensorflow examples and videos
- add a page on TGI, vLLM, and llama.cpp


Diff:
diff --git a/chapters/en/_toctree.yml b/chapters/en/_toctree.yml
index 7f8d105ca..b300b245f 100644
--- a/chapters/en/_toctree.yml
+++ b/chapters/en/_toctree.yml
@@ -46,6 +46,8 @@
   - local: chapter2/7
     title: Basic usage completed!
   - local: chapter2/8
+    title: Optimized Inference Deployment
+  - local: chapter2/9
     title: End-of-chapter quiz
     quiz: 2
 
diff --git a/chapters/en/chapter2/1.mdx b/chapters/en/chapter2/1.mdx
index 16347ca94..70e290a9d 100644
--- a/chapters/en/chapter2/1.mdx
+++ b/chapters/en/chapter2/1.mdx
@@ -10,7 +10,7 @@ As you saw in [Chapter 1](/course/chapter1), Transformer models are usually very
 The 🤗 Transformers library was created to solve this problem. Its goal is to provide a single API through which any Transformer model can be loaded, trained, and saved. The library's main features are:
 
 - **Ease of use**: Downloading, loading, and using a state-of-the-art NLP model for inference can be done in just two lines of code.
-- **Flexibility**: At their core, all models are simple PyTorch `nn.Module` or TensorFlow `tf.keras.Model` classes and can be handled like any other models in their respective machine learning (ML) frameworks.
+- **Flexibility**: At their core, all models are simple PyTorch `nn.Module` classes and can be handled like any other models in their respective machine learning (ML) frameworks.
 - **Simplicity**: Hardly any abstractions are made across the library. The "All in one file" is a core concept: a model's forward pass is entirely defined in a single file, so that the code itself is understandable and hackable.
 
 This last feature makes 🤗 Transformers quite different from other ML libraries. The models are not built on modules 
diff --git a/chapters/en/chapter2/2.mdx b/chapters/en/chapter2/2.mdx
index 2a35669d7..205e07e51 100644
--- a/chapters/en/chapter2/2.mdx
+++ b/chapters/en/chapter2/2.mdx
@@ -2,8 +2,6 @@
 
 # Behind the pipeline[[behind-the-pipeline]]
 
-{#if fw === 'pt'}
-
 <CourseFloatingBanner chapter={2}
   classNames="absolute z-10 right-0 top-0"
   notebooks={[
@@ -11,26 +9,7 @@
     {label: "Aws Studio", value: "https://studiolab.sagemaker.aws/import/github/huggingface/notebooks/blob/master/course/en/chapter2/section2_pt.ipynb"},
 ]} />
 
-{:else}
-
-<CourseFloatingBanner chapter={2}
-  classNames="absolute z-10 right-0 top-0"
-  notebooks={[
-    {label: "Google Colab", value: "https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/en/chapter2/section2_tf.ipynb"},
-    {label: "Aws Studio", value: "https://studiolab.sagemaker.aws/import/github/huggingface/notebooks/blob/master/course/en/chapter2/section2_tf.ipynb"},
-]} />
-
-{/if}
-
-<Tip>
-This is the first section where the content is slightly different depending on whether you use PyTorch or TensorFlow. Toggle the switch on top of the title to select the platform you prefer!
-</Tip>
-
-{#if fw === 'pt'}
 <Youtube id="1pedAIvTWXk"/>
-{:else}
-<Youtube id="wVN12smEvqg"/>
-{/if}
 
 Let's start with a complete example, taking a look at what happened behind the scenes when we executed the following code in [Chapter 1](/course/chapter1):
 
@@ -83,11 +62,10 @@ tokenizer = AutoTokenizer.from_pretrained(checkpoint)
 
 Once we have the tokenizer, we can directly pass our sentences to it and we'll get back a dictionary that's ready to feed to our model! The only thing left to do is to convert the list of input IDs to tensors.
 
-You can use 🤗 Transformers without having to worry about which ML framework is used as a backend; it might be PyTorch or TensorFlow, or Flax for some models. However, Transformer models only accept *tensors* as input. If this is your first time hearing about tensors, you can think of them as NumPy arrays instead. A NumPy array can be a scalar (0D), a vector (1D), a matrix (2D), or have more dimensions. It's effectively a tensor; other ML frameworks' tensors behave similarly, and are usually as simple to instantiate as NumPy arrays.
+You can use 🤗 Transformers without having to worry about which ML framework is used as a backend; it might be PyTorch or Flax for some models. However, Transformer models only accept *tensors* as input. If this is your first time hearing about tensors, you can think of them as NumPy arrays instead. A NumPy array can be a scalar (0D), a vector (1D), a matrix (2D), or have more dimensions. It's effectively a tensor; other ML frameworks' tensors behave similarly, and are usually as simple to instantiate as NumPy arrays.
 
-To specify the type of tensors we want to get back (PyTorch, TensorFlow, or plain NumPy), we use the `return_tensors` argument:
+To specify the type of tensors we want to get back (PyTorch or plain NumPy), we use the `return_tensors` argument:
 
-{#if fw === 'pt'}
 ```python
 raw_inputs = [
     "I've been waiting for a HuggingFace course my whole life.",
@@ -96,21 +74,9 @@ raw_inputs = [
 inputs = tokenizer(raw_inputs, padding=True, truncation=True, return_tensors="pt")
 print(inputs)
 ```
-{:else}
-```python
-raw_inputs = [
-    "I've been waiting for a HuggingFace course my whole life.",
-    "I hate this so much!",
-]
-inputs = tokenizer(raw_inputs, padding=True, truncation=True, return_tensors="tf")
-print(inputs)
-```
-{/if}
 
 Don't worry about padding and truncation just yet; we'll explain those later. The main things to remember here are that you can pass one sentence or a list of sentences, as well as specifying the type of tensors you want to get back (if no type is passed, you will get a list of lists as a result).
 
-{#if fw === 'pt'}
-
 Here's what the results look like as PyTorch tensors:
 
 ```python out
@@ -125,31 +91,11 @@ Here's what the results look like as PyTorch tensors:
     ])
 }
 ```
-{:else}
-
-Here's what the results look like as TensorFlow tensors:
-
-```python out
-{
-    'input_ids': <tf.Tensor: shape=(2, 16), dtype=int32, numpy=
-        array([
-            [  101,  1045,  1005,  2310,  2042,  3403,  2005,  1037, 17662, 12172,  2607,  2026,  2878,  2166,  1012,   102],
-            [  101,  1045,  5223,  2023,  2061,  2172,   999,   102,     0,     0,     0,     0,     0,     0,     0,     0]
-        ], dtype=int32)>, 
-    'attention_mask': <tf.Tensor: shape=(2, 16), dtype=int32, numpy=
-        array([
-            [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
-            [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]
-        ], dtype=int32)>
-}
-```
-{/if}
 
 The output itself is a dictionary containing two keys, `input_ids` and `attention_mask`. `input_ids` contains two rows of integers (one for each sentence) that are the unique identifiers of the tokens in each sentence. We'll explain what the `attention_mask` is later in this chapter. 
 
 ## Going through the model[[going-through-the-model]]
 
-{#if fw === 'pt'}
 We can download our pretrained model the same way we did with our tokenizer. 🤗 Transformers provides an `AutoModel` class which also has a `from_pretrained()` method:
 
 ```python
@@ -158,16 +104,6 @@ from transformers import AutoModel
 checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
 model = AutoModel.from_pretrained(checkpoint)
 ```
-{:else}
-We can download our pretrained model the same way we did with our tokenizer. 🤗 Transformers provides an `TFAutoModel` class which also has a `from_pretrained` method:
-
-```python
-from transformers import TFAutoModel
-
-checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
-model = TFAutoModel.from_pretrained(checkpoint)
-```
-{/if}
 
 In this code snippet, we have downloaded the same checkpoint we used in our pipeline before (it should actually have been cached already) and instantiated a model with it.
 
@@ -189,7 +125,6 @@ It is said to be "high dimensional" because of the last value. The hidden size c
 
 We can see this if we feed the inputs we preprocessed to our model:
 
-{#if fw === 'pt'}
 ```python
 outputs = model(**inputs)
 print(outputs.last_hidden_state.shape)
@@ -198,16 +133,6 @@ print(outputs.last_hidden_state.shape)
 ```python out
 torch.Size([2, 16, 768])
 ```
-{:else}
-```py
-outputs = model(inputs)
-print(outputs.last_hidden_state.shape)
-```
-
-```python out
-(2, 16, 768)
-```
-{/if}
 
 Note that the outputs of 🤗 Transformers models behave like `namedtuple`s or dictionaries. You can access the elements by attributes (like we did) or by key (`outputs["last_hidden_state"]`), or even by index if you know exactly where the thing you are looking for is (`outputs[0]`).
 
@@ -235,7 +160,6 @@ There are many different architectures available in 🤗 Transformers, with each
 - `*ForTokenClassification`
 - and others 🤗
 
-{#if fw === 'pt'}
 For our example, we will need a model with a sequence classification head (to be able to classify the sentences as positive or negative). So, we won't actually use the `AutoModel` class, but `AutoModelForSequenceClassification`:
 
 ```python
@@ -245,17 +169,6 @@ checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
 model = AutoModelForSequenceClassification.from_pretrained(checkpoint)
 outputs = model(**inputs)
 ```
-{:else}
-For our example, we will need a model with a sequence classification head (to be able to classify the sentences as positive or negative). So, we won't actually use the `TFAutoModel` class, but `TFAutoModelForSequenceClassification`:
-
-```python
-from transformers import TFAutoModelForSequenceClassification
-
-checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
-model = TFAutoModelForSequenceClassification.from_pretrained(checkpoint)
-outputs = model(inputs)
-```
-{/if}
 
 Now if we look at the shape of our outputs, the dimensionality will be much lower: the model head takes as input the high-dimensional vectors we saw before, and outputs vectors containing two values (one per label):
 
@@ -263,15 +176,9 @@ Now if we look at the shape of our outputs, the dimensionality will be much lowe
 print(outputs.logits.shape)
 ```
 
-{#if fw === 'pt'}
 ```python out
 torch.Size([2, 2])
 ```
-{:else}
-```python out
-(2, 2)
-```
-{/if}
 
 Since we have just two sentences and two labels, the result we get from our model is of shape 2 x 2.
 
@@ -283,49 +190,24 @@ The values we get as output from our model don't necessarily make sense by thems
 print(outputs.logits)
 ```
 
-{#if fw === 'pt'}
 ```python out
 tensor([[-1.5607,  1.6123],
         [ 4.1692, -3.3464]], grad_fn=<AddmmBackward>)
 ```
-{:else}
-```python out
-<tf.Tensor: shape=(2, 2), dtype=float32, numpy=
-    array([[-1.5606991,  1.6122842],
-           [ 4.169231 , -3.3464472]], dtype=float32)>
-```
-{/if}
 
 Our model predicted `[-1.5607, 1.6123]` for the first sentence and `[ 4.1692, -3.3464]` for the second one. Those are not probabilities but *logits*, the raw, unnormalized scores outputted by the last layer of the model. To be converted to probabilities, they need to go through a [SoftMax](https://en.wikipedia.org/wiki/Softmax_function) layer (all 🤗 Transformers models output the logits, as the loss function for training will generally fuse the last activation function, such as SoftMax, with the actual loss function, such as cross entropy):
 
-{#if fw === 'pt'}
 ```py
 import torch
 
 predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
 print(predictions)
 ```
-{:else}
-```py
-import tensorflow as tf
-
-predictions = tf.math.softmax(outputs.logits, axis=-1)
-print(predictions)
-```
-{/if}
 
-{#if fw === 'pt'}
 ```python out
 tensor([[4.0195e-02, 9.5980e-01],
         [9.9946e-01, 5.4418e-04]], grad_fn=<SoftmaxBackward>)
 ```
-{:else}
-```python out
-tf.Tensor(
-[[4.01951671e-02 9.59804833e-01]
- [9.9945587e-01 5.4418424e-04]], shape=(2, 2), dtype=float32)
-```
-{/if}
 
 Now we can see that the model predicted `[0.0402, 0.9598]` for the first sentence and `[0.9995,  0.0005]` for the second one. These are recognizable probability scores.
 
diff --git a/chapters/en/chapter2/3.mdx b/chapters/en/chapter2/3.mdx
index acc653704..91d3ddd00 100644
--- a/chapters/en/chapter2/3.mdx
+++ b/chapters/en/chapter2/3.mdx
@@ -1,8 +1,6 @@
 <FrameworkSwitchCourse {fw} />
 
-# Models[[models]]
-
-{#if fw === 'pt'}
+# Models[[the-models]]
 
 <CourseFloatingBanner chapter={2}
   classNames="absolute z-10 right-0 top-0"
@@ -11,210 +9,281 @@
     {label: "Aws Studio", value: "https://studiolab.sagemaker.aws/import/github/huggingface/notebooks/blob/master/course/en/chapter2/section3_pt.ipynb"},
 ]} />
 
-{:else}
+<Youtube id="AhChOFRegn4"/>
 
-<CourseFloatingBanner chapter={2}
-  classNames="absolute z-10 right-0 top-0"
-  notebooks={[
-    {label: "Google Colab", value: "https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/en/chapter2/section3_tf.ipynb"},
-    {label: "Aws Studio", value: "https://studiolab.sagemaker.aws/import/github/huggingface/notebooks/blob/master/course/en/chapter2/section3_tf.ipynb"},
-]} />
+In this section, we'll take a closer look at creating and using models. We'll use the `AutoModel` class, which is handy when you want to instantiate any model from a checkpoint.
 
-{/if}
+## Creating a Transformer[[creating-a-transformer]]
 
-{#if fw === 'pt'}
-<Youtube id="AhChOFRegn4"/>
-{:else}
-<Youtube id="d3JVgghSOew"/>
-{/if}
+Let's begin by examining what happens when we instantiate an `AutoModel`:
 
-{#if fw === 'pt'}
-In this section we'll take a closer look at creating and using a model. We'll use the `AutoModel` class, which is handy when you want to instantiate any model from a checkpoint.
+```py
+from transformers import AutoModel
 
-The `AutoModel` class and all of its relatives are actually simple wrappers over the wide variety of models available in the library. It's a clever wrapper as it can automatically guess the appropriate model architecture for your checkpoint, and then instantiates a model with this architecture.
+model = AutoModel.from_pretrained("bert-base-cased")
+```
 
-{:else}
-In this section we'll take a closer look at creating and using a model. We'll use the `TFAutoModel` class, which is handy when you want to instantiate any model from a checkpoint.
+Similar to the tokenizer, the `from_pretrained()` method will download and cache the model data from the Hugging Face Hub. As mentioned previously, the checkpoint name corresponds to a specific model architecture and weights, in this case a BERT model with a basic architecture (12 layers, 768 hidden size, 12 attention heads) and cased inputs (meaning that the uppercase/lowercase distinction is important). There are many checkpoints available on the Hub — you can explore them [here](https://huggingface.co/models).
 
-The `TFAutoModel` class and all of its relatives are actually simple wrappers over the wide variety of models available in the library. It's a clever wrapper as it can automatically guess the appropriate model architecture for your checkpoint, and then instantiates a model with this architecture.
+The `AutoModel` class and its associates are actually simple wrappers designed to fetch the appropriate model architecture for a given checkpoint. It's an "auto" class meaning it will guess the appropriate model architecture for you and instantiate the correct model class. However, if you know the type of model you want to use, you can use the class that defines its architecture directly:
 
-{/if}
+```py
+from transformers import BertModel
 
-However, if you know the type of model you want to use, you can use the class that defines its architecture directly. Let's take a look at how this works with a BERT model.
+model = BertModel.from_pretrained("bert-base-cased")
+```
 
-## Creating a Transformer[[creating-a-transformer]]
+## Loading and saving[[loading-and-saving]]
 
-The first thing we'll need to do to initialize a BERT model is load a configuration object:
+Saving a model is as simple as saving a tokenizer. In fact, the models actually have the same `save_pretrained()` method, which saves the model's weights and architecture configuration:
 
-{#if fw === 'pt'}
 ```py
-from transformers import BertConfig, BertModel
+model.save_pretrained("directory_on_my_computer")
+```
 
-# Building the config
-config = BertConfig()
+This will save two files to your disk:
 
-# Building the model from the config
-model = BertModel(config)
 ```
-{:else}
+ls directory_on_my_computer
+
+config.json pytorch_model.bin
+```
+
+If you look inside the *config.json* file, you'll see all the necessary attributes needed to build the model architecture. This file also contains some metadata, such as where the checkpoint originated and what 🤗 Transformers version you were using when you last saved the checkpoint.
+
+The *pytorch_model.bin* file is known as the state dictionary; it contains all your model's weights. The two files work together: the configuration file is needed to know about the model architecture, while the model weights are the parameters of the model.
+
+To reuse a saved model, use the `from_pretrained()` method again:
+
 ```py
-from transformers import BertConfig, TFBertModel
+from transformers import AutoModel
+
+model = AutoModel.from_pretrained("directory_on_my_computer")
+```
+
+A wonderful feature of the 🤗 Transformers library is the ability to easily share models and tokenizers with the community. To do this, make sure you have an account on [Hugging Face](https://huggingface.co). If you're using a notebook, you can easily log in with this:
 
-# Building the config
-config = BertConfig()
+```python
+from huggingface_hub import notebook_login
 
-# Building the model from the config
-model = TFBertModel(config)
+notebook_login()
 ```
-{/if}
 
-The configuration contains many attributes that are used to build the model:
+Otherwise, at your terminal run:
+
+```bash
+huggingface-cli login
+```
+
+Then you can push the model to the Hub with the `push_to_hub()` method:
 
 ```py
-print(config)
+model.push_to_hub("my-awesome-model")
 ```
 
-```python out
-BertConfig {
-  [...]
-  "hidden_size": 768,
-  "intermediate_size": 3072,
-  "max_position_embeddings": 512,
-  "num_attention_heads": 12,
-  "num_hidden_layers": 12,
-  [...]
-}
+This will upload the model files to the Hub, in a repository under your namespace named *my-awesome-model*. Then, anyone can load your model with the `from_pretrained()` method!
+
+```py
+from transformers import AutoModel
+
+model = AutoModel.from_pretrained("your-username/my-awesome-model")
 ```
 
-While you haven't seen what all of these attributes do yet, you should recognize some of them: the `hidden_size` attribute defines the size of the `hidden_states` vector, and `num_hidden_layers` defines the number of layers the Transformer model has.
+You can do a lot more with the Hub API:
+- Push a model from a local repository
+- Update specific files without re-uploading everything
+- Add model cards to document the model's abilities, limitations, known biases, etc.
 
-### Different loading methods[[different-loading-methods]]
+See [the documentation](https://huggingface.co/docs/huggingface_hub/how-to-upstream) for a complete tutorial on this, or check out the advanced [Chapter 4](/course/chapter4).
 
-Creating a model from the default configuration initializes it with random values:
+## Encoding text[[encoding-text]]
+
+Transformer models handle text by turning the inputs into numbers. Here we will look at exactly what happens when your text is processed by the tokenizer. We've already seen in [Chapter 1](/course/chapter1) that tokenizers split the text into tokens and then convert these tokens into numbers. We can see this conversion through a simple tokenizer:
 
-{#if fw === 'pt'}
 ```py
-from transformers import BertConfig, BertModel
+from transformers import AutoTokenizer
 
-config = BertConfig()
-model = BertModel(config)
+tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
 
-# Model is randomly initialized!
+encoded_input = tokenizer("Hello, I'm a single sentence!")
+print(encoded_input)
 ```
-{:else}
-```py
-from transformers import BertConfig, TFBertModel
 
-config = BertConfig()
-model = TFBertModel(config)
+```python out
+{'input_ids': [101, 8667, 117, 1000, 1045, 1005, 1049, 2235, 17662, 12172, 1012, 102], 
+ 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 
+ 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
+```
+
+We get a dictionary with the following fields:
+- input_ids: numerical representations of your tokens
+- token_type_ids: these tell the model which part of the input is sentence A and which is sentence B (discussed more in the next section)
+- attention_mask: this indicates which tokens should be attended to and which should not (discussed more in a bit)
+
+We can decode the input IDs to get back the original text:
+
+```py
+tokenizer.decode(encoded_input["input_ids"])
+```
 
-# Model is randomly initialized!
+```python out
+"[CLS] Hello, I'm a single sentence! [SEP]"
 ```
-{/if}
 
-The model can be used in this state, but it will output gibberish; it needs to be trained first. We could train the model from scratch on the task at hand, but as you saw in [Chapter 1](/course/chapter1), this would require a long time and a lot of data, and it would have a non-negligible environmental impact. To avoid unnecessary and duplicated effort, it's imperative to be able to share and reuse models that have already been trained.
+You'll notice that the tokenizer has added special tokens — `[CLS]` and `[SEP]` — required by the model. Not all models need special tokens; they're utilized when a model was pretrained with them, in which case the tokenizer needs to add them as that model expects these tokens.
 
-Loading a Transformer model that is already trained is simple — we can do this using the `from_pretrained()` method:
+You can encode multiple sentences at once, either by batching them together (we'll discuss this soon) or by passing a list:
 
-{#if fw === 'pt'}
 ```py
-from transformers import BertModel
+encoded_input = tokenizer("How are you?", "I'm fine, thank you!")
+print(encoded_input)
+```
 
-model = BertModel.from_pretrained("bert-base-cased")
+```python out
+{'input_ids': [[101, 1731, 1132, 1128, 136, 102], [101, 1045, 1005, 1049, 2503, 117, 5763, 1128, 136, 102]], 
+ 'token_type_ids': [[0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], 
+ 'attention_mask': [[1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}
 ```
 
-As you saw earlier, we could replace `BertModel` with the equivalent `AutoModel` class. We'll do this from now on as this produces checkpoint-agnostic code; if your code works for one checkpoint, it should work seamlessly with another. This applies even if the architecture is different, as long as the checkpoint was trained for a similar task (for example, a sentiment analysis task).
+Note that when passing multiple sentences, the tokenizer returns a list for each sentence for each dictionary value. We can also ask the tokenizer to return tensors directly from PyTorch:
 
-{:else}
 ```py
-from transformers import TFBertModel
+encoded_input = tokenizer("How are you?", "I'm fine, thank you!", return_tensors="pt")
+print(encoded_input)
+```
 
-model = TFBertModel.from_pretrained("bert-base-cased")
+```python out
+{'input_ids': tensor([[  101,  1731,  1132,  1128,   136,   102],
+         [  101,  1045,  1005,  1049,  2503,   117,  5763,  1128,   136,   102]]), 
+ 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0],
+         [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 
+ 'attention_mask': tensor([[1, 1, 1, 1, 1, 1],
+         [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}
 ```
 
-As you saw earlier, we could replace `TFBertModel` with the equivalent `TFAutoModel` class. We'll do this from now on as this produces checkpoint-agnostic code; if your code works for one checkpoint, it should work seamlessly with another. This applies even if the architecture is different, as long as the checkpoint was trained for a similar task (for example, a sentiment analysis task).
+But there's a problem: the two lists don't have the same length! Arrays and tensors need to be rectangular, so we can't simply convert these lists to a PyTorch tensor (or NumPy array). The tokenizer provides an option for that: padding.
 
-{/if}
+### Padding inputs[[padding-inputs]]
 
-In the code sample above we didn't use `BertConfig`, and instead loaded a pretrained model via the `bert-base-cased` identifier. This is a model checkpoint that was trained by the authors of BERT themselves; you can find more details about it in its [model card](https://huggingface.co/bert-base-cased).
+If we ask the tokenizer to pad the inputs, it will make all sentences the same length by adding a special padding token to the sentences that are shorter than the longest one:
 
-This model is now initialized with all the weights of the checkpoint. It can be used directly for inference on the tasks it was trained on, and it can also be fine-tuned on a new task. By training with pretrained weights rather than from scratch, we can quickly achieve good results.
+```py
+encoded_input = tokenizer(
+    ["How are you?", "I'm fine, thank you!"], padding=True, return_tensors="pt"
+)
+print(encoded_input)
+```
 
-The weights have been downloaded and cached (so future calls to the `from_pretrained()` method won't re-download them) in the cache folder, which defaults to *~/.cache/huggingface/transformers*. You can customize your cache folder by setting the `HF_HOME` environment variable.
+```python out
+{'input_ids': tensor([[  101,  1731,  1132,  1128,   136,   102,     0,     0,     0,     0],
+         [  101,  1045,  1005,  1049,  2503,   117,  5763,  1128,   136,   102]]), 
+ 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
+         [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 
+ 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 0, 0, 0, 0],
+         [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}
+```
 
-The identifier used to load the model can be the identifier of any model on the Model Hub, as long as it is compatible with the BERT architecture. The entire list of available BERT checkpoints can be found [here](https://huggingface.co/models?filter=bert).
+Now we have rectangular tensors! Note that the padding tokens have been encoded into input IDs with ID 0, and they have an attention mask value of 0 as well. This is because those padding tokens shouldn't be analyzed by the model: they're not part of the actual sentence.
 
-### Saving methods[[saving-methods]]
+### Truncating inputs[[truncating-inputs]]
 
-Saving a model is as easy as loading one — we use the `save_pretrained()` method, which is analogous to the `from_pretrained()` method:
+The tensors might get too big to be processed by the model. For instance, BERT was only pretrained with sequences up to 512 tokens, so it cannot process longer sequences. If you have sequences longer than the model can handle, you'll need to truncate them with the `truncation` parameter:
 
 ```py
-model.save_pretrained("directory_on_my_computer")
+encoded_input = tokenizer(
+    "This is a very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very very long sentence.",
+    truncation=True,
+)
+print(encoded_input["input_ids"])
 ```
 
-This saves two files to your disk:
-
-{#if fw === 'pt'}
+```python out
+[101, 1188, 1110, 170, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1505, 1179, 5650, 119, 102]
 ```
-ls directory_on_my_computer
 
-config.json pytorch_model.bin
-```
-{:else}
-```
-ls directory_on_my_computer
+By combining the padding and truncation arguments, you can make sure your tensors have the exact size you need:
 
-config.json tf_model.h5
+```py
+encoded_input = tokenizer(
+    ["How are you?", "I'm fine, thank you!"],
+    padding=True,
+    truncation=True,
+    max_length=5,
+    return_tensors="pt",
+)
+print(encoded_input)
 ```
-{/if}
 
-If you take a look at the *config.json* file, you'll recognize the attributes necessary to build the model architecture. This file also contains some metadata, such as where the checkpoint originated and what 🤗 Transformers version you were using when you last saved the checkpoint.
+```python out
+{'input_ids': tensor([[  101,  1731,  1132,  1128,   102],
+         [  101,  1045,  1005,  1049,   102]]), 
+ 'token_type_ids': tensor([[0, 0, 0, 0, 0],
+         [0, 0, 0, 0, 0]]), 
+ 'attention_mask': tensor([[1, 1, 1, 1, 1],
+         [1, 1, 1, 1, 1]])}
+```
 
-{#if fw === 'pt'}
-The *pytorch_model.bin* file is known as the *state dictionary*; it contains all your model's weights. The two files go hand in hand; the configuration is necessary to know your model's architecture, while the model weights are your model's parameters.
+### Adding special tokens
 
-{:else}
-The *tf_model.h5* file is known as the *state dictionary*; it contains all your model's weights. The two files go hand in hand; the configuration is necessary to know your model's architecture, while the model weights are your model's parameters.
+Special tokens (or at least the concept of them) is particularly important to BERT and derived models. These tokens are added to better represent the sentence boundaries, such as the beginning of a sentence (`[CLS]`) or separator between sentences (`[SEP]`). Let's look at a simple example:
 
-{/if}
+```py
+encoded_input = tokenizer("How are you?")
+print(encoded_input["input_ids"])
+tokenizer.decode(encoded_input["input_ids"])
+```
 
-## Using a Transformer model for inference[[using-a-transformer-model-for-inference]]
+```python out
+[101, 1731, 1132, 1128, 136, 102]
+'[CLS] How are you? [SEP]'
+```
 
-Now that you know how to load and save a model, let's try using it to make some predictions. Transformer models can only process numbers — numbers that the tokenizer generates. But before we discuss tokenizers, let's explore what inputs the model accepts.
+These special tokens are automatically added by the tokenizer. Not all models need special tokens; they are primarily used when a model was pretrained with them, in which case the tokenizer will add them since the model expects them.
 
-Tokenizers can take care of casting the inputs to the appropriate framework's tensors, but to help you understand what's going on, we'll take a quick look at what must be done before sending the inputs to the model.
+### Why is all of this necessary?
 
-Let's say we have a couple of sequences:
+Here's a concrete example. Consider these encoded sequences:
 
 ```py
-sequences = ["Hello!", "Cool.", "Nice!"]
+sequences = [
+    "I've been waiting for a HuggingFace course my whole life.",
+    "I hate this so much!",
+]
 ```
 
-The tokenizer converts these to vocabulary indices which are typically called *input IDs*. Each sequence is now a list of numbers! The resulting output is:
+Once tokenized, we have:
 
-```py no-format
+```python
 encoded_sequences = [
-    [101, 7592, 999, 102],
-    [101, 4658, 1012, 102],
-    [101, 3835, 999, 102],
+    [
+        101,
+        1045,
+        1005,
+        2310,
+        2042,
+        3403,
+        2005,
+        1037,
+        17662,
+        12172,
+        2607,
+        2026,
+        2878,
+        2166,
+        1012,
+        102,
+    ],
+    [101, 1045, 5223, 2023, 2061, 2172, 999, 102],
 ]
 ```
 
 This is a list of encoded sequences: a list of lists. Tensors only accept rectangular shapes (think matrices). This "array" is already of rectangular shape, so converting it to a tensor is easy:
 
-{#if fw === 'pt'}
 ```py
 import torch
 
 model_inputs = torch.tensor(encoded_sequences)
 ```
-{:else}
-```py
-import tensorflow as tf
-
-model_inputs = tf.constant(encoded_sequences)
-```
-{/if}
 
 ### Using the tensors as inputs to the model[[using-the-tensors-as-inputs-to-the-model]]
 
diff --git a/chapters/en/chapter2/4.mdx b/chapters/en/chapter2/4.mdx
index 30167ddbd..d07264690 100644
--- a/chapters/en/chapter2/4.mdx
+++ b/chapters/en/chapter2/4.mdx
@@ -2,8 +2,6 @@
 
 # Tokenizers[[tokenizers]]
 
-{#if fw === 'pt'}
-
 <CourseFloatingBanner chapter={2}
   classNames="absolute z-10 right-0 top-0"
   notebooks={[
@@ -11,17 +9,6 @@
     {label: "Aws Studio", value: "https://studiolab.sagemaker.aws/import/github/huggingface/notebooks/blob/master/course/en/chapter2/section4_pt.ipynb"},
 ]} />
 
-{:else}
-
-<CourseFloatingBanner chapter={2}
-  classNames="absolute z-10 right-0 top-0"
-  notebooks={[
-    {label: "Google Colab", value: "https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/en/chapter2/section4_tf.ipynb"},
-    {label: "Aws Studio", value: "https://studiolab.sagemaker.aws/import/github/huggingface/notebooks/blob/master/course/en/chapter2/section4_tf.ipynb"},
-]} />
-
-{/if}
-
 <Youtube id="VFp38yj8h3A"/>
 
 Tokenizers are one of the core components of the NLP pipeline. They serve one purpose: to translate text into data that can be processed by the model. Models can only process numbers, so tokenizers need to convert our text inputs to numerical data. In this section, we'll explore exactly what happens in the tokenization pipeline. 
@@ -131,14 +118,8 @@ from transformers import BertTokenizer
 tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
 ```
 
-{#if fw === 'pt'}
 Similar to `AutoModel`, the `AutoTokenizer` class will grab the proper tokenizer class in the library based on the checkpoint name, and can be used directly with any checkpoint:
 
-{:else}
-Similar to `TFAutoModel`, the `AutoTokenizer` class will grab the proper tokenizer class in the library based on the checkpoint name, and can be used directly with any checkpoint:
-
-{/if}
-
 ```py
 from transformers import AutoTokenizer
 
diff --git a/chapters/en/chapter2/5.mdx b/chapters/en/chapter2/5.mdx
index 33060505b..299a15c5f 100644
--- a/chapters/en/chapter2/5.mdx
+++ b/chapters/en/chapter2/5.mdx
@@ -2,8 +2,6 @@
 
 # Handling multiple sequences[[handling-multiple-sequences]]
 
-{#if fw === 'pt'}
-
 <CourseFloatingBanner chapter={2}
   classNames="absolute z-10 right-0 top-0"
   notebooks={[
@@ -11,22 +9,7 @@
     {label: "Aws Studio", value: "https://studiolab.sagemaker.aws/import/github/huggingface/notebooks/blob/master/course/en/chapter2/section5_pt.ipynb"},
 ]} />
 
-{:else}
-
-<CourseFloatingBanner chapter={2}
-  classNames="absolute z-10 right-0 top-0"
-  notebooks={[
-    {label: "Google Colab", value: "https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/en/chapter2/section5_tf.ipynb"},
-    {label: "Aws Studio", value: "https://studiolab.sagemaker.aws/import/github/huggingface/notebooks/blob/master/course/en/chapter2/section5_tf.ipynb"},
-]} />
-
-{/if}
-
-{#if fw === 'pt'}
 <Youtube id="M6adb1j2jPI"/>
-{:else}
-<Youtube id="ROxrFOEbsQE"/>
-{/if}
 
 In the previous section, we explored the simplest of use cases: doing inference on a single sequence of a small length. However, some questions emerge already:
 
@@ -41,7 +24,6 @@ Let's see what kinds of problems these questions pose, and how we can solve them
 
 In the previous exercise you saw how sequences get translated into lists of numbers. Let's convert this list of numbers to a tensor and send it to the model:
 
-{#if fw === 'pt'}
 ```py
 import torch
 from transformers import AutoTokenizer, AutoModelForSequenceClassification
@@ -62,34 +44,11 @@ model(input_ids)
 ```python out
 IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
 ```
-{:else}
-```py
-import tensorflow as tf
-from transformers import AutoTokenizer, TFAutoModelForSequenceClassification
-
-checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
-tokenizer = AutoTokenizer.from_pretrained(checkpoint)
-model = TFAutoModelForSequenceClassification.from_pretrained(checkpoint)
-
-sequence = "I've been waiting for a HuggingFace course my whole life."
-
-tokens = tokenizer.tokenize(sequence)
-ids = tokenizer.convert_tokens_to_ids(tokens)
-input_ids = tf.constant(ids)
-# This line will fail.
-model(input_ids)
-```
-
-```py out
-InvalidArgumentError: Input to reshape is a tensor with 14 values, but the requested shape has 196 [Op:Reshape]
-```
-{/if}
 
 Oh no! Why did this fail? We followed the steps from the pipeline in section 2.
 
 The problem is that we sent a single sequence to the model, whereas 🤗 Transformers models expect multiple sentences by default. Here we tried to do everything the tokenizer did behind the scenes when we applied it to a `sequence`. But if you look closely, you'll see that the tokenizer didn't just convert the list of input IDs into a tensor, it added a dimension on top of it:
 
-{#if fw === 'pt'}
 ```py
 tokenized_inputs = tokenizer(sequence, return_tensors="pt")
 print(tokenized_inputs["input_ids"])
@@ -99,22 +58,9 @@ print(tokenized_inputs["input_ids"])
 tensor([[  101,  1045,  1005,  2310,  2042,  3403,  2005,  1037, 17662, 12172,
           2607,  2026,  2878,  2166,  1012,   102]])
 ```
-{:else}
-```py
-tokenized_inputs = tokenizer(sequence, return_tensors="tf")
-print(tokenized_inputs["input_ids"])
-```
-
-```py out
-<tf.Tensor: shape=(1, 16), dtype=int32, numpy=
-array([[  101,  1045,  1005,  2310,  2042,  3403,  2005,  1037, 17662,
-        12172,  2607,  2026,  2878,  2166,  1012,   102]], dtype=int32)>
-```
-{/if}
 
 Let's try again and add a new dimension:
 
-{#if fw === 'pt'}
 ```py
 import torch
 from transformers import AutoTokenizer, AutoModelForSequenceClassification
@@ -134,43 +80,13 @@ print("Input IDs:", input_ids)
 output = model(input_ids)
 print("Logits:", output.logits)
 ```
-{:else}
-```py
-import tensorflow as tf
-from transformers import AutoTokenizer, TFAutoModelForSequenceClassification
-
-checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
-tokenizer = AutoTokenizer.from_pretrained(checkpoint)
-model = TFAutoModelForSequenceClassification.from_pretrained(checkpoint)
-
-sequence = "I've been waiting for a HuggingFace course my whole life."
-
-tokens = tokenizer.tokenize(sequence)
-ids = tokenizer.convert_tokens_to_ids(tokens)
-
-input_ids = tf.constant([ids])
-print("Input IDs:", input_ids)
-
-output = model(input_ids)
-print("Logits:", output.logits)
-```
-{/if}
 
 We print the input IDs as well as the resulting logits — here's the output:
 
-{#if fw === 'pt'}
 ```python out
 Input IDs: [[ 1045,  1005,  2310,  2042,  3403,  2005,  1037, 17662, 12172,  2607, 2026,  2878,  2166,  1012]]
 Logits: [[-2.7276,  2.8789]]
 ```
-{:else}
-```py out
-Input IDs: tf.Tensor(
-[[ 1045  1005  2310  2042  3403  2005  1037 17662 12172  2607  2026  2878
-   2166  1012]], shape=(1, 14), dtype=int32)
-Logits: tf.Tensor([[-2.7276208  2.8789377]], shape=(1, 2), dtype=float32)
-```
-{/if}
 
 *Batching* is the act of sending multiple sentences through the model, all at once. If you only have one sentence, you can just build a batch with a single sequence: 
 
@@ -212,7 +128,6 @@ batched_ids = [
 
 The padding token ID can be found in `tokenizer.pad_token_id`. Let's use it and send our two sentences through the model individually and batched together:
 
-{#if fw === 'pt'}
 ```py no-format
 model = AutoModelForSequenceClassification.from_pretrained(checkpoint)
 
@@ -234,30 +149,6 @@ tensor([[ 0.5803, -0.4125]], grad_fn=<AddmmBackward>)
 tensor([[ 1.5694, -1.3895],
         [ 1.3373, -1.2163]], grad_fn=<AddmmBackward>)
 ```
-{:else}
-```py no-format
-model = TFAutoModelForSequenceClassification.from_pretrained(checkpoint)
-
-sequence1_ids = [[200, 200, 200]]
-sequence2_ids = [[200, 200]]
-batched_ids = [
-    [200, 200, 200],
-    [200, 200, tokenizer.pad_token_id],
-]
-
-print(model(tf.constant(sequence1_ids)).logits)
-print(model(tf.constant(sequence2_ids)).logits)
-print(model(tf.constant(batched_ids)).logits)
-```
-
-```py out
-tf.Tensor([[ 1.5693678 -1.3894581]], shape=(1, 2), dtype=float32)
-tf.Tensor([[ 0.5803005  -0.41252428]], shape=(1, 2), dtype=float32)
-tf.Tensor(
-[[ 1.5693681 -1.3894582]
- [ 1.3373486 -1.2163193]], shape=(2, 2), dtype=float32)
-```
-{/if}
 
 There's something wrong with the logits in our batched predictions: the second row should be the same as the logits for the second sentence, but we've got completely different values!
 
@@ -269,7 +160,6 @@ This is because the key feature of Transformer models is attention layers that *
 
 Let's complete the previous example with an attention mask:
 
-{#if fw === 'pt'}
 ```py no-format
 batched_ids = [
     [200, 200, 200],
@@ -289,28 +179,6 @@ print(outputs.logits)
 tensor([[ 1.5694, -1.3895],
         [ 0.5803, -0.4125]], grad_fn=<AddmmBackward>)
 ```
-{:else}
-```py no-format
-batched_ids = [
-    [200, 200, 200],
-    [200, 200, tokenizer.pad_token_id],
-]
-
-attention_mask = [
-    [1, 1, 1],
-    [1, 1, 0],
-]
-
-outputs = model(tf.constant(batched_ids), attention_mask=tf.constant(attention_mask))
-print(outputs.logits)
-```
-
-```py out
-tf.Tensor(
-[[ 1.5693681  -1.3894582 ]
- [ 0.5803021  -0.41252586]], shape=(2, 2), dtype=float32)
-```
-{/if}
 
 Now we get the same logits for the second sentence in the batch.
 
diff --git a/chapters/en/chapter2/6.mdx b/chapters/en/chapter2/6.mdx
index d26118501..3a0dac876 100644
--- a/chapters/en/chapter2/6.mdx
+++ b/chapters/en/chapter2/6.mdx
@@ -2,8 +2,6 @@
 
 # Putting it all together[[putting-it-all-together]]
 
-{#if fw === 'pt'}
-
 <CourseFloatingBanner chapter={2}
   classNames="absolute z-10 right-0 top-0"
   notebooks={[
@@ -11,17 +9,6 @@
     {label: "Aws Studio", value: "https://studiolab.sagemaker.aws/import/github/huggingface/notebooks/blob/master/course/en/chapter2/section6_pt.ipynb"},
 ]} />
 
-{:else}
-
-<CourseFloatingBanner chapter={2}
-  classNames="absolute z-10 right-0 top-0"
-  notebooks={[
-    {label: "Google Colab", value: "https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/en/chapter2/section6_tf.ipynb"},
-    {label: "Aws Studio", value: "https://studiolab.sagemaker.aws/import/github/huggingface/notebooks/blob/master/course/en/chapter2/section6_tf.ipynb"},
-]} />
-
-{/if}
-
 In the last few sections, we've been trying our best to do most of the work by hand. We've explored how tokenizers work and looked at tokenization, conversion to input IDs, padding, truncation, and attention masks.
 
 However, as we saw in section 2, the 🤗 Transformers API can handle all of this for us with a high-level function that we'll dive into here. When you call your `tokenizer` directly on the sentence, you get back inputs that are ready to pass through your model:
@@ -82,7 +69,7 @@ model_inputs = tokenizer(sequences, truncation=True)
 model_inputs = tokenizer(sequences, max_length=8, truncation=True)
 ```
 
-The `tokenizer` object can handle the conversion to specific framework tensors, which can then be directly sent to the model. For example, in the following code sample we are prompting the tokenizer to return tensors from the different frameworks — `"pt"` returns PyTorch tensors, `"tf"` returns TensorFlow tensors, and `"np"` returns NumPy arrays:
+The `tokenizer` object can handle the conversion to specific framework tensors, which can then be directly sent to the model. For example, in the following code sample we are prompting the tokenizer to return tensors from the different frameworks — `"pt"` returns PyTorch tensors and `"np"` returns NumPy arrays:
 
 ```py
 sequences = ["I've been waiting for a HuggingFace course my whole life.", "So have I!"]
@@ -90,9 +77,6 @@ sequences = ["I've been waiting for a HuggingFace course my whole life.", "So ha
 # Returns PyTorch tensors
 model_inputs = tokenizer(sequences, padding=True, return_tensors="pt")
 
-# Returns TensorFlow tensors
-model_inputs = tokenizer(sequences, padding=True, return_tensors="tf")
-
 # Returns NumPy arrays
 model_inputs = tokenizer(sequences, padding=True, return_tensors="np")
 ```
@@ -135,7 +119,6 @@ The tokenizer added the special word `[CLS]` at the beginning and the special wo
 
 Now that we've seen all the individual steps the `tokenizer` object uses when applied on texts, let's see one final time how it can handle multiple sequences (padding!), very long sequences (truncation!), and multiple types of tensors with its main API:
 
-{#if fw === 'pt'}
 ```py
 import torch
 from transformers import AutoTokenizer, AutoModelForSequenceClassification
@@ -148,17 +131,3 @@ sequences = ["I've been waiting for a HuggingFace course my whole life.", "So ha
 tokens = tokenizer(sequences, padding=True, truncation=True, return_tensors="pt")
 output = model(**tokens)
 ```
-{:else}
-```py
-import tensorflow as tf
-from transformers import AutoTokenizer, TFAutoModelForSequenceClassification
-
-checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
-tokenizer = AutoTokenizer.from_pretrained(checkpoint)
-model = TFAutoModelForSequenceClassification.from_pretrained(checkpoint)
-sequences = ["I've been waiting for a HuggingFace course my whole life.", "So have I!"]
-
-tokens = tokenizer(sequences, padding=True, truncation=True, return_tensors="tf")
-output = model(**tokens)
-```
-{/if}
diff --git a/chapters/en/chapter2/8.mdx b/chapters/en/chapter2/8.mdx
index c41f27936..f2f7ab431 100644
--- a/chapters/en/chapter2/8.mdx
+++ b/chapters/en/chapter2/8.mdx
@@ -1,310 +1,821 @@
-<FrameworkSwitchCourse {fw} />
-
-<!-- DISABLE-FRONTMATTER-SECTIONS -->
-
-# End-of-chapter quiz[[end-of-chapter-quiz]]
-
-<CourseFloatingBanner
-    chapter={2}
-    classNames="absolute z-10 right-0 top-0"
-/>
-
-### 1. What is the order of the language modeling pipeline?
-
-<Question
-	choices={[
-		{
-			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
-			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
-		},
-		{
-			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
-			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
-		},
-		{
-			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
-			explain: "Correct! The tokenizer can be used for both tokenizing and de-tokenizing.",
-            correct: true
-		}
-	]}
-/>
-
-### 2. How many dimensions does the tensor output by the base Transformer model have, and what are they?
-
-<Question
-	choices={[
-		{
-			text: "2: The sequence length and the batch size",
-			explain: "False! The tensor output by the model has a third dimension: hidden size."
-		},
-		{
-			text: "2: The sequence length and the hidden size",
-			explain: "False! All Transformer models handle batches, even with a single sequence; that would be a batch size of 1!"
-		},
-		{
-			text: "3: The sequence length, the batch size, and the hidden size",
-			explain: "Correct!",
-            correct: true
-		}
-	]}
-/>
-
-### 3. Which of the following is an example of subword tokenization?
-
-<Question
-	choices={[
-		{
-			text: "WordPiece",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-		},
-		{
-			text: "Character-based tokenization",
-			explain: "Character-based tokenization is not a type of subword tokenization."
-		},
-		{
-			text: "Splitting on whitespace and punctuation",
-			explain: "That's a word-based tokenization scheme!"
-		},
-		{
-			text: "BPE",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-        },
-		{
-			text: "Unigram",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-        },
-		{
-			text: "None of the above",
-			explain: "Incorrect!"
-        }
-	]}
-/>
-
-### 4. What is a model head?
-
-<Question
-	choices={[
-		{
-			text: "A component of the base Transformer network that redirects tensors to their correct layers",
-			explain: "Incorrect! There's no such component."
-		},
-		{
-			text: "Also known as the self-attention mechanism, it adapts the representation of a token according to the other tokens of the sequence",
-			explain: "Incorrect! The self-attention layer does contain attention \"heads,\" but these are not adaptation heads."
-		},
-		{
-			text: "An additional component, usually made up of one or a few layers, to convert the transformer predictions to a task-specific output",
-			explain: "That's right. Adaptation heads, also known simply as heads, come up in different forms: language modeling heads, question answering heads, sequence classification heads... ",
-			correct: true
-		} 
-	]}
-/>
-
-{#if fw === 'pt'}
-### 5. What is an AutoModel?
-
-<Question
-	choices={[
-		{
-			text: "A model that automatically trains on your data",
-			explain: "Incorrect. Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
-		},
-		{
-			text: "An object that returns the correct architecture based on the checkpoint",
-			explain: "Exactly: the <code>AutoModel</code> only needs to know the checkpoint from which to initialize to return the correct architecture.",
-			correct: true
-		},
-		{
-			text: "A model that automatically detects the language used for its inputs to load the correct weights",
-			explain: "Incorrect; while some checkpoints and models are capable of handling multiple languages, there are no built-in tools for automatic checkpoint selection according to language. You should head over to the <a href='https://huggingface.co/models'>Model Hub</a> to find the best checkpoint for your task!"
-		} 
-	]}
-/>
-
-{:else}
-### 5. What is an TFAutoModel?
-
-<Question
-	choices={[
-		{
-			text: "A model that automatically trains on your data",
-			explain: "Incorrect. Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
-		},
-		{
-			text: "An object that returns the correct architecture based on the checkpoint",
-			explain: "Exactly: the <code>TFAutoModel</code> only needs to know the checkpoint from which to initialize to return the correct architecture.",
-			correct: true
-		},
-		{
-			text: "A model that automatically detects the language used for its inputs to load the correct weights",
-			explain: "Incorrect; while some checkpoints and models are capable of handling multiple languages, there are no built-in tools for automatic checkpoint selection according to language. You should head over to the <a href='https://huggingface.co/models'>Model Hub</a> to find the best checkpoint for your task!"
-		} 
-	]}
-/>
-
-{/if}
-
-### 6. What are the techniques to be aware of when batching sequences of different lengths together?
-
-<Question
-	choices={[
-		{
-			text: "Truncating",
-			explain: "Yes, truncation is a correct way of evening out sequences so that they fit in a rectangular shape. Is it the only one, though?",
-			correct: true
-		},
-		{
-			text: "Returning tensors",
-			explain: "While the other techniques allow you to return rectangular tensors, returning tensors isn't helpful when batching sequences together."
-		},
-		{
-			text: "Padding",
-			explain: "Yes, padding is a correct way of evening out sequences so that they fit in a rectangular shape. Is it the only one, though?",
-			correct: true
-		}, 
-		{
-			text: "Attention masking",
-			explain: "Absolutely! Attention masks are of prime importance when handling sequences of different lengths. That's not the only technique to be aware of, however.",
-			correct: true
-		} 
-	]}
-/>
-
-### 7. What is the point of applying a SoftMax function to the logits output by a sequence classification model?
-
-<Question
-	choices={[
-		{
-			text: "It softens the logits so that they're more reliable.",
-			explain: "No, the SoftMax function does not affect the reliability of results."
-		},
-		{
-			text: "It applies a lower and upper bound so that they're understandable.",
-			explain: "Correct! The resulting values are bound between 0 and 1. That's not the only reason we use a SoftMax function, though.",
-            correct: true
-		},
-		{
-			text: "The total sum of the output is then 1, resulting in a possible probabilistic interpretation.",
-			explain: "Correct! That's not the only reason we use a SoftMax function, though.",
-            correct: true
-		}
-	]}
-/>
-
-### 8. What method is most of the tokenizer API centered around?
-
-<Question
-	choices={[
-		{
-			text: "<code>encode</code>, as it can encode text into IDs and IDs into predictions",
-			explain: "Wrong! While the <code>encode</code> method does exist on tokenizers, it does not exist on models."
-		},
-		{
-			text: "Calling the tokenizer object directly.",
-			explain: "Exactly! The <code>__call__</code> method of the tokenizer is a very powerful method which can handle pretty much anything. It is also the method used to retrieve predictions from a model.",
-			correct: true
-		},
-		{
-			text: "<code>pad</code>",
-			explain: "Wrong! Padding is very useful, but it's just one part of the tokenizer API."
-		},
-		{
-			text: "<code>tokenize</code>",
-			explain: "The <code>tokenize</code> method is arguably one of the most useful methods, but it isn't the core of the tokenizer API."
-		}
-	]}
-/>
-
-### 9. What does the `result` variable contain in this code sample?
-
-```py
-from transformers import AutoTokenizer
-
-tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
-result = tokenizer.tokenize("Hello!")
-```
-
-<Question
-	choices={[
-		{
-			text: "A list of strings, each string being a token",
-			explain: "Absolutely! Convert this to IDs, and send them to a model!",
-            correct: true
-		},
-		{
-			text: "A list of IDs",
-			explain: "Incorrect; that's what the <code>__call__</code> or <code>convert_tokens_to_ids</code> method is for!"
-		},
-		{
-			text: "A string containing all of the tokens",
-			explain: "This would be suboptimal, as the goal is to split the string into multiple tokens."
-		}
-	]}
-/>
-
-{#if fw === 'pt'}
-### 10. Is there something wrong with the following code?
-
-```py
-from transformers import AutoTokenizer, AutoModel
-
-tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
-model = AutoModel.from_pretrained("gpt2")
-
-encoded = tokenizer("Hey!", return_tensors="pt")
-result = model(**encoded)
-```
-
-<Question
-	choices={[
-		{
-			text: "No, it seems correct.",
-			explain: "Unfortunately, coupling a model with a tokenizer that was trained with a different checkpoint is rarely a good idea. The model was not trained to make sense out of this tokenizer's output, so the model output (if it can even run!) will not make any sense."
-		},
-		{
-			text: "The tokenizer and model should always be from the same checkpoint.",
-			explain: "Right!",
-            correct: true
-		},
-		{
-			text: "It's good practice to pad and truncate with the tokenizer as every input is a batch.",
-			explain: "It's true that every model input needs to be a batch. However, truncating or padding this sequence wouldn't necessarily make sense as there is only one of it, and those are techniques to batch together a list of sentences."
-		}
-	]}
-/>
-
-{:else}
-### 10. Is there something wrong with the following code?
-
-```py
-from transformers import AutoTokenizer, TFAutoModel
-
-tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
-model = TFAutoModel.from_pretrained("gpt2")
-
-encoded = tokenizer("Hey!", return_tensors="pt")
-result = model(**encoded)
-```
-
-<Question
-	choices={[
-		{
-			text: "No, it seems correct.",
-			explain: "Unfortunately, coupling a model with a tokenizer that was trained with a different checkpoint is rarely a good idea. The model was not trained to make sense out of this tokenizer's output, so the model output (if it can even run!) will not make any sense."
-		},
-		{
-			text: "The tokenizer and model should always be from the same checkpoint.",
-			explain: "Right!",
-            correct: true
-		},
-		{
-			text: "It's good practice to pad and truncate with the tokenizer as every input is a batch.",
-			explain: "It's true that every model input needs to be a batch. However, truncating or padding this sequence wouldn't necessarily make sense as there is only one of it, and those are techniques to batch together a list of sentences."
-		}
-	]}
-/>
-
-{/if}
+# Optimized Inference Deployment
+
+In this section, we'll explore advanced frameworks for optimizing LLM deployments: Text Generation Inference (TGI), vLLM, and llama.cpp. These applications are primarily used in production environments to serve LLMs to users. This section focuses on how to deploy these frameworks in production rather than how to use them for inference on a single machine.
+
+We'll cover how these tools maximize inference efficiency and simplify production deployments of Large Language Models.
+
+## Framework Selection Guide
+
+TGI, vLLM, and llama.cpp serve similar purposes but have distinct characteristics that make them better suited for different use cases. Let's look at the key differences between them, focusing on performance and integration.
+
+### Memory Management and Performance
+
+**TGI** is designed to be stable and predictable in production, using fixed sequence lengths to keep memory usage consistent. TGI manages memory using Flash Attention 2 and continuous batching techniques. This means it can process attention calculations very efficiently and keep the GPU busy by constantly feeding it work. The system can move parts of the model between CPU and GPU when needed, which helps handle larger models. 
+
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/tgi/flash-attn.png" alt="Flash Attention" />
+
+<Tip title="How Flash Attention Works">
+Flash Attention is a technique that optimizes the attention mechanism in transformer models by addressing memory bandwidth bottlenecks. As discussed earlier in [Chapter 1.8](/course/chapter1/8), the attention mechanism has quadratic complexity and memory usage, making it inefficient for long sequences.
+
+The key innovation is in how it manages memory transfers between High Bandwidth Memory (HBM) and faster SRAM cache. Traditional attention repeatedly transfers data between HBM and SRAM, creating bottlenecks by leaving the GPU idle. Flash Attention loads data once into SRAM and performs all calculations there, minimizing expensive memory transfers. 
+
+While the benefits are most significant during training, Flash Attention's reduced VRAM usage and improved efficiency make it valuable for inference as well, enabling faster and more scalable LLM serving.
+</Tip>
+
+**vLLM** takes a different approach by using PagedAttention. Just like how a computer manages its memory in pages, vLLM splits the model's memory into smaller blocks. This clever system means it can handle different-sized requests more flexibly and doesn't waste memory space. It's particularly good at sharing memory between different requests and reduces memory fragmentation, which makes the whole system more efficient.
+
+<Tip title="How PagedAttention Works">
+PagedAttention is a technique that addresses another critical bottleneck in LLM inference: KV cache memory management. As discussed in [Chapter 1.8](/course/chapter1/8), during text generation, the model stores attention keys and values (KV cache) for each generated token to reduce redundant computations. The KV cache can become enormous, especially with long sequences or multiple concurrent requests.
+
+vLLM's key innovation lies in how it manages this cache:
+
+1. **Memory Paging**: Instead of treating the KV cache as one large block, it's divided into fixed-size "pages" (similar to virtual memory in operating systems).
+2. **Non-contiguous Storage**: Pages don't need to be stored contiguously in GPU memory, allowing for more flexible memory allocation.
+3. **Page Table Management**: A page table tracks which pages belong to which sequence, enabling efficient lookup and access.
+4. **Memory Sharing**: For operations like parallel sampling, pages storing the KV cache for the prompt can be shared across multiple sequences.
+
+The PagedAttention approach can lead to up to 24x higher throughput compared to traditional methods, making it a game-changer for production LLM deployments. If you want to go really deep into how PagedAttention works, you can read the [the guide from the vLLM documentation](https://docs.vllm.ai/en/latest/design/kernel/paged_attention.html).
+</Tip>
+
+**llama.cpp** is a highly optimized C/C++ implementation originally designed for running LLaMA models on consumer hardware. It focuses on CPU efficiency with optional GPU acceleration and is ideal for resource-constrained environments. llama.cpp uses quantization techniques to reduce model size and memory requirements while maintaining good performance. It implements optimized kernels for various CPU architectures and supports basic KV cache management for efficient token generation.
+
+<Tip title="How llama.cpp Quantization Works">
+Quantization in llama.cpp reduces the precision of model weights from 32-bit or 16-bit floating point to lower precision formats like 8-bit integers (INT8), 4-bit, or even lower. This significantly reduces memory usage and improves inference speed with minimal quality loss.
+
+Key quantization features in llama.cpp include:
+1. **Multiple Quantization Levels**: Supports 8-bit, 4-bit, 3-bit, and even 2-bit quantization
+2. **GGML/GGUF Format**: Uses custom tensor formats optimized for quantized inference
+3. **Mixed Precision**: Can apply different quantization levels to different parts of the model
+4. **Hardware-Specific Optimizations**: Includes optimized code paths for various CPU architectures (AVX2, AVX-512, NEON)
+
+This approach enables running billion-parameter models on consumer hardware with limited memory, making it perfect for local deployments and edge devices.
+</Tip>
+
+
+
+### Deployment and Integration
+
+Let's move on to the deployment and integration differences between the frameworks.
+
+**TGI** excels in enterprise-level deployment with its production-ready features. It comes with built-in Kubernetes support and includes everything you need for running in production, like monitoring through Prometheus and Grafana, automatic scaling, and comprehensive safety features. The system also includes enterprise-grade logging and various protective measures like content filtering and rate limiting to keep your deployment secure and stable.
+
+**vLLM** takes a more flexible, developer-friendly approach to deployment. It's built with Python at its core and can easily replace OpenAI's API in your existing applications. The framework focuses on delivering raw performance and can be customized to fit your specific needs. It works particularly well with Ray for managing clusters, making it a great choice when you need high performance and adaptability.
+
+**llama.cpp** prioritizes simplicity and portability. Its server implementation is lightweight and can run on a wide range of hardware, from powerful servers to consumer laptops and even some high-end mobile devices. With minimal dependencies and a simple C/C++ core, it's easy to deploy in environments where installing Python frameworks would be challenging. The server provides an OpenAI-compatible API while maintaining a much smaller resource footprint than other solutions.
+
+
+
+## Getting Started
+
+Let's explore how to use these frameworks for deploying LLMs, starting with installation and basic setup.
+
+### Installation and Basic Setup
+
+<hfoptions id="inference-frameworks" >
+
+<hfoption value="tgi" label="TGI">
+
+TGI is easy to install and use, with deep integration into the Hugging Face ecosystem.
+
+First, launch the TGI server using Docker:
+
+```sh
+docker run --gpus all \
+    --shm-size 1g \
+    -p 8080:80 \
+    -v ~/.cache/huggingface:/data \
+    ghcr.io/huggingface/text-generation-inference:latest \
+    --model-id HuggingFaceTB/SmolLM2-360M-Instruct
+```
+
+Then interact with it using Hugging Face's InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+# Initialize client pointing to TGI endpoint
+client = InferenceClient(
+    model="http://localhost:8080",  # URL to the TGI server
+)
+
+# Text generation
+response = client.text_generation(
+    "Tell me a story",
+    max_new_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+    details=True,
+    stop_sequences=[],
+)
+print(response.generated_text)
+
+# For chat format
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
+
+Alternatively, you can use the OpenAI client:
+
+```python
+from openai import OpenAI
+
+# Initialize client pointing to TGI endpoint
+client = OpenAI(
+    base_url="http://localhost:8080/v1",  # Make sure to include /v1
+    api_key="not-needed",  # TGI doesn't require an API key by default
+)
+
+# Chat completion
+response = client.chat.completions.create(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
+</hfoption>
+<hfoption value="llama.cpp" label="llama.cpp">
+
+llama.cpp is easy to install and use, requiring minimal dependencies and supporting both CPU and GPU inference.
+
+First, install and build llama.cpp:
+
+```sh
+# Clone the repository
+git clone https://github.com/ggerganov/llama.cpp
+cd llama.cpp
+
+# Build the project
+make
+
+# Download the SmolLM2-1.7B-Instruct-GGUF model
+curl -L -O https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct-GGUF/resolve/main/smollm2-1.7b-instruct.Q4_K_M.gguf
+```
+
+Then, launch the server (with OpenAI API compatibility):
+
+```sh
+# Start the server
+./server \
+    -m smollm2-1.7b-instruct.Q4_K_M.gguf \
+    --host 0.0.0.0 \
+    --port 8080 \
+    -c 4096 \
+    --n-gpu-layers 0  # Set to a higher number to use GPU
+```
+
+Interact with the server using Hugging Face's InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+# Initialize client pointing to llama.cpp server
+client = InferenceClient(
+    model="http://localhost:8080/v1",  # URL to the llama.cpp server
+    token="sk-no-key-required",  # llama.cpp server requires this placeholder
+)
+
+# Text generation
+response = client.text_generation(
+    "Tell me a story",
+    max_new_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+    details=True,
+)
+print(response.generated_text)
+
+# For chat format
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
+
+Alternatively, you can use the OpenAI client:
+
+```python
+from openai import OpenAI
+
+# Initialize client pointing to llama.cpp server
+client = OpenAI(
+    base_url="http://localhost:8080/v1",
+    api_key="sk-no-key-required",  # llama.cpp server requires this placeholder
+)
+
+# Chat completion
+response = client.chat.completions.create(
+    model="smollm2-1.7b-instruct",  # Model identifier can be anything as server only loads one model
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
+</hfoption>
+<hfoption value="vllm" label="vLLM">
+
+vLLM is easy to install and use, with both OpenAI API compatibility and a native Python interface.
+
+First, launch the vLLM OpenAI-compatible server:
+
+```sh
+python -m vllm.entrypoints.openai.api_server \
+    --model HuggingFaceTB/SmolLM2-360M-Instruct \
+    --host 0.0.0.0 \
+    --port 8000
+```
+
+Then interact with it using Hugging Face's InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+# Initialize client pointing to vLLM endpoint
+client = InferenceClient(
+    model="http://localhost:8000/v1",  # URL to the vLLM server
+)
+
+# Text generation
+response = client.text_generation(
+    "Tell me a story",
+    max_new_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+    details=True,
+)
+print(response.generated_text)
+
+# For chat format
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
+
+Alternatively, you can use the OpenAI client:
+
+```python
+from openai import OpenAI
+
+# Initialize client pointing to vLLM endpoint
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="not-needed",  # vLLM doesn't require an API key by default
+)
+
+# Chat completion
+response = client.chat.completions.create(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
+</hfoption>
+
+</hfoptions>
+
+### Basic Text Generation
+
+Let's look at examples of text generation with the frameworks:
+
+<hfoptions id="inference-frameworks" >
+
+<hfoption value="tgi" label="TGI">
+
+First, deploy TGI with advanced parameters:
+```sh
+docker run --gpus all \
+    --shm-size 1g \
+    -p 8080:80 \
+    -v ~/.cache/huggingface:/data \
+    ghcr.io/huggingface/text-generation-inference:latest \
+    --model-id HuggingFaceTB/SmolLM2-360M-Instruct \
+    --max-total-tokens 4096 \
+    --max-input-length 3072 \
+    --max-batch-total-tokens 8192 \
+    --waiting-served-ratio 1.2
+```
+
+Use the InferenceClient for flexible text generation:
+```python
+from huggingface_hub import InferenceClient
+
+client = InferenceClient(model="http://localhost:8080")
+
+# Advanced parameters example
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,
+    max_tokens=200,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+
+# Raw text generation
+response = client.text_generation(
+    "Write a creative story about space exploration",
+    max_new_tokens=200,
+    temperature=0.8,
+    top_p=0.95,
+    repetition_penalty=1.1,
+    do_sample=True,
+    details=True,
+)
+print(response.generated_text)
+```
+
+Or use the OpenAI client:
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
+
+# Advanced parameters example
+response = client.chat.completions.create(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,  # Higher for more creativity
+)
+print(response.choices[0].message.content)
+```
+</hfoption>
+<hfoption value="llama.cpp" label="llama.cpp">
+
+For llama.cpp, you can set advanced parameters when launching the server:
+
+```sh
+./server \
+    -m smollm2-1.7b-instruct.Q4_K_M.gguf \
+    --host 0.0.0.0 \
+    --port 8080 \
+    -c 4096 \            # Context size
+    --threads 8 \        # CPU threads to use
+    --batch-size 512 \   # Batch size for prompt evaluation
+    --n-gpu-layers 0     # GPU layers (0 = CPU only)
+```
+
+Use the InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+client = InferenceClient(model="http://localhost:8080/v1", token="sk-no-key-required")
+
+# Advanced parameters example
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,
+    max_tokens=200,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+
+# For direct text generation
+response = client.text_generation(
+    "Write a creative story about space exploration",
+    max_new_tokens=200,
+    temperature=0.8,
+    top_p=0.95,
+    repetition_penalty=1.1,
+    details=True,
+)
+print(response.generated_text)
+```
+
+Or use the OpenAI client for generation with control over the sampling parameters:
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8080/v1", api_key="sk-no-key-required")
+
+# Advanced parameters example
+response = client.chat.completions.create(
+    model="smollm2-1.7b-instruct",
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Nucleus sampling probability
+    frequency_penalty=0.5,  # Reduce repetition of frequent tokens
+    presence_penalty=0.5,  # Reduce repetition by penalizing tokens already present
+    max_tokens=200,  # Maximum generation length
+)
+print(response.choices[0].message.content)
+```
+
+You can also use llama.cpp's native library for even more control:
+
+```python
+# Using llama-cpp-python package for direct model access
+from llama_cpp import Llama
+
+# Load the model
+llm = Llama(
+    model_path="smollm2-1.7b-instruct.Q4_K_M.gguf",
+    n_ctx=4096,  # Context window size
+    n_threads=8,  # CPU threads
+    n_gpu_layers=0,  # GPU layers (0 = CPU only)
+)
+
+# Format prompt according to the model's expected format
+prompt = """
system
+You are a creative storyteller.
+

+
user
+Write a creative story
+

+
assistant
+"""
+
+# Generate response with precise parameter control
+output = llm(
+    prompt,
+    max_tokens=200,
+    temperature=0.8,
+    top_p=0.95,
+    frequency_penalty=0.5,
+    presence_penalty=0.5,
+    stop=["
"],
+)
+
+print(output["choices"][0]["text"])
+```
+</hfoption>
+<hfoption value="vllm" label="vLLM">
+
+For advanced usage with vLLM, you can use the InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+client = InferenceClient(model="http://localhost:8000/v1")
+
+# Advanced parameters example
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,
+    max_tokens=200,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+
+# For direct text generation
+response = client.text_generation(
+    "Write a creative story about space exploration",
+    max_new_tokens=200,
+    temperature=0.8,
+    top_p=0.95,
+    details=True,
+)
+print(response.generated_text)
+```
+
+You can also use the OpenAI client:
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
+
+# Advanced parameters example
+response = client.chat.completions.create(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,
+    top_p=0.95,
+    max_tokens=200,
+)
+print(response.choices[0].message.content)
+```
+
+vLLM also provides a native Python interface with fine-grained control:
+
+```python
+from vllm import LLM, SamplingParams
+
+# Initialize the model with advanced parameters
+llm = LLM(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    gpu_memory_utilization=0.85,
+    max_num_batched_tokens=8192,
+    max_num_seqs=256,
+    block_size=16,
+)
+
+# Configure sampling parameters
+sampling_params = SamplingParams(
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Consider top 95% probability mass
+    max_tokens=100,  # Maximum length
+    presence_penalty=1.1,  # Reduce repetition
+    frequency_penalty=1.1,  # Reduce repetition
+    stop=["\n\n", "###"],  # Stop sequences
+)
+
+# Generate text
+prompt = "Write a creative story"
+outputs = llm.generate(prompt, sampling_params)
+print(outputs[0].outputs[0].text)
+
+# For chat-style interactions
+chat_prompt = [
+    {"role": "system", "content": "You are a creative storyteller."},
+    {"role": "user", "content": "Write a creative story"},
+]
+formatted_prompt = llm.get_chat_template()(chat_prompt)  # Uses model's chat template
+outputs = llm.generate(formatted_prompt, sampling_params)
+print(outputs[0].outputs[0].text)
+```
+</hfoption>
+
+</hfoptions>
+
+## Advanced Generation Control
+
+### Token Selection and Sampling
+
+The process of generating text involves selecting the next token at each step. This selection process can be controlled through various parameters:
+
+1. **Raw Logits**: The initial output probabilities for each token
+2. **Temperature**: Controls randomness in selection (higher = more creative)
+3. **Top-p (Nucleus) Sampling**: Filters to top tokens making up X% of probability mass
+4. **Top-k Filtering**: Limits selection to k most likely tokens
+
+Here's how to configure these parameters:
+
+<hfoptions id="inference-frameworks" >
+
+<hfoption value="tgi" label="TGI">
+
+```python
+client.generate(
+    "Write a creative story",
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Consider top 95% probability mass
+    top_k=50,  # Consider top 50 tokens
+    max_new_tokens=100,  # Maximum length
+    repetition_penalty=1.1,  # Reduce repetition
+)
+```
+</hfoption>
+<hfoption value="llama.cpp" label="llama.cpp">
+
+```python
+# Via OpenAI API compatibility
+response = client.completions.create(
+    model="smollm2-1.7b-instruct",  # Model name (can be any string for llama.cpp server)
+    prompt="Write a creative story",
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Consider top 95% probability mass
+    frequency_penalty=1.1,  # Reduce repetition
+    presence_penalty=0.1,  # Reduce repetition
+    max_tokens=100,  # Maximum length
+)
+
+# Via llama-cpp-python direct access
+output = llm(
+    "Write a creative story",
+    temperature=0.8,
+    top_p=0.95,
+    top_k=50,
+    max_tokens=100,
+    repeat_penalty=1.1,
+)
+```
+</hfoption>
+<hfoption value="vllm" label="vLLM">
+
+```python
+params = SamplingParams(
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Consider top 95% probability mass
+    top_k=50,  # Consider top 50 tokens
+    max_tokens=100,  # Maximum length
+    presence_penalty=0.1,  # Reduce repetition
+)
+llm.generate("Write a creative story", sampling_params=params)
+```
+</hfoption>
+
+</hfoptions>
+
+### Controlling Repetition
+
+Both frameworks provide ways to prevent repetitive text generation:
+
+<hfoptions id="inference-frameworks" >
+
+<hfoption value="tgi" label="TGI">
+```python
+client.generate(
+    "Write a varied text",
+    repetition_penalty=1.1,  # Penalize repeated tokens
+    no_repeat_ngram_size=3,  # Prevent 3-gram repetition
+)
+```
+</hfoption>
+<hfoption value="llama.cpp" label="llama.cpp">
+
+```python
+# Via OpenAI API
+response = client.completions.create(
+    model="smollm2-1.7b-instruct",
+    prompt="Write a varied text",
+    frequency_penalty=1.1,  # Penalize frequent tokens
+    presence_penalty=0.8,  # Penalize tokens already present
+)
+
+# Via direct library
+output = llm(
+    "Write a varied text",
+    repeat_penalty=1.1,  # Penalize repeated tokens
+    frequency_penalty=0.5,  # Additional frequency penalty
+    presence_penalty=0.5,  # Additional presence penalty
+)
+```
+</hfoption>
+<hfoption value="vllm" label="vLLM">
+
+```python
+params = SamplingParams(
+    presence_penalty=0.1,  # Penalize token presence
+    frequency_penalty=0.1,  # Penalize token frequency
+)
+```
+</hfoption>
+
+</hfoptions>
+
+### Length Control and Stop Sequences
+
+You can control generation length and specify when to stop:
+
+<hfoptions id="inference-frameworks" >
+
+<hfoption value="tgi" label="TGI">
+```python
+client.generate(
+    "Generate a short paragraph",
+    max_new_tokens=100,
+    min_new_tokens=10,
+    stop_sequences=["\n\n", "###"],
+)
+```
+</hfoption>
+<hfoption value="llama.cpp" label="llama.cpp">
+
+```python
+# Via OpenAI API
+response = client.completions.create(
+    model="smollm2-1.7b-instruct",
+    prompt="Generate a short paragraph",
+    max_tokens=100,
+    stop=["\n\n", "###"],
+)
+
+# Via direct library
+output = llm("Generate a short paragraph", max_tokens=100, stop=["\n\n", "###"])
+```
+</hfoption>
+<hfoption value="vllm" label="vLLM">
+
+```python
+params = SamplingParams(
+    max_tokens=100,
+    min_tokens=10,
+    stop=["###", "\n\n"],
+    ignore_eos=False,
+    skip_special_tokens=True,
+)
+```
+</hfoption>
+
+</hfoptions>
+
+## Memory Management
+
+Both frameworks implement advanced memory management techniques for efficient inference.
+
+<hfoptions id="inference-frameworks" >
+
+<hfoption value="tgi" label="TGI">
+TGI uses Flash Attention 2 and continuous batching:
+
+```sh
+# Docker deployment with memory optimization
+docker run --gpus all -p 8080:80 \
+    --shm-size 1g \
+    ghcr.io/huggingface/text-generation-inference:latest \
+    --model-id HuggingFaceTB/SmolLM2-1.7B-Instruct \
+    --max-batch-total-tokens 8192 \
+    --max-input-length 4096
+```
+</hfoption>
+<hfoption value="llama.cpp" label="llama.cpp">
+
+llama.cpp uses quantization and optimized memory layout:
+
+```sh
+# Server with memory optimizations
+./server \
+    -m smollm2-1.7b-instruct.Q4_K_M.gguf \
+    --host 0.0.0.0 \
+    --port 8080 \
+    -c 2048 \               # Context size
+    --threads 4 \           # CPU threads
+    --n-gpu-layers 32 \     # Use more GPU layers for larger models
+    --mlock \               # Lock memory to prevent swapping
+    --cont-batching         # Enable continuous batching
+```
+
+For models too large for your GPU, you can use CPU offloading:
+
+```sh
+./server \
+    -m smollm2-1.7b-instruct.Q4_K_M.gguf \
+    --n-gpu-layers 20 \     # Keep first 20 layers on GPU
+    --threads 8             # Use more CPU threads for CPU layers
+```
+</hfoption>
+<hfoption value="vllm" label="vLLM">
+
+vLLM uses PagedAttention for optimal memory management:
+
+```python
+from vllm.engine.arg_utils import AsyncEngineArgs
+
+engine_args = AsyncEngineArgs(
+    model="HuggingFaceTB/SmolLM2-1.7B-Instruct",
+    gpu_memory_utilization=0.85,
+    max_num_batched_tokens=8192,
+    block_size=16,
+)
+
+llm = LLM(engine_args=engine_args)
+```
+</hfoption>
+
+</hfoptions>
+
+## Resources
+
+- [Text Generation Inference Documentation](https://huggingface.co/docs/text-generation-inference)
+- [TGI GitHub Repository](https://github.com/huggingface/text-generation-inference)
+- [vLLM Documentation](https://vllm.readthedocs.io/)
+- [vLLM GitHub Repository](https://github.com/vllm-project/vllm)
+- [PagedAttention Paper](https://arxiv.org/abs/2309.06180)
+- [llama.cpp GitHub Repository](https://github.com/ggerganov/llama.cpp)
+- [llama-cpp-python Repository](https://github.com/abetlen/llama-cpp-python)
\ No newline at end of file
diff --git a/chapters/en/chapter2/9.mdx b/chapters/en/chapter2/9.mdx
new file mode 100644
index 000000000..1a375422a
--- /dev/null
+++ b/chapters/en/chapter2/9.mdx
@@ -0,0 +1,252 @@
+<FrameworkSwitchCourse {fw} />
+
+<!-- DISABLE-FRONTMATTER-SECTIONS -->
+
+# End-of-chapter quiz[[end-of-chapter-quiz]]
+
+<CourseFloatingBanner
+    chapter={2}
+    classNames="absolute z-10 right-0 top-0"
+/>
+
+### 1. What is the order of the language modeling pipeline?
+
+<Question
+	choices={[
+		{
+			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
+			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
+		},
+		{
+			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
+			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
+		},
+		{
+			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
+			explain: "The tokenizer can be used for both tokenizing and de-tokenizing.",
+            correct: true
+		}
+	]}
+/>
+
+### 2. How many dimensions does the tensor output by the base Transformer model have, and what are they?
+
+<Question
+	choices={[
+		{
+			text: "2: The sequence length and the batch size",
+			explain: "False! The tensor output by the model has a third dimension: hidden size."
+		},
+		{
+			text: "2: The sequence length and the hidden size",
+			explain: "False! All Transformer models handle batches, even with a single sequence; that would be a batch size of 1!"
+		},
+		{
+			text: "3: The sequence length, the batch size, and the hidden size",
+			explain: "Nicely done!",
+            correct: true
+		}
+	]}
+/>
+
+### 3. Which of the following is an example of subword tokenization?
+
+<Question
+	choices={[
+		{
+			text: "WordPiece",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+		},
+		{
+			text: "Character-based tokenization",
+			explain: "Character-based tokenization is not a type of subword tokenization."
+		},
+		{
+			text: "Splitting on whitespace and punctuation",
+			explain: "That's a word-based tokenization scheme!"
+		},
+		{
+			text: "BPE",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+        },
+		{
+			text: "Unigram",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+        },
+		{
+			text: "None of the above",
+			explain: "Wrong!"
+        }
+	]}
+/>
+
+### 4. What is a model head?
+
+<Question
+	choices={[
+		{
+			text: "A component of the base Transformer network that redirects tensors to their correct layers",
+			explain: "There's no such component."
+		},
+		{
+			text: "Also known as the self-attention mechanism, it adapts the representation of a token according to the other tokens of the sequence",
+			explain: "The self-attention layer does contain attention \"heads,\" but these are not adaptation heads."
+		},
+		{
+			text: "An additional component, usually made up of one or a few layers, to convert the transformer predictions to a task-specific output",
+			explain: "That's right. Adaptation heads, also known simply as heads, come up in different forms: language modeling heads, question answering heads, sequence classification heads... ",
+			correct: true
+		} 
+	]}
+/>
+
+### 5. What is an AutoModel?
+
+<Question
+	choices={[
+		{
+			text: "A model that automatically trains on your data",
+			explain: "Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
+		},
+		{
+			text: "An object that returns the correct architecture based on the checkpoint",
+			explain: "Exactly: the <code>AutoModel</code> only needs to know the checkpoint from which to initialize to return the correct architecture.",
+			correct: true
+		},
+		{
+			text: "A model that automatically detects the language used for its inputs to load the correct weights",
+			explain: "While some checkpoints and models are capable of handling multiple languages, there are no built-in tools for automatic checkpoint selection according to language. You should head over to the <a href='https://huggingface.co/models'>Model Hub</a> to find the best checkpoint for your task!"
+		} 
+	]}
+/>
+
+### 6. What are the techniques to be aware of when batching sequences of different lengths together?
+
+<Question
+	choices={[
+		{
+			text: "Truncating",
+			explain: "Yes, truncation is a correct way of evening out sequences so that they fit in a rectangular shape. Is it the only one, though?",
+			correct: true
+		},
+		{
+			text: "Returning tensors",
+			explain: "While the other techniques allow you to return rectangular tensors, returning tensors isn't helpful when batching sequences together."
+		},
+		{
+			text: "Padding",
+			explain: "Yes, padding is a correct way of evening out sequences so that they fit in a rectangular shape. Is it the only one, though?",
+			correct: true
+		}, 
+		{
+			text: "Attention masking",
+			explain: "Absolutely! Attention masks are of prime importance when handling sequences of different lengths. That's not the only technique to be aware of, however.",
+			correct: true
+		} 
+	]}
+/>
+
+### 7. What is the point of applying a SoftMax function to the logits output by a sequence classification model?
+
+<Question
+	choices={[
+		{
+			text: "It softens the logits so that they're more reliable.",
+			explain: "No, the SoftMax function does not affect the reliability of results."
+		},
+		{
+			text: "It applies a lower and upper bound so that they're understandable.",
+			explain: "The resulting values are bound between 0 and 1. That's not the only reason we use a SoftMax function, though.",
+            correct: true
+		},
+		{
+			text: "The total sum of the output is then 1, resulting in a possible probabilistic interpretation.",
+			explain: "Correct! That's not the only reason we use a SoftMax function, though.",
+            correct: true
+		}
+	]}
+/>
+
+### 8. What method is most of the tokenizer API centered around?
+
+<Question
+	choices={[
+		{
+			text: "<code>encode</code>, as it can encode text into IDs and IDs into predictions",
+			explain: "Wrong! While the <code>encode</code> method does exist on tokenizers, it does not exist on models."
+		},
+		{
+			text: "Calling the tokenizer object directly.",
+			explain: "Exactly! The <code>__call__</code> method of the tokenizer is a very powerful method which can handle pretty much anything. It is also the method used to retrieve predictions from a model.",
+			correct: true
+		},
+		{
+			text: "<code>pad</code>",
+			explain: "Wrong! Padding is very useful, but it's just one part of the tokenizer API."
+		},
+		{
+			text: "<code>tokenize</code>",
+			explain: "The <code>tokenize</code> method is arguably one of the most useful methods, but it isn't the core of the tokenizer API."
+		}
+	]}
+/>
+
+### 9. What does the `result` variable contain in this code sample?
+
+```py
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
+result = tokenizer.tokenize("Hello!")
+```
+
+<Question
+	choices={[
+		{
+			text: "A list of strings, each string being a token",
+			explain: "Absolutely! Convert this to IDs, and send them to a model!",
+            correct: true
+		},
+		{
+			text: "A list of IDs",
+			explain: "Incorrect; that's what the <code>__call__</code> or <code>convert_tokens_to_ids</code> method is for!"
+		},
+		{
+			text: "A string containing all of the tokens",
+			explain: "This would be suboptimal, as the goal is to split the string into multiple tokens."
+		}
+	]}
+/>
+
+### 10. Is there something wrong with the following code?
+
+```py
+from transformers import AutoTokenizer, AutoModel
+
+tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
+model = AutoModel.from_pretrained("gpt2")
+
+encoded = tokenizer("Hey!", return_tensors="pt")
+result = model(**encoded)
+```
+
+<Question
+	choices={[
+		{
+			text: "No, it seems correct.",
+			explain: "Unfortunately, coupling a model with a tokenizer that was trained with a different checkpoint is rarely a good idea. The model was not trained to make sense out of this tokenizer's output, so the model output (if it can even run!) will not make any sense."
+		},
+		{
+			text: "The tokenizer and model should always be from the same checkpoint.",
+			explain: "Right!",
+            correct: true
+		},
+		{
+			text: "It's good practice to pad and truncate with the tokenizer as every input is a batch.",
+			explain: "It's true that every model input needs to be a batch. However, truncating or padding this sequence wouldn't necessarily make sense as there is only one of it, and those are techniques to batch together a list of sentences."
+		}
+	]}
+/>


Reviews:
user: login: sergiopaniego
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-14T13:48:42Z

Review comments:
diff_hunk: @@ -1,310 +1,779 @@
-<FrameworkSwitchCourse {fw} />
-
-<!-- DISABLE-FRONTMATTER-SECTIONS -->
-
-# End-of-chapter quiz[[end-of-chapter-quiz]]
-
-<CourseFloatingBanner
-    chapter={2}
-    classNames="absolute z-10 right-0 top-0"
-/>
-
-### 1. What is the order of the language modeling pipeline?
-
-<Question
-	choices={[
-		{
-			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
-			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
-		},
-		{
-			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
-			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
-		},
-		{
-			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
-			explain: "Correct! The tokenizer can be used for both tokenizing and de-tokenizing.",
-            correct: true
-		}
-	]}
-/>
-
-### 2. How many dimensions does the tensor output by the base Transformer model have, and what are they?
-
-<Question
-	choices={[
-		{
-			text: "2: The sequence length and the batch size",
-			explain: "False! The tensor output by the model has a third dimension: hidden size."
-		},
-		{
-			text: "2: The sequence length and the hidden size",
-			explain: "False! All Transformer models handle batches, even with a single sequence; that would be a batch size of 1!"
-		},
-		{
-			text: "3: The sequence length, the batch size, and the hidden size",
-			explain: "Correct!",
-            correct: true
-		}
-	]}
-/>
-
-### 3. Which of the following is an example of subword tokenization?
-
-<Question
-	choices={[
-		{
-			text: "WordPiece",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-		},
-		{
-			text: "Character-based tokenization",
-			explain: "Character-based tokenization is not a type of subword tokenization."
-		},
-		{
-			text: "Splitting on whitespace and punctuation",
-			explain: "That's a word-based tokenization scheme!"
-		},
-		{
-			text: "BPE",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-        },
-		{
-			text: "Unigram",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-        },
-		{
-			text: "None of the above",
-			explain: "Incorrect!"
-        }
-	]}
-/>
-
-### 4. What is a model head?
-
-<Question
-	choices={[
-		{
-			text: "A component of the base Transformer network that redirects tensors to their correct layers",
-			explain: "Incorrect! There's no such component."
-		},
-		{
-			text: "Also known as the self-attention mechanism, it adapts the representation of a token according to the other tokens of the sequence",
-			explain: "Incorrect! The self-attention layer does contain attention \"heads,\" but these are not adaptation heads."
-		},
-		{
-			text: "An additional component, usually made up of one or a few layers, to convert the transformer predictions to a task-specific output",
-			explain: "That's right. Adaptation heads, also known simply as heads, come up in different forms: language modeling heads, question answering heads, sequence classification heads... ",
-			correct: true
-		} 
-	]}
-/>
-
-{#if fw === 'pt'}
-### 5. What is an AutoModel?
-
-<Question
-	choices={[
-		{
-			text: "A model that automatically trains on your data",
-			explain: "Incorrect. Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
-		},
-		{
-			text: "An object that returns the correct architecture based on the checkpoint",
-			explain: "Exactly: the <code>AutoModel</code> only needs to know the checkpoint from which to initialize to return the correct architecture.",
-			correct: true
-		},
-		{
-			text: "A model that automatically detects the language used for its inputs to load the correct weights",
-			explain: "Incorrect; while some checkpoints and models are capable of handling multiple languages, there are no built-in tools for automatic checkpoint selection according to language. You should head over to the <a href='https://huggingface.co/models'>Model Hub</a> to find the best checkpoint for your task!"
-		} 
-	]}
-/>
+# Optimized Inference Deployment
path: chapters/en/chapter2/8.mdx
user: login: sergiopaniego
body: There is an issue in the building process. I think it comes from the `framework` s tags in this file. I think at least is missing `<FrameworkSwitchCourse {fw} />`
created_at: 2025-05-14T12:48:08Z
author_association: MEMBER
diff_hunk: @@ -11,210 +9,281 @@
     {label: "Aws Studio", value: "https://studiolab.sagemaker.aws/import/github/huggingface/notebooks/blob/master/course/en/chapter2/section3_pt.ipynb"},
 ]} />
 
-{:else}
+<Youtube id="AhChOFRegn4"/>
 
-<CourseFloatingBanner chapter={2}
-  classNames="absolute z-10 right-0 top-0"
-  notebooks={[
-    {label: "Google Colab", value: "https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/en/chapter2/section3_tf.ipynb"},
-    {label: "Aws Studio", value: "https://studiolab.sagemaker.aws/import/github/huggingface/notebooks/blob/master/course/en/chapter2/section3_tf.ipynb"},
-]} />
+In this section, we'll take a closer look at creating and using models. We'll use the `AutoModel` class, which is handy when you want to instantiate any model from a checkpoint.
 
-{/if}
+## Creating a Transformer[[creating-a-transformer]]
 
-{#if fw === 'pt'}
-<Youtube id="AhChOFRegn4"/>
-{:else}
-<Youtube id="d3JVgghSOew"/>
-{/if}
+Let's begin by examining what happens when we instantiate an `AutoModel`:
 
-{#if fw === 'pt'}
-In this section we'll take a closer look at creating and using a model. We'll use the `AutoModel` class, which is handy when you want to instantiate any model from a checkpoint.
+```py
+from transformers import AutoModel
 
-The `AutoModel` class and all of its relatives are actually simple wrappers over the wide variety of models available in the library. It's a clever wrapper as it can automatically guess the appropriate model architecture for your checkpoint, and then instantiates a model with this architecture.
+model = AutoModel.from_pretrained("bert-base-cased")
+```
 
-{:else}
-In this section we'll take a closer look at creating and using a model. We'll use the `TFAutoModel` class, which is handy when you want to instantiate any model from a checkpoint.
+Similar to the tokenizer, the `from_pretrained()` method will download and cache the model data from the Hugging Face Hub. As mentioned previously, the checkpoint name corresponds to a specific model architecture and weights, in this case a BERT model with a basic architecture (12 layers, 768 hidden size, 12 attention heads) and cased inputs (meaning that the uppercase/lowercase distinction is important). There are many checkpoints available on the Hub — you can explore them [here](https://huggingface.co/models).
 
-The `TFAutoModel` class and all of its relatives are actually simple wrappers over the wide variety of models available in the library. It's a clever wrapper as it can automatically guess the appropriate model architecture for your checkpoint, and then instantiates a model with this architecture.
+The `AutoModel` class and its associates are actually simple wrappers designed to fetch the appropriate model architecture for a given checkpoint. It's an "auto" class meaning it will guess the appropriate model architecture for you and instantiate the correct model class. However, if you know the type of model you want to use, you can use the class that defines its architecture directly:
 
-{/if}
+```py
+from transformers import BertModel
 
-However, if you know the type of model you want to use, you can use the class that defines its architecture directly. Let's take a look at how this works with a BERT model.
+model = BertModel.from_pretrained("bert-base-cased")
+```
 
-## Creating a Transformer[[creating-a-transformer]]
+## Loading and saving[[loading-and-saving]]
 
-The first thing we'll need to do to initialize a BERT model is load a configuration object:
+Saving a model is as simple as saving a tokenizer. In fact, the models actually have the same `save_pretrained()` method, which saves the model's weights and architecture configuration:
 
-{#if fw === 'pt'}
 ```py
-from transformers import BertConfig, BertModel
+model.save_pretrained("directory_on_my_computer")
+```
 
-# Building the config
-config = BertConfig()
+This will save two files to your disk:
 
-# Building the model from the config
-model = BertModel(config)
 ```
-{:else}
+ls directory_on_my_computer
+
+config.json pytorch_model.bin
+```
+
+If you look inside the *config.json* file, you'll see all the necessary attributes needed to build the model architecture. This file also contains some metadata, such as where the checkpoint originated and what 🤗 Transformers version you were using when you last saved the checkpoint.
+
+The *pytorch_model.bin* file is known as the state dictionary; it contains all your model's weights. The two files work together: the configuration file is needed to know about the model architecture, while the model weights are the parameters of the model.
+
+To reuse a saved model, use the `from_pretrained()` method again:
+
 ```py
-from transformers import BertConfig, TFBertModel
+from transformers import AutoModel
+
+model = AutoModel.from_pretrained("directory_on_my_computer")
+```
+
+A wonderful feature of the 🤗 Transformers library is the ability to easily share models and tokenizers with the community. To do this, make sure you have an account on [Hugging Face](https://huggingface.co). If you're using a notebook, you can easily log in with this:
 
-# Building the config
-config = BertConfig()
+```python
+from huggingface_hub import notebook_login
 
-# Building the model from the config
-model = TFBertModel(config)
+notebook_login()
 ```
-{/if}
 
-The configuration contains many attributes that are used to build the model:
+Otherwise, at your terminal run:
+
+```bash
+huggingface-cli login
+```
+
+Then you can push the model to the Hub with the `push_to_hub()` method:
 
 ```py
-print(config)
+model.push_to_hub("my-awesome-model")
 ```
 
-```python out
-BertConfig {
-  [...]
-  "hidden_size": 768,
-  "intermediate_size": 3072,
-  "max_position_embeddings": 512,
-  "num_attention_heads": 12,
-  "num_hidden_layers": 12,
-  [...]
-}
+This will upload the model files to the Hub, in a repository under your namespace named *my-awesome-model*. Then, anyone can load your model with the `from_pretrained()` method!
+
+```py
+from transformers import AutoModel
+
+model = AutoModel.from_pretrained("your-username/my-awesome-model")
 ```
 
-While you haven't seen what all of these attributes do yet, you should recognize some of them: the `hidden_size` attribute defines the size of the `hidden_states` vector, and `num_hidden_layers` defines the number of layers the Transformer model has.
+You can do a lot more with the Hub API:
+- Push a model from a local repository
+- Update specific files without re-uploading everything
+- Add model cards to document the model's abilities, limitations, known biases, etc.
 
-### Different loading methods[[different-loading-methods]]
+See [the documentation](https://huggingface.co/docs/huggingface_hub/how-to-upstream) for a complete tutorial on this, or check out the advanced [Chapter 4](/course/chapter4/share).
path: chapters/en/chapter2/3.mdx
user: login: sergiopaniego
body: ```suggestion
See [the documentation](https://huggingface.co/docs/huggingface_hub/how-to-upstream) for a complete tutorial on this, or check out the advanced [Chapter 4](/course/chapter4).
```
created_at: 2025-05-14T13:08:47Z
author_association: MEMBER
diff_hunk: @@ -11,210 +9,281 @@
     {label: "Aws Studio", value: "https://studiolab.sagemaker.aws/import/github/huggingface/notebooks/blob/master/course/en/chapter2/section3_pt.ipynb"},
 ]} />
 
-{:else}
+<Youtube id="AhChOFRegn4"/>
 
-<CourseFloatingBanner chapter={2}
-  classNames="absolute z-10 right-0 top-0"
-  notebooks={[
-    {label: "Google Colab", value: "https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/en/chapter2/section3_tf.ipynb"},
-    {label: "Aws Studio", value: "https://studiolab.sagemaker.aws/import/github/huggingface/notebooks/blob/master/course/en/chapter2/section3_tf.ipynb"},
-]} />
+In this section, we'll take a closer look at creating and using models. We'll use the `AutoModel` class, which is handy when you want to instantiate any model from a checkpoint.
 
-{/if}
+## Creating a Transformer[[creating-a-transformer]]
 
-{#if fw === 'pt'}
-<Youtube id="AhChOFRegn4"/>
-{:else}
-<Youtube id="d3JVgghSOew"/>
-{/if}
+Let's begin by examining what happens when we instantiate an `AutoModel`:
 
-{#if fw === 'pt'}
-In this section we'll take a closer look at creating and using a model. We'll use the `AutoModel` class, which is handy when you want to instantiate any model from a checkpoint.
+```py
+from transformers import AutoModel
 
-The `AutoModel` class and all of its relatives are actually simple wrappers over the wide variety of models available in the library. It's a clever wrapper as it can automatically guess the appropriate model architecture for your checkpoint, and then instantiates a model with this architecture.
+model = AutoModel.from_pretrained("bert-base-cased")
+```
 
-{:else}
-In this section we'll take a closer look at creating and using a model. We'll use the `TFAutoModel` class, which is handy when you want to instantiate any model from a checkpoint.
+Similar to the tokenizer, the `from_pretrained()` method will download and cache the model data from the Hugging Face Hub. As mentioned previously, the checkpoint name corresponds to a specific model architecture and weights, in this case a BERT model with a basic architecture (12 layers, 768 hidden size, 12 attention heads) and cased inputs (meaning that the uppercase/lowercase distinction is important). There are many checkpoints available on the Hub — you can explore them [here](https://huggingface.co/models).
 
-The `TFAutoModel` class and all of its relatives are actually simple wrappers over the wide variety of models available in the library. It's a clever wrapper as it can automatically guess the appropriate model architecture for your checkpoint, and then instantiates a model with this architecture.
+The `AutoModel` class and its associates are actually simple wrappers designed to fetch the appropriate model architecture for a given checkpoint. It's an "auto" class meaning it will guess the appropriate model architecture for you and instantiate the correct model class. However, if you know the type of model you want to use, you can use the class that defines its architecture directly:
 
-{/if}
+```py
+from transformers import BertModel
 
-However, if you know the type of model you want to use, you can use the class that defines its architecture directly. Let's take a look at how this works with a BERT model.
+model = BertModel.from_pretrained("bert-base-cased")
+```
 
-## Creating a Transformer[[creating-a-transformer]]
+## Loading and saving[[loading-and-saving]]
 
-The first thing we'll need to do to initialize a BERT model is load a configuration object:
+Saving a model is as simple as saving a tokenizer. In fact, the models actually have the same `save_pretrained()` method, which saves the model's weights and architecture configuration:
 
-{#if fw === 'pt'}
 ```py
-from transformers import BertConfig, BertModel
+model.save_pretrained("directory_on_my_computer")
+```
 
-# Building the config
-config = BertConfig()
+This will save two files to your disk:
 
-# Building the model from the config
-model = BertModel(config)
 ```
-{:else}
+ls directory_on_my_computer
+
+config.json pytorch_model.bin
+```
+
+If you look inside the *config.json* file, you'll see all the necessary attributes needed to build the model architecture. This file also contains some metadata, such as where the checkpoint originated and what 🤗 Transformers version you were using when you last saved the checkpoint.
+
+The *pytorch_model.bin* file is known as the state dictionary; it contains all your model's weights. The two files work together: the configuration file is needed to know about the model architecture, while the model weights are the parameters of the model.
+
+To reuse a saved model, use the `from_pretrained()` method again:
+
 ```py
-from transformers import BertConfig, TFBertModel
+from transformers import AutoModel
+
+model = AutoModel.from_pretrained("directory_on_my_computer")
+```
+
+A wonderful feature of the 🤗 Transformers library is the ability to easily share models and tokenizers with the community. To do this, make sure you have an account on [Hugging Face](https://huggingface.co). If you're using a notebook, you can easily log in with this:
 
-# Building the config
-config = BertConfig()
+```python
+from huggingface_hub import notebook_login
 
-# Building the model from the config
-model = TFBertModel(config)
+notebook_login()
 ```
-{/if}
 
-The configuration contains many attributes that are used to build the model:
+Otherwise, at your terminal run:
+
+```bash
+huggingface-cli login
+```
+
+Then you can push the model to the Hub with the `push_to_hub()` method:
 
 ```py
-print(config)
+model.push_to_hub("my-awesome-model")
 ```
 
-```python out
-BertConfig {
-  [...]
-  "hidden_size": 768,
-  "intermediate_size": 3072,
-  "max_position_embeddings": 512,
-  "num_attention_heads": 12,
-  "num_hidden_layers": 12,
-  [...]
-}
+This will upload the model files to the Hub, in a repository under your namespace named *my-awesome-model*. Then, anyone can load your model with the `from_pretrained()` method!
+
+```py
+from transformers import AutoModel
+
+model = AutoModel.from_pretrained("your-username/my-awesome-model")
 ```
 
-While you haven't seen what all of these attributes do yet, you should recognize some of them: the `hidden_size` attribute defines the size of the `hidden_states` vector, and `num_hidden_layers` defines the number of layers the Transformer model has.
+You can do a lot more with the Hub API:
+- Push a model from a local repository
+- Update specific files without re-uploading everything
+- Add model cards to document the model's abilities, limitations, known biases, etc.
 
-### Different loading methods[[different-loading-methods]]
+See [the documentation](https://huggingface.co/docs/huggingface_hub/how-to-upstream) for a complete tutorial on this, or check out the advanced [Chapter 4](/course/chapter4/share).
 
-Creating a model from the default configuration initializes it with random values:
+## Encoding text[[encoding-text]]
+
+Transformer models handle text by turning the inputs into numbers. Here we will look at exactly what happens when your text is processed by the tokenizer. We've already seen in [Chapter 1](/course/chapter1) that tokenizers split the text into tokens and then convert these tokens into numbers. We can see this conversion through a simple tokenizer:
 
-{#if fw === 'pt'}
 ```py
-from transformers import BertConfig, BertModel
+from transformers import AutoTokenizer
 
-config = BertConfig()
-model = BertModel(config)
+tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
 
-# Model is randomly initialized!
+encoded_input = tokenizer("Hello, I'm a single sentence!")
+print(encoded_input)
 ```
-{:else}
-```py
-from transformers import BertConfig, TFBertModel
 
-config = BertConfig()
-model = TFBertModel(config)
+```python out
+{'input_ids': [101, 8667, 117, 1000, 1045, 1005, 1049, 2235, 17662, 12172, 1012, 102], 
+ 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 
+ 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
+```
+
+We get a dictionary with the following fields:
+- input_ids: numerical representations of your tokens
+- token_type_ids: these tell the model which part of the input is sentence A and which is sentence B (discussed more in the next section)
+- attention_mask: this indicates which tokens should be attended to and which should not (discussed more in a bit)
+
+We can decode the input IDs to get back the original text:
+
+```py
+tokenizer.decode(encoded_input["input_ids"])
+```
 
-# Model is randomly initialized!
+```python out
+"[CLS] Hello, I'm a single sentence! [SEP]"
 ```
-{/if}
 
-The model can be used in this state, but it will output gibberish; it needs to be trained first. We could train the model from scratch on the task at hand, but as you saw in [Chapter 1](/course/chapter1), this would require a long time and a lot of data, and it would have a non-negligible environmental impact. To avoid unnecessary and duplicated effort, it's imperative to be able to share and reuse models that have already been trained.
+You'll notice that the tokenizer has added special tokens — `[CLS]` and `[SEP]` — required by the model. Not all models need special tokens; they're utilized when a model was pretrained with them, in which case the tokenizer needs to add them as that model expects these tokens.
path: chapters/en/chapter2/3.mdx
user: login: sergiopaniego
body: This sentence feels convoluted
created_at: 2025-05-14T13:18:16Z
author_association: MEMBER
diff_hunk: @@ -1,310 +1,779 @@
-<FrameworkSwitchCourse {fw} />
-
-<!-- DISABLE-FRONTMATTER-SECTIONS -->
-
-# End-of-chapter quiz[[end-of-chapter-quiz]]
-
-<CourseFloatingBanner
-    chapter={2}
-    classNames="absolute z-10 right-0 top-0"
-/>
-
-### 1. What is the order of the language modeling pipeline?
-
-<Question
-	choices={[
-		{
-			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
-			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
-		},
-		{
-			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
-			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
-		},
-		{
-			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
-			explain: "Correct! The tokenizer can be used for both tokenizing and de-tokenizing.",
-            correct: true
-		}
-	]}
-/>
-
-### 2. How many dimensions does the tensor output by the base Transformer model have, and what are they?
-
-<Question
-	choices={[
-		{
-			text: "2: The sequence length and the batch size",
-			explain: "False! The tensor output by the model has a third dimension: hidden size."
-		},
-		{
-			text: "2: The sequence length and the hidden size",
-			explain: "False! All Transformer models handle batches, even with a single sequence; that would be a batch size of 1!"
-		},
-		{
-			text: "3: The sequence length, the batch size, and the hidden size",
-			explain: "Correct!",
-            correct: true
-		}
-	]}
-/>
-
-### 3. Which of the following is an example of subword tokenization?
-
-<Question
-	choices={[
-		{
-			text: "WordPiece",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-		},
-		{
-			text: "Character-based tokenization",
-			explain: "Character-based tokenization is not a type of subword tokenization."
-		},
-		{
-			text: "Splitting on whitespace and punctuation",
-			explain: "That's a word-based tokenization scheme!"
-		},
-		{
-			text: "BPE",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-        },
-		{
-			text: "Unigram",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-        },
-		{
-			text: "None of the above",
-			explain: "Incorrect!"
-        }
-	]}
-/>
-
-### 4. What is a model head?
-
-<Question
-	choices={[
-		{
-			text: "A component of the base Transformer network that redirects tensors to their correct layers",
-			explain: "Incorrect! There's no such component."
-		},
-		{
-			text: "Also known as the self-attention mechanism, it adapts the representation of a token according to the other tokens of the sequence",
-			explain: "Incorrect! The self-attention layer does contain attention \"heads,\" but these are not adaptation heads."
-		},
-		{
-			text: "An additional component, usually made up of one or a few layers, to convert the transformer predictions to a task-specific output",
-			explain: "That's right. Adaptation heads, also known simply as heads, come up in different forms: language modeling heads, question answering heads, sequence classification heads... ",
-			correct: true
-		} 
-	]}
-/>
-
-{#if fw === 'pt'}
-### 5. What is an AutoModel?
-
-<Question
-	choices={[
-		{
-			text: "A model that automatically trains on your data",
-			explain: "Incorrect. Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
-		},
-		{
-			text: "An object that returns the correct architecture based on the checkpoint",
-			explain: "Exactly: the <code>AutoModel</code> only needs to know the checkpoint from which to initialize to return the correct architecture.",
-			correct: true
-		},
-		{
-			text: "A model that automatically detects the language used for its inputs to load the correct weights",
-			explain: "Incorrect; while some checkpoints and models are capable of handling multiple languages, there are no built-in tools for automatic checkpoint selection according to language. You should head over to the <a href='https://huggingface.co/models'>Model Hub</a> to find the best checkpoint for your task!"
-		} 
-	]}
-/>
+# Optimized Inference Deployment
 
+In this section, we'll explore advanced frameworks for optimizing LLM deployments: Text Generation Inference (TGI), vLLM, and llama.cpp. These applications are primarily used in production environments to serve LLMs to users. This section focuses on how to deploy these frameworks in production rather than how to use them for inference on a single machine.
+
+We'll cover how these tools maximize inference efficiency and simplify production deployments of Large Language Models.
+
+## Framework Selection Guide
+
+TGI, vLLM, and llama.cpp serve similar purposes but have distinct characteristics that make them better suited for different use cases. Let's look at the key differences between them, focusing on performance and integration.
+
+### Memory Management and Performance
+
+**TGI** is designed to be stable and predictable in production, using fixed sequence lengths to keep memory usage consistent. TGI manages memory using Flash Attention 2 and continuous batching techniques. This means it can process attention calculations very efficiently and keep the GPU busy by constantly feeding it work. The system can move parts of the model between CPU and GPU when needed, which helps handle larger models. 
+
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/tgi/flash-attn.png" alt="Flash Attention" />
+
+<Tip title="How Flash Attention Works">
+Flash Attention is a technique that optimizes the attention mechanism in transformer models by addressing memory bandwidth bottlenecks. As discussed earlier in [section 12.3](2.mdx), the attention mechanism has quadratic complexity and memory usage, making it inefficient for long sequences.
path: chapters/en/chapter2/8.mdx
user: login: sergiopaniego
body: ```suggestion
Flash Attention is a technique that optimizes the attention mechanism in transformer models by addressing memory bandwidth bottlenecks. As discussed earlier in [Chapter 1.8](/course/chapter1/8), the attention mechanism has quadratic complexity and memory usage, making it inefficient for long sequences.
```
created_at: 2025-05-14T13:28:40Z
author_association: MEMBER
diff_hunk: @@ -1,310 +1,779 @@
-<FrameworkSwitchCourse {fw} />
-
-<!-- DISABLE-FRONTMATTER-SECTIONS -->
-
-# End-of-chapter quiz[[end-of-chapter-quiz]]
-
-<CourseFloatingBanner
-    chapter={2}
-    classNames="absolute z-10 right-0 top-0"
-/>
-
-### 1. What is the order of the language modeling pipeline?
-
-<Question
-	choices={[
-		{
-			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
-			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
-		},
-		{
-			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
-			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
-		},
-		{
-			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
-			explain: "Correct! The tokenizer can be used for both tokenizing and de-tokenizing.",
-            correct: true
-		}
-	]}
-/>
-
-### 2. How many dimensions does the tensor output by the base Transformer model have, and what are they?
-
-<Question
-	choices={[
-		{
-			text: "2: The sequence length and the batch size",
-			explain: "False! The tensor output by the model has a third dimension: hidden size."
-		},
-		{
-			text: "2: The sequence length and the hidden size",
-			explain: "False! All Transformer models handle batches, even with a single sequence; that would be a batch size of 1!"
-		},
-		{
-			text: "3: The sequence length, the batch size, and the hidden size",
-			explain: "Correct!",
-            correct: true
-		}
-	]}
-/>
-
-### 3. Which of the following is an example of subword tokenization?
-
-<Question
-	choices={[
-		{
-			text: "WordPiece",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-		},
-		{
-			text: "Character-based tokenization",
-			explain: "Character-based tokenization is not a type of subword tokenization."
-		},
-		{
-			text: "Splitting on whitespace and punctuation",
-			explain: "That's a word-based tokenization scheme!"
-		},
-		{
-			text: "BPE",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-        },
-		{
-			text: "Unigram",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-        },
-		{
-			text: "None of the above",
-			explain: "Incorrect!"
-        }
-	]}
-/>
-
-### 4. What is a model head?
-
-<Question
-	choices={[
-		{
-			text: "A component of the base Transformer network that redirects tensors to their correct layers",
-			explain: "Incorrect! There's no such component."
-		},
-		{
-			text: "Also known as the self-attention mechanism, it adapts the representation of a token according to the other tokens of the sequence",
-			explain: "Incorrect! The self-attention layer does contain attention \"heads,\" but these are not adaptation heads."
-		},
-		{
-			text: "An additional component, usually made up of one or a few layers, to convert the transformer predictions to a task-specific output",
-			explain: "That's right. Adaptation heads, also known simply as heads, come up in different forms: language modeling heads, question answering heads, sequence classification heads... ",
-			correct: true
-		} 
-	]}
-/>
-
-{#if fw === 'pt'}
-### 5. What is an AutoModel?
-
-<Question
-	choices={[
-		{
-			text: "A model that automatically trains on your data",
-			explain: "Incorrect. Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
-		},
-		{
-			text: "An object that returns the correct architecture based on the checkpoint",
-			explain: "Exactly: the <code>AutoModel</code> only needs to know the checkpoint from which to initialize to return the correct architecture.",
-			correct: true
-		},
-		{
-			text: "A model that automatically detects the language used for its inputs to load the correct weights",
-			explain: "Incorrect; while some checkpoints and models are capable of handling multiple languages, there are no built-in tools for automatic checkpoint selection according to language. You should head over to the <a href='https://huggingface.co/models'>Model Hub</a> to find the best checkpoint for your task!"
-		} 
-	]}
-/>
+# Optimized Inference Deployment
 
+In this section, we'll explore advanced frameworks for optimizing LLM deployments: Text Generation Inference (TGI), vLLM, and llama.cpp. These applications are primarily used in production environments to serve LLMs to users. This section focuses on how to deploy these frameworks in production rather than how to use them for inference on a single machine.
+
+We'll cover how these tools maximize inference efficiency and simplify production deployments of Large Language Models.
+
+## Framework Selection Guide
+
+TGI, vLLM, and llama.cpp serve similar purposes but have distinct characteristics that make them better suited for different use cases. Let's look at the key differences between them, focusing on performance and integration.
+
+### Memory Management and Performance
+
+**TGI** is designed to be stable and predictable in production, using fixed sequence lengths to keep memory usage consistent. TGI manages memory using Flash Attention 2 and continuous batching techniques. This means it can process attention calculations very efficiently and keep the GPU busy by constantly feeding it work. The system can move parts of the model between CPU and GPU when needed, which helps handle larger models. 
+
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/tgi/flash-attn.png" alt="Flash Attention" />
+
+<Tip title="How Flash Attention Works">
+Flash Attention is a technique that optimizes the attention mechanism in transformer models by addressing memory bandwidth bottlenecks. As discussed earlier in [section 12.3](2.mdx), the attention mechanism has quadratic complexity and memory usage, making it inefficient for long sequences.
+
+The key innovation is in how it manages memory transfers between High Bandwidth Memory (HBM) and faster SRAM cache. Traditional attention repeatedly transfers data between HBM and SRAM, creating bottlenecks by leaving the GPU idle. Flash Attention loads data once into SRAM and performs all calculations there, minimizing expensive memory transfers. 
+
+While the benefits are most significant during training, Flash Attention's reduced VRAM usage and improved efficiency make it valuable for inference as well, enabling faster and more scalable LLM serving.
+</Tip>
+
+**vLLM** takes a different approach by using PagedAttention. Just like how a computer manages its memory in pages, vLLM splits the model's memory into smaller blocks. This clever system means it can handle different-sized requests more flexibly and doesn't waste memory space. It's particularly good at sharing memory between different requests and reduces memory fragmentation, which makes the whole system more efficient.
+
+<Tip title="How Paged Attention Works">
path: chapters/en/chapter2/8.mdx
user: login: sergiopaniego
body: Maintaining consistency in naming:
```suggestion
<Tip title="How PagedAttention Works">
```
created_at: 2025-05-14T13:31:21Z
author_association: MEMBER
diff_hunk: @@ -1,310 +1,779 @@
-<FrameworkSwitchCourse {fw} />
-
-<!-- DISABLE-FRONTMATTER-SECTIONS -->
-
-# End-of-chapter quiz[[end-of-chapter-quiz]]
-
-<CourseFloatingBanner
-    chapter={2}
-    classNames="absolute z-10 right-0 top-0"
-/>
-
-### 1. What is the order of the language modeling pipeline?
-
-<Question
-	choices={[
-		{
-			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
-			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
-		},
-		{
-			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
-			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
-		},
-		{
-			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
-			explain: "Correct! The tokenizer can be used for both tokenizing and de-tokenizing.",
-            correct: true
-		}
-	]}
-/>
-
-### 2. How many dimensions does the tensor output by the base Transformer model have, and what are they?
-
-<Question
-	choices={[
-		{
-			text: "2: The sequence length and the batch size",
-			explain: "False! The tensor output by the model has a third dimension: hidden size."
-		},
-		{
-			text: "2: The sequence length and the hidden size",
-			explain: "False! All Transformer models handle batches, even with a single sequence; that would be a batch size of 1!"
-		},
-		{
-			text: "3: The sequence length, the batch size, and the hidden size",
-			explain: "Correct!",
-            correct: true
-		}
-	]}
-/>
-
-### 3. Which of the following is an example of subword tokenization?
-
-<Question
-	choices={[
-		{
-			text: "WordPiece",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-		},
-		{
-			text: "Character-based tokenization",
-			explain: "Character-based tokenization is not a type of subword tokenization."
-		},
-		{
-			text: "Splitting on whitespace and punctuation",
-			explain: "That's a word-based tokenization scheme!"
-		},
-		{
-			text: "BPE",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-        },
-		{
-			text: "Unigram",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-        },
-		{
-			text: "None of the above",
-			explain: "Incorrect!"
-        }
-	]}
-/>
-
-### 4. What is a model head?
-
-<Question
-	choices={[
-		{
-			text: "A component of the base Transformer network that redirects tensors to their correct layers",
-			explain: "Incorrect! There's no such component."
-		},
-		{
-			text: "Also known as the self-attention mechanism, it adapts the representation of a token according to the other tokens of the sequence",
-			explain: "Incorrect! The self-attention layer does contain attention \"heads,\" but these are not adaptation heads."
-		},
-		{
-			text: "An additional component, usually made up of one or a few layers, to convert the transformer predictions to a task-specific output",
-			explain: "That's right. Adaptation heads, also known simply as heads, come up in different forms: language modeling heads, question answering heads, sequence classification heads... ",
-			correct: true
-		} 
-	]}
-/>
-
-{#if fw === 'pt'}
-### 5. What is an AutoModel?
-
-<Question
-	choices={[
-		{
-			text: "A model that automatically trains on your data",
-			explain: "Incorrect. Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
-		},
-		{
-			text: "An object that returns the correct architecture based on the checkpoint",
-			explain: "Exactly: the <code>AutoModel</code> only needs to know the checkpoint from which to initialize to return the correct architecture.",
-			correct: true
-		},
-		{
-			text: "A model that automatically detects the language used for its inputs to load the correct weights",
-			explain: "Incorrect; while some checkpoints and models are capable of handling multiple languages, there are no built-in tools for automatic checkpoint selection according to language. You should head over to the <a href='https://huggingface.co/models'>Model Hub</a> to find the best checkpoint for your task!"
-		} 
-	]}
-/>
+# Optimized Inference Deployment
 
+In this section, we'll explore advanced frameworks for optimizing LLM deployments: Text Generation Inference (TGI), vLLM, and llama.cpp. These applications are primarily used in production environments to serve LLMs to users. This section focuses on how to deploy these frameworks in production rather than how to use them for inference on a single machine.
+
+We'll cover how these tools maximize inference efficiency and simplify production deployments of Large Language Models.
+
+## Framework Selection Guide
+
+TGI, vLLM, and llama.cpp serve similar purposes but have distinct characteristics that make them better suited for different use cases. Let's look at the key differences between them, focusing on performance and integration.
+
+### Memory Management and Performance
+
+**TGI** is designed to be stable and predictable in production, using fixed sequence lengths to keep memory usage consistent. TGI manages memory using Flash Attention 2 and continuous batching techniques. This means it can process attention calculations very efficiently and keep the GPU busy by constantly feeding it work. The system can move parts of the model between CPU and GPU when needed, which helps handle larger models. 
+
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/tgi/flash-attn.png" alt="Flash Attention" />
+
+<Tip title="How Flash Attention Works">
+Flash Attention is a technique that optimizes the attention mechanism in transformer models by addressing memory bandwidth bottlenecks. As discussed earlier in [section 12.3](2.mdx), the attention mechanism has quadratic complexity and memory usage, making it inefficient for long sequences.
+
+The key innovation is in how it manages memory transfers between High Bandwidth Memory (HBM) and faster SRAM cache. Traditional attention repeatedly transfers data between HBM and SRAM, creating bottlenecks by leaving the GPU idle. Flash Attention loads data once into SRAM and performs all calculations there, minimizing expensive memory transfers. 
+
+While the benefits are most significant during training, Flash Attention's reduced VRAM usage and improved efficiency make it valuable for inference as well, enabling faster and more scalable LLM serving.
+</Tip>
+
+**vLLM** takes a different approach by using PagedAttention. Just like how a computer manages its memory in pages, vLLM splits the model's memory into smaller blocks. This clever system means it can handle different-sized requests more flexibly and doesn't waste memory space. It's particularly good at sharing memory between different requests and reduces memory fragmentation, which makes the whole system more efficient.
+
+<Tip title="How Paged Attention Works">
+Paged Attention is a technique that addresses another critical bottleneck in LLM inference: KV cache memory management. As discussed in [section 12.3](2.mdx), during text generation, the model stores attention keys and values (KV cache) for each generated token to reduce redundant computations. The KV cache can become enormous, especially with long sequences or multiple concurrent requests.
path: chapters/en/chapter2/8.mdx
user: login: sergiopaniego
body: ```suggestion
PagedAttention is a technique that addresses another critical bottleneck in LLM inference: KV cache memory management. As discussed in [Chapter 1.8](/course/chapter1/8), during text generation, the model stores attention keys and values (KV cache) for each generated token to reduce redundant computations. The KV cache can become enormous, especially with long sequences or multiple concurrent requests.
```
created_at: 2025-05-14T13:31:30Z
author_association: MEMBER
diff_hunk: @@ -1,310 +1,779 @@
-<FrameworkSwitchCourse {fw} />
-
-<!-- DISABLE-FRONTMATTER-SECTIONS -->
-
-# End-of-chapter quiz[[end-of-chapter-quiz]]
-
-<CourseFloatingBanner
-    chapter={2}
-    classNames="absolute z-10 right-0 top-0"
-/>
-
-### 1. What is the order of the language modeling pipeline?
-
-<Question
-	choices={[
-		{
-			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
-			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
-		},
-		{
-			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
-			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
-		},
-		{
-			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
-			explain: "Correct! The tokenizer can be used for both tokenizing and de-tokenizing.",
-            correct: true
-		}
-	]}
-/>
-
-### 2. How many dimensions does the tensor output by the base Transformer model have, and what are they?
-
-<Question
-	choices={[
-		{
-			text: "2: The sequence length and the batch size",
-			explain: "False! The tensor output by the model has a third dimension: hidden size."
-		},
-		{
-			text: "2: The sequence length and the hidden size",
-			explain: "False! All Transformer models handle batches, even with a single sequence; that would be a batch size of 1!"
-		},
-		{
-			text: "3: The sequence length, the batch size, and the hidden size",
-			explain: "Correct!",
-            correct: true
-		}
-	]}
-/>
-
-### 3. Which of the following is an example of subword tokenization?
-
-<Question
-	choices={[
-		{
-			text: "WordPiece",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-		},
-		{
-			text: "Character-based tokenization",
-			explain: "Character-based tokenization is not a type of subword tokenization."
-		},
-		{
-			text: "Splitting on whitespace and punctuation",
-			explain: "That's a word-based tokenization scheme!"
-		},
-		{
-			text: "BPE",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-        },
-		{
-			text: "Unigram",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-        },
-		{
-			text: "None of the above",
-			explain: "Incorrect!"
-        }
-	]}
-/>
-
-### 4. What is a model head?
-
-<Question
-	choices={[
-		{
-			text: "A component of the base Transformer network that redirects tensors to their correct layers",
-			explain: "Incorrect! There's no such component."
-		},
-		{
-			text: "Also known as the self-attention mechanism, it adapts the representation of a token according to the other tokens of the sequence",
-			explain: "Incorrect! The self-attention layer does contain attention \"heads,\" but these are not adaptation heads."
-		},
-		{
-			text: "An additional component, usually made up of one or a few layers, to convert the transformer predictions to a task-specific output",
-			explain: "That's right. Adaptation heads, also known simply as heads, come up in different forms: language modeling heads, question answering heads, sequence classification heads... ",
-			correct: true
-		} 
-	]}
-/>
-
-{#if fw === 'pt'}
-### 5. What is an AutoModel?
-
-<Question
-	choices={[
-		{
-			text: "A model that automatically trains on your data",
-			explain: "Incorrect. Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
-		},
-		{
-			text: "An object that returns the correct architecture based on the checkpoint",
-			explain: "Exactly: the <code>AutoModel</code> only needs to know the checkpoint from which to initialize to return the correct architecture.",
-			correct: true
-		},
-		{
-			text: "A model that automatically detects the language used for its inputs to load the correct weights",
-			explain: "Incorrect; while some checkpoints and models are capable of handling multiple languages, there are no built-in tools for automatic checkpoint selection according to language. You should head over to the <a href='https://huggingface.co/models'>Model Hub</a> to find the best checkpoint for your task!"
-		} 
-	]}
-/>
+# Optimized Inference Deployment
 
+In this section, we'll explore advanced frameworks for optimizing LLM deployments: Text Generation Inference (TGI), vLLM, and llama.cpp. These applications are primarily used in production environments to serve LLMs to users. This section focuses on how to deploy these frameworks in production rather than how to use them for inference on a single machine.
+
+We'll cover how these tools maximize inference efficiency and simplify production deployments of Large Language Models.
+
+## Framework Selection Guide
+
+TGI, vLLM, and llama.cpp serve similar purposes but have distinct characteristics that make them better suited for different use cases. Let's look at the key differences between them, focusing on performance and integration.
+
+### Memory Management and Performance
+
+**TGI** is designed to be stable and predictable in production, using fixed sequence lengths to keep memory usage consistent. TGI manages memory using Flash Attention 2 and continuous batching techniques. This means it can process attention calculations very efficiently and keep the GPU busy by constantly feeding it work. The system can move parts of the model between CPU and GPU when needed, which helps handle larger models. 
+
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/tgi/flash-attn.png" alt="Flash Attention" />
+
+<Tip title="How Flash Attention Works">
+Flash Attention is a technique that optimizes the attention mechanism in transformer models by addressing memory bandwidth bottlenecks. As discussed earlier in [section 12.3](2.mdx), the attention mechanism has quadratic complexity and memory usage, making it inefficient for long sequences.
+
+The key innovation is in how it manages memory transfers between High Bandwidth Memory (HBM) and faster SRAM cache. Traditional attention repeatedly transfers data between HBM and SRAM, creating bottlenecks by leaving the GPU idle. Flash Attention loads data once into SRAM and performs all calculations there, minimizing expensive memory transfers. 
+
+While the benefits are most significant during training, Flash Attention's reduced VRAM usage and improved efficiency make it valuable for inference as well, enabling faster and more scalable LLM serving.
+</Tip>
+
+**vLLM** takes a different approach by using PagedAttention. Just like how a computer manages its memory in pages, vLLM splits the model's memory into smaller blocks. This clever system means it can handle different-sized requests more flexibly and doesn't waste memory space. It's particularly good at sharing memory between different requests and reduces memory fragmentation, which makes the whole system more efficient.
+
+<Tip title="How Paged Attention Works">
+Paged Attention is a technique that addresses another critical bottleneck in LLM inference: KV cache memory management. As discussed in [section 12.3](2.mdx), during text generation, the model stores attention keys and values (KV cache) for each generated token to reduce redundant computations. The KV cache can become enormous, especially with long sequences or multiple concurrent requests.
+
+vLLM's key innovation lies in how it manages this cache:
+
+1. **Memory Paging**: Instead of treating the KV cache as one large block, it's divided into fixed-size "pages" (similar to virtual memory in operating systems).
+2. **Non-contiguous Storage**: Pages don't need to be stored contiguously in GPU memory, allowing for more flexible memory allocation.
+3. **Page Table Management**: A page table tracks which pages belong to which sequence, enabling efficient lookup and access.
+4. **Memory Sharing**: For operations like parallel sampling, pages storing the KV cache for the prompt can be shared across multiple sequences.
+
+The PagedAttention approach can lead to up to 24x higher throughput compared to traditional methods, making it a game-changer for production LLM deployments. If you want to go really deep into how PagedAttention works, you can read the [the guide from the vLLM documentation](https://docs.vllm.ai/en/latest/design/kernel/paged_attention.html).
+</Tip>
+
+**llama.cpp** is a highly optimized C/C++ implementation originally designed for running LLaMA models on consumer hardware. It focuses on CPU efficiency with optional GPU acceleration and is ideal for resource-constrained environments. llama.cpp uses quantization techniques to reduce model size and memory requirements while maintaining good performance. It implements optimized kernels for various CPU architectures and supports basic KV cache management for efficient token generation.
+
+<Tip title="How llama.cpp Quantization Works">
+Quantization in llama.cpp reduces the precision of model weights from 32-bit or 16-bit floating point to lower precision formats like 8-bit integers (INT8), 4-bit, or even lower. This significantly reduces memory usage and improves inference speed with minimal quality loss.
+
+Key quantization features in llama.cpp include:
+1. **Multiple Quantization Levels**: Supports 8-bit, 4-bit, 3-bit, and even 2-bit quantization
+2. **GGML/GGUF Format**: Uses custom tensor formats optimized for quantized inference
+3. **Mixed Precision**: Can apply different quantization levels to different parts of the model
+4. **Hardware-Specific Optimizations**: Includes optimized code paths for various CPU architectures (AVX2, AVX-512, NEON)
+
+This approach enables running billion-parameter models on consumer hardware with limited memory, making it perfect for local deployments and edge devices.
+</Tip>
+
+
+
+### Deployment and Integration
+
+Let's move on to the deployment and integration differences between the frameworks.
+
+**TGI** excels in enterprise-level deployment with its production-ready features. It comes with built-in Kubernetes support and includes everything you need for running in production, like monitoring through Prometheus and Grafana, automatic scaling, and comprehensive safety features. The system also includes enterprise-grade logging and various protective measures like content filtering and rate limiting to keep your deployment secure and stable.
+
+**vLLM** takes a more flexible, developer-friendly approach to deployment. It's built with Python at its core and can easily replace OpenAI's API in your existing applications. The framework focuses on delivering raw performance and can be customized to fit your specific needs. It works particularly well with Ray for managing clusters, making it a great choice when you need high performance and adaptability.
+
+**llama.cpp** prioritizes simplicity and portability. Its server implementation is lightweight and can run on a wide range of hardware, from powerful servers to consumer laptops and even some high-end mobile devices. With minimal dependencies and a simple C/C++ core, it's easy to deploy in environments where installing Python frameworks would be challenging. The server provides an OpenAI-compatible API while maintaining a much smaller resource footprint than other solutions.
+
+
+
+## Getting Started
+
+Let's explore how to use these frameworks for deploying LLMs, starting with installation and basic setup.
+
+### Installation and Basic Setup
+
+{#if framework === 'tgi'}
+
+TGI is easy to install and use, with deep integration into the Hugging Face ecosystem.
+
+First, launch the TGI server using Docker:
+
+```sh
+docker run --gpus all \
+    --shm-size 1g \
+    -p 8080:80 \
+    -v ~/.cache/huggingface:/data \
+    ghcr.io/huggingface/text-generation-inference:latest \
+    --model-id HuggingFaceTB/SmolLM2-360M-Instruct
+```
+
+Then interact with it using Hugging Face's InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+# Initialize client pointing to TGI endpoint
+client = InferenceClient(
+    model="http://localhost:8080",  # URL to the TGI server
+)
+
+# Text generation
+response = client.text_generation(
+    "Tell me a story",
+    max_new_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+    details=True,
+    stop_sequences=[],
+)
+print(response.generated_text)
+
+# For chat format
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
+
+Alternatively, you can use the OpenAI client:
+
+```python
+from openai import OpenAI
+
+# Initialize client pointing to TGI endpoint
+client = OpenAI(
+    base_url="http://localhost:8080/v1",  # Make sure to include /v1
+    api_key="not-needed",  # TGI doesn't require an API key by default
+)
+
+# Chat completion
+response = client.chat.completions.create(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
+{:else if framework === 'llama.cpp'}
+
+llama.cpp is easy to install and use, requiring minimal dependencies and supporting both CPU and GPU inference.
+
+First, install and build llama.cpp:
+
+```sh
+# Clone the repository
+git clone https://github.com/ggerganov/llama.cpp
+cd llama.cpp
+
+# Build the project
+make
+
+# Download the SmolLM2-1.7B-Instruct-GGUF model
+curl -L -O https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct-GGUF/resolve/main/smollm2-1.7b-instruct.Q4_K_M.gguf
+```
+
+Then, launch the server (with OpenAI API compatibility):
+
+```sh
+# Start the server
+./server \
+    -m smollm2-1.7b-instruct.Q4_K_M.gguf \
+    --host 0.0.0.0 \
+    --port 8080 \
+    -c 4096 \
+    --n-gpu-layers 0  # Set to a higher number to use GPU
+```
+
+Interact with the server using Hugging Face's InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+# Initialize client pointing to llama.cpp server
+client = InferenceClient(
+    model="http://localhost:8080/v1",  # URL to the llama.cpp server
+    token="sk-no-key-required",  # llama.cpp server requires this placeholder
+)
+
+# Text generation
+response = client.text_generation(
+    "Tell me a story",
+    max_new_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+    details=True,
+)
+print(response.generated_text)
+
+# For chat format
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
+
+Alternatively, you can use the OpenAI client:
+
+```python
+from openai import OpenAI
+
+# Initialize client pointing to llama.cpp server
+client = OpenAI(
+    base_url="http://localhost:8080/v1",
+    api_key="sk-no-key-required",  # llama.cpp server requires this placeholder
+)
+
+# Chat completion
+response = client.chat.completions.create(
+    model="smollm2-1.7b-instruct",  # Model identifier can be anything as server only loads one model
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
 {:else}
-### 5. What is an TFAutoModel?
-
-<Question
-	choices={[
-		{
-			text: "A model that automatically trains on your data",
-			explain: "Incorrect. Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
-		},
-		{
-			text: "An object that returns the correct architecture based on the checkpoint",
-			explain: "Exactly: the <code>TFAutoModel</code> only needs to know the checkpoint from which to initialize to return the correct architecture.",
-			correct: true
-		},
-		{
-			text: "A model that automatically detects the language used for its inputs to load the correct weights",
-			explain: "Incorrect; while some checkpoints and models are capable of handling multiple languages, there are no built-in tools for automatic checkpoint selection according to language. You should head over to the <a href='https://huggingface.co/models'>Model Hub</a> to find the best checkpoint for your task!"
-		} 
-	]}
-/>
 
+vLLM is easy to install and use, with both OpenAI API compatibility and a native Python interface.
+
+First, launch the vLLM OpenAI-compatible server:
+
+```sh
+python -m vllm.entrypoints.openai.api_server \
+    --model HuggingFaceTB/SmolLM2-360M-Instruct \
+    --host 0.0.0.0 \
+    --port 8000
+```
+
+Then interact with it using Hugging Face's InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+# Initialize client pointing to vLLM endpoint
+client = InferenceClient(
+    model="http://localhost:8000/v1",  # URL to the vLLM server
+)
+
+# Text generation
+response = client.text_generation(
+    "Tell me a story",
+    max_new_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+    details=True,
+)
+print(response.generated_text)
+
+# For chat format
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
+
+Alternatively, you can use the OpenAI client:
+
+```python
+from openai import OpenAI
+
+# Initialize client pointing to vLLM endpoint
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="not-needed",  # vLLM doesn't require an API key by default
+)
+
+# Chat completion
+response = client.chat.completions.create(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
 {/if}
 
-### 6. What are the techniques to be aware of when batching sequences of different lengths together?
-
-<Question
-	choices={[
-		{
-			text: "Truncating",
-			explain: "Yes, truncation is a correct way of evening out sequences so that they fit in a rectangular shape. Is it the only one, though?",
-			correct: true
-		},
-		{
-			text: "Returning tensors",
-			explain: "While the other techniques allow you to return rectangular tensors, returning tensors isn't helpful when batching sequences together."
-		},
-		{
-			text: "Padding",
-			explain: "Yes, padding is a correct way of evening out sequences so that they fit in a rectangular shape. Is it the only one, though?",
-			correct: true
-		}, 
-		{
-			text: "Attention masking",
-			explain: "Absolutely! Attention masks are of prime importance when handling sequences of different lengths. That's not the only technique to be aware of, however.",
-			correct: true
-		} 
-	]}
-/>
-
-### 7. What is the point of applying a SoftMax function to the logits output by a sequence classification model?
-
-<Question
-	choices={[
-		{
-			text: "It softens the logits so that they're more reliable.",
-			explain: "No, the SoftMax function does not affect the reliability of results."
-		},
-		{
-			text: "It applies a lower and upper bound so that they're understandable.",
-			explain: "Correct! The resulting values are bound between 0 and 1. That's not the only reason we use a SoftMax function, though.",
-            correct: true
-		},
-		{
-			text: "The total sum of the output is then 1, resulting in a possible probabilistic interpretation.",
-			explain: "Correct! That's not the only reason we use a SoftMax function, though.",
-            correct: true
-		}
-	]}
-/>
-
-### 8. What method is most of the tokenizer API centered around?
-
-<Question
-	choices={[
-		{
-			text: "<code>encode</code>, as it can encode text into IDs and IDs into predictions",
-			explain: "Wrong! While the <code>encode</code> method does exist on tokenizers, it does not exist on models."
-		},
-		{
-			text: "Calling the tokenizer object directly.",
-			explain: "Exactly! The <code>__call__</code> method of the tokenizer is a very powerful method which can handle pretty much anything. It is also the method used to retrieve predictions from a model.",
-			correct: true
-		},
-		{
-			text: "<code>pad</code>",
-			explain: "Wrong! Padding is very useful, but it's just one part of the tokenizer API."
-		},
-		{
-			text: "<code>tokenize</code>",
-			explain: "The <code>tokenize</code> method is arguably one of the most useful methods, but it isn't the core of the tokenizer API."
-		}
-	]}
-/>
-
-### 9. What does the `result` variable contain in this code sample?
-
-```py
-from transformers import AutoTokenizer
-
-tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
-result = tokenizer.tokenize("Hello!")
-```
-
-<Question
-	choices={[
-		{
-			text: "A list of strings, each string being a token",
-			explain: "Absolutely! Convert this to IDs, and send them to a model!",
-            correct: true
-		},
-		{
-			text: "A list of IDs",
-			explain: "Incorrect; that's what the <code>__call__</code> or <code>convert_tokens_to_ids</code> method is for!"
-		},
-		{
-			text: "A string containing all of the tokens",
-			explain: "This would be suboptimal, as the goal is to split the string into multiple tokens."
-		}
-	]}
-/>
-
-{#if fw === 'pt'}
-### 10. Is there something wrong with the following code?
-
-```py
-from transformers import AutoTokenizer, AutoModel
-
-tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
-model = AutoModel.from_pretrained("gpt2")
-
-encoded = tokenizer("Hey!", return_tensors="pt")
-result = model(**encoded)
-```
-
-<Question
-	choices={[
-		{
-			text: "No, it seems correct.",
-			explain: "Unfortunately, coupling a model with a tokenizer that was trained with a different checkpoint is rarely a good idea. The model was not trained to make sense out of this tokenizer's output, so the model output (if it can even run!) will not make any sense."
-		},
-		{
-			text: "The tokenizer and model should always be from the same checkpoint.",
-			explain: "Right!",
-            correct: true
-		},
-		{
-			text: "It's good practice to pad and truncate with the tokenizer as every input is a batch.",
-			explain: "It's true that every model input needs to be a batch. However, truncating or padding this sequence wouldn't necessarily make sense as there is only one of it, and those are techniques to batch together a list of sentences."
-		}
-	]}
-/>
+### Basic Text Generation
+
+Let's look at examples of text generation with the frameworks:
+
+{#if framework === 'tgi'}
 
+First, deploy TGI with advanced parameters:
+```sh
+docker run --gpus all \
+    --shm-size 1g \
+    -p 8080:80 \
+    -v ~/.cache/huggingface:/data \
+    ghcr.io/huggingface/text-generation-inference:latest \
+    --model-id HuggingFaceTB/SmolLM2-360M-Instruct \
+    --max-total-tokens 4096 \
+    --max-input-length 3072 \
+    --max-batch-total-tokens 8192 \
+    --waiting-served-ratio 1.2
+```
+
+Use the InferenceClient for flexible text generation:
+```python
+from huggingface_hub import InferenceClient
+
+client = InferenceClient(model="http://localhost:8080")
+
+# Advanced parameters example
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,
+    max_tokens=200,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+
+# Raw text generation
+response = client.text_generation(
+    "Write a creative story about space exploration",
+    max_new_tokens=200,
+    temperature=0.8,
+    top_p=0.95,
+    repetition_penalty=1.1,
+    do_sample=True,
+    details=True,
+)
+print(response.generated_text)
+```
+
+Or use the OpenAI client:
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
+
+# Advanced parameters example
+response = client.chat.completions.create(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,  # Higher for more creativity
+)
+print(response.choices[0].message.content)
+```
+{:else if framework === 'llama.cpp'}
+
+For llama.cpp, you can set advanced parameters when launching the server:
+
+```sh
+./server \
+    -m smollm2-1.7b-instruct.Q4_K_M.gguf \
+    --host 0.0.0.0 \
+    --port 8080 \
+    -c 4096 \            # Context size
+    --threads 8 \        # CPU threads to use
+    --batch-size 512 \   # Batch size for prompt evaluation
+    --n-gpu-layers 0     # GPU layers (0 = CPU only)
+```
+
+Use the InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+client = InferenceClient(model="http://localhost:8080/v1", token="sk-no-key-required")
+
+# Advanced parameters example
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,
+    max_tokens=200,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+
+# For direct text generation
+response = client.text_generation(
+    "Write a creative story about space exploration",
+    max_new_tokens=200,
+    temperature=0.8,
+    top_p=0.95,
+    repetition_penalty=1.1,
+    details=True,
+)
+print(response.generated_text)
+```
+
+Or use the OpenAI client for generation with control over the sampling parameters:
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8080/v1", api_key="sk-no-key-required")
+
+# Advanced parameters example
+response = client.chat.completions.create(
+    model="smollm2-1.7b-instruct",
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Nucleus sampling probability
+    frequency_penalty=0.5,  # Reduce repetition of frequent tokens
+    presence_penalty=0.5,  # Reduce repetition by penalizing tokens already present
+    max_tokens=200,  # Maximum generation length
+)
+print(response.choices[0].message.content)
+```
+
+You can also use llama.cpp's native library for even more control:
+
+```python
+# Using llama-cpp-python package for direct model access
+from llama_cpp import Llama
+
+# Load the model
+llm = Llama(
+    model_path="smollm2-1.7b-instruct.Q4_K_M.gguf",
+    n_ctx=4096,  # Context window size
+    n_threads=8,  # CPU threads
+    n_gpu_layers=0,  # GPU layers (0 = CPU only)
+)
+
+# Format prompt according to the model's expected format
+prompt = """
system
+You are a creative storyteller.
+

+
user
+Write a creative story
+

+
assistant
+"""
+
+# Generate response with precise parameter control
+output = llm(
+    prompt,
+    max_tokens=200,
+    temperature=0.8,
+    top_p=0.95,
+    frequency_penalty=0.5,
+    presence_penalty=0.5,
+    stop=["
"],
+)
+
+print(output["choices"][0]["text"])
+```
+{:else}
+
+For advanced usage with vLLM, you can use the InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+client = InferenceClient(model="http://localhost:8000/v1")
+
+# Advanced parameters example
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,
+    max_tokens=200,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+
+# For direct text generation
+response = client.text_generation(
+    "Write a creative story about space exploration",
+    max_new_tokens=200,
+    temperature=0.8,
+    top_p=0.95,
+    details=True,
+)
+print(response.generated_text)
+```
+
+You can also use the OpenAI client:
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
+
+# Advanced parameters example
+response = client.chat.completions.create(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,
+    top_p=0.95,
+    max_tokens=200,
+)
+print(response.choices[0].message.content)
+```
+
+vLLM also provides a native Python interface with fine-grained control:
+
+```python
+from vllm import LLM, SamplingParams
+
+# Initialize the model with advanced parameters
+llm = LLM(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    gpu_memory_utilization=0.85,
+    max_num_batched_tokens=8192,
+    max_num_seqs=256,
+    block_size=16,
+)
+
+# Configure sampling parameters
+sampling_params = SamplingParams(
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Consider top 95% probability mass
+    max_tokens=100,  # Maximum length
+    presence_penalty=1.1,  # Reduce repetition
+    frequency_penalty=1.1,  # Reduce repetition
+    stop=["\n\n", "###"],  # Stop sequences
+)
+
+# Generate text
+prompt = "Write a creative story"
+outputs = llm.generate(prompt, sampling_params)
+print(outputs[0].outputs[0].text)
+
+# For chat-style interactions
+chat_prompt = [
+    {"role": "system", "content": "You are a creative storyteller."},
+    {"role": "user", "content": "Write a creative story"},
+]
+formatted_prompt = llm.get_chat_template()(chat_prompt)  # Uses model's chat template
+outputs = llm.generate(formatted_prompt, sampling_params)
+print(outputs[0].outputs[0].text)
+```
+{/if}
+
+## Advanced Generation Control
+
+### Token Selection and Sampling
+
+The process of generating text involves selecting the next token at each step. This selection process can be controlled through various parameters:
+
+1. **Raw Logits**: The initial output probabilities for each token
+2. **Temperature**: Controls randomness in selection (higher = more creative)
+3. **Top-p (Nucleus) Sampling**: Filters to top tokens making up X% of probability mass
+4. **Top-k Filtering**: Limits selection to k most likely tokens
+
+Here's how to configure these parameters:
+
+{#if framework === 'tgi'}
+```python
+client.generate(
+    "Write a creative story",
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Consider top 95% probability mass
+    top_k=50,  # Consider top 50 tokens
+    max_new_tokens=100,  # Maximum length
+    repetition_penalty=1.1,  # Reduce repetition
+)
+```
+{:else if framework === 'llama.cpp'}
+```python
+# Via OpenAI API compatibility
+response = client.completions.create(
+    model="smollm2-1.7b-instruct",  # Model name (can be any string for llama.cpp server)
+    prompt="Write a creative story",
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Consider top 95% probability mass
+    frequency_penalty=1.1,  # Reduce repetition
+    presence_penalty=0.1,  # Reduce repetition
+    max_tokens=100,  # Maximum length
+)
+
+# Via llama-cpp-python direct access
+output = llm(
+    "Write a creative story",
+    temperature=0.8,
+    top_p=0.95,
+    top_k=50,
+    max_tokens=100,
+    repeat_penalty=1.1,
+)
+```
+{:else}
+```python
+params = SamplingParams(
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Consider top 95% probability mass
+    top_k=50,  # Consider top 50 tokens
+    max_tokens=100,  # Maximum length
+    presence_penalty=0.1,  # Reduce repetition
+)
+llm.generate("Write a creative story", sampling_params=params)
+```
+{/if}
+
+### Controlling Repetition
+
+Both frameworks provide ways to prevent repetitive text generation:
+
+{#if framework === 'tgi'}
+```python
+client.generate(
+    "Write a varied text",
+    repetition_penalty=1.1,  # Penalize repeated tokens
+    no_repeat_ngram_size=3,  # Prevent 3-gram repetition
+)
+```
+{:else if framework === 'llama.cpp'}
+```python
+# Via OpenAI API
+response = client.completions.create(
+    model="smollm2-1.7b-instruct",
+    prompt="Write a varied text",
+    frequency_penalty=1.1,  # Penalize frequent tokens
+    presence_penalty=0.8,  # Penalize tokens already present
+)
+
+# Via direct library
+output = llm(
+    "Write a varied text",
+    repeat_penalty=1.1,  # Penalize repeated tokens
+    frequency_penalty=0.5,  # Additional frequency penalty
+    presence_penalty=0.5,  # Additional presence penalty
+)
+```
+{:else}
+```python
+params = SamplingParams(
+    presence_penalty=0.1,  # Penalize token presence
+    frequency_penalty=0.1,  # Penalize token frequency
+)
+```
+{/if}
+
+### Length Control and Stop Sequences
+
+You can control generation length and specify when to stop:
+
+{#if framework === 'tgi'}
+```python
+client.generate(
+    "Generate a short paragraph",
+    max_new_tokens=100,
+    min_new_tokens=10,
+    stop_sequences=["\n\n", "###"],
+)
+```
+{:else if framework === 'llama.cpp'}
+```python
+# Via OpenAI API
+response = client.completions.create(
+    model="smollm2-1.7b-instruct",
+    prompt="Generate a short paragraph",
+    max_tokens=100,
+    stop=["\n\n", "###"],
+)
+
+# Via direct library
+output = llm("Generate a short paragraph", max_tokens=100, stop=["\n\n", "###"])
+```
+{:else}
+```python
+params = SamplingParams(
+    max_tokens=100,
+    min_tokens=10,
+    stop=["###", "\n\n"],
+    ignore_eos=False,
+    skip_special_tokens=True,
+)
+```
+{/if}
+
+## Memory Management
+
+Both frameworks implement advanced memory management techniques for efficient inference.
+
+{#if framework === 'tgi'}
+TGI uses Flash Attention 2 and continuous batching:
+
+```sh
+# Docker deployment with memory optimization
+docker run --gpus all -p 8080:80 \
+    --shm-size 1g \
+    ghcr.io/huggingface/text-generation-inference:latest \
+    --model-id HuggingFaceTB/SmolLM2-1.7B-Instruct \
+    --max-batch-total-tokens 8192 \
+    --max-input-length 4096
+```
+{:else if framework === 'llama.cpp'}
+llama.cpp uses quantization and optimized memory layout:
+
+```sh
+# Server with memory optimizations
+./server \
+    -m smollm2-1.7b-instruct.Q4_K_M.gguf \
+    --host 0.0.0.0 \
+    --port 8080 \
+    -c 2048 \               # Context size
+    --threads 4 \           # CPU threads
+    --n-gpu-layers 32 \     # Use more GPU layers for larger models
+    --mlock \               # Lock memory to prevent swapping
+    --cont-batching         # Enable continuous batching
+```
+
+For models too large for your GPU, you can use CPU offloading:
+
+```sh
+./server \
+    -m smollm2-1.7b-instruct.Q4_K_M.gguf \
+    --n-gpu-layers 20 \     # Keep first 20 layers on GPU
+    --threads 8             # Use more CPU threads for CPU layers
+```
 {:else}
-### 10. Is there something wrong with the following code?
-
-```py
-from transformers import AutoTokenizer, TFAutoModel
-
-tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
-model = TFAutoModel.from_pretrained("gpt2")
-
-encoded = tokenizer("Hey!", return_tensors="pt")
-result = model(**encoded)
-```
-
-<Question
-	choices={[
-		{
-			text: "No, it seems correct.",
-			explain: "Unfortunately, coupling a model with a tokenizer that was trained with a different checkpoint is rarely a good idea. The model was not trained to make sense out of this tokenizer's output, so the model output (if it can even run!) will not make any sense."
-		},
-		{
-			text: "The tokenizer and model should always be from the same checkpoint.",
-			explain: "Right!",
-            correct: true
-		},
-		{
-			text: "It's good practice to pad and truncate with the tokenizer as every input is a batch.",
-			explain: "It's true that every model input needs to be a batch. However, truncating or padding this sequence wouldn't necessarily make sense as there is only one of it, and those are techniques to batch together a list of sentences."
-		}
-	]}
-/>
+vLLM uses PagedAttention for optimal memory management:
+
+```python
+from vllm.engine.arg_utils import AsyncEngineArgs
 
+engine_args = AsyncEngineArgs(
+    model="HuggingFaceTB/SmolLM2-1.7B-Instruct",
+    gpu_memory_utilization=0.85,
+    max_num_batched_tokens=8192,
+    block_size=16,
+)
+
+llm = LLM(engine_args=engine_args)
+```
 {/if}
+
+## Resources
+
+- [Text Generation Inference Documentation](https://huggingface.co/docs/text-generation-inference)
+- [TGI GitHub Repository](https://github.com/huggingface/text-generation-inference)
+- [vLLM Documentation](https://vllm.readthedocs.io/)
+- [vLLM GitHub Repository](https://github.com/vllm-project/vllm)
+- [PagedAttention Paper](https://arxiv.org/abs/2309.06180)
+- [HuggingFace Blog: TGI in Production](https://huggingface.co/blog/tgi-serving)
path: chapters/en/chapter2/8.mdx
user: login: sergiopaniego
body: 404
created_at: 2025-05-14T13:40:08Z
author_association: MEMBER
diff_hunk: @@ -1,310 +1,779 @@
-<FrameworkSwitchCourse {fw} />
-
-<!-- DISABLE-FRONTMATTER-SECTIONS -->
-
-# End-of-chapter quiz[[end-of-chapter-quiz]]
-
-<CourseFloatingBanner
-    chapter={2}
-    classNames="absolute z-10 right-0 top-0"
-/>
-
-### 1. What is the order of the language modeling pipeline?
-
-<Question
-	choices={[
-		{
-			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
-			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
-		},
-		{
-			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
-			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
-		},
-		{
-			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
-			explain: "Correct! The tokenizer can be used for both tokenizing and de-tokenizing.",
-            correct: true
-		}
-	]}
-/>
-
-### 2. How many dimensions does the tensor output by the base Transformer model have, and what are they?
-
-<Question
-	choices={[
-		{
-			text: "2: The sequence length and the batch size",
-			explain: "False! The tensor output by the model has a third dimension: hidden size."
-		},
-		{
-			text: "2: The sequence length and the hidden size",
-			explain: "False! All Transformer models handle batches, even with a single sequence; that would be a batch size of 1!"
-		},
-		{
-			text: "3: The sequence length, the batch size, and the hidden size",
-			explain: "Correct!",
-            correct: true
-		}
-	]}
-/>
-
-### 3. Which of the following is an example of subword tokenization?
-
-<Question
-	choices={[
-		{
-			text: "WordPiece",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-		},
-		{
-			text: "Character-based tokenization",
-			explain: "Character-based tokenization is not a type of subword tokenization."
-		},
-		{
-			text: "Splitting on whitespace and punctuation",
-			explain: "That's a word-based tokenization scheme!"
-		},
-		{
-			text: "BPE",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-        },
-		{
-			text: "Unigram",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-        },
-		{
-			text: "None of the above",
-			explain: "Incorrect!"
-        }
-	]}
-/>
-
-### 4. What is a model head?
-
-<Question
-	choices={[
-		{
-			text: "A component of the base Transformer network that redirects tensors to their correct layers",
-			explain: "Incorrect! There's no such component."
-		},
-		{
-			text: "Also known as the self-attention mechanism, it adapts the representation of a token according to the other tokens of the sequence",
-			explain: "Incorrect! The self-attention layer does contain attention \"heads,\" but these are not adaptation heads."
-		},
-		{
-			text: "An additional component, usually made up of one or a few layers, to convert the transformer predictions to a task-specific output",
-			explain: "That's right. Adaptation heads, also known simply as heads, come up in different forms: language modeling heads, question answering heads, sequence classification heads... ",
-			correct: true
-		} 
-	]}
-/>
-
-{#if fw === 'pt'}
-### 5. What is an AutoModel?
-
-<Question
-	choices={[
-		{
-			text: "A model that automatically trains on your data",
-			explain: "Incorrect. Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
-		},
-		{
-			text: "An object that returns the correct architecture based on the checkpoint",
-			explain: "Exactly: the <code>AutoModel</code> only needs to know the checkpoint from which to initialize to return the correct architecture.",
-			correct: true
-		},
-		{
-			text: "A model that automatically detects the language used for its inputs to load the correct weights",
-			explain: "Incorrect; while some checkpoints and models are capable of handling multiple languages, there are no built-in tools for automatic checkpoint selection according to language. You should head over to the <a href='https://huggingface.co/models'>Model Hub</a> to find the best checkpoint for your task!"
-		} 
-	]}
-/>
+# Optimized Inference Deployment
 
+In this section, we'll explore advanced frameworks for optimizing LLM deployments: Text Generation Inference (TGI), vLLM, and llama.cpp. These applications are primarily used in production environments to serve LLMs to users. This section focuses on how to deploy these frameworks in production rather than how to use them for inference on a single machine.
+
+We'll cover how these tools maximize inference efficiency and simplify production deployments of Large Language Models.
+
+## Framework Selection Guide
+
+TGI, vLLM, and llama.cpp serve similar purposes but have distinct characteristics that make them better suited for different use cases. Let's look at the key differences between them, focusing on performance and integration.
+
+### Memory Management and Performance
+
+**TGI** is designed to be stable and predictable in production, using fixed sequence lengths to keep memory usage consistent. TGI manages memory using Flash Attention 2 and continuous batching techniques. This means it can process attention calculations very efficiently and keep the GPU busy by constantly feeding it work. The system can move parts of the model between CPU and GPU when needed, which helps handle larger models. 
+
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/tgi/flash-attn.png" alt="Flash Attention" />
+
+<Tip title="How Flash Attention Works">
+Flash Attention is a technique that optimizes the attention mechanism in transformer models by addressing memory bandwidth bottlenecks. As discussed earlier in [section 12.3](2.mdx), the attention mechanism has quadratic complexity and memory usage, making it inefficient for long sequences.
+
+The key innovation is in how it manages memory transfers between High Bandwidth Memory (HBM) and faster SRAM cache. Traditional attention repeatedly transfers data between HBM and SRAM, creating bottlenecks by leaving the GPU idle. Flash Attention loads data once into SRAM and performs all calculations there, minimizing expensive memory transfers. 
+
+While the benefits are most significant during training, Flash Attention's reduced VRAM usage and improved efficiency make it valuable for inference as well, enabling faster and more scalable LLM serving.
+</Tip>
+
+**vLLM** takes a different approach by using PagedAttention. Just like how a computer manages its memory in pages, vLLM splits the model's memory into smaller blocks. This clever system means it can handle different-sized requests more flexibly and doesn't waste memory space. It's particularly good at sharing memory between different requests and reduces memory fragmentation, which makes the whole system more efficient.
+
+<Tip title="How Paged Attention Works">
+Paged Attention is a technique that addresses another critical bottleneck in LLM inference: KV cache memory management. As discussed in [section 12.3](2.mdx), during text generation, the model stores attention keys and values (KV cache) for each generated token to reduce redundant computations. The KV cache can become enormous, especially with long sequences or multiple concurrent requests.
+
+vLLM's key innovation lies in how it manages this cache:
+
+1. **Memory Paging**: Instead of treating the KV cache as one large block, it's divided into fixed-size "pages" (similar to virtual memory in operating systems).
+2. **Non-contiguous Storage**: Pages don't need to be stored contiguously in GPU memory, allowing for more flexible memory allocation.
+3. **Page Table Management**: A page table tracks which pages belong to which sequence, enabling efficient lookup and access.
+4. **Memory Sharing**: For operations like parallel sampling, pages storing the KV cache for the prompt can be shared across multiple sequences.
+
+The PagedAttention approach can lead to up to 24x higher throughput compared to traditional methods, making it a game-changer for production LLM deployments. If you want to go really deep into how PagedAttention works, you can read the [the guide from the vLLM documentation](https://docs.vllm.ai/en/latest/design/kernel/paged_attention.html).
+</Tip>
+
+**llama.cpp** is a highly optimized C/C++ implementation originally designed for running LLaMA models on consumer hardware. It focuses on CPU efficiency with optional GPU acceleration and is ideal for resource-constrained environments. llama.cpp uses quantization techniques to reduce model size and memory requirements while maintaining good performance. It implements optimized kernels for various CPU architectures and supports basic KV cache management for efficient token generation.
+
+<Tip title="How llama.cpp Quantization Works">
+Quantization in llama.cpp reduces the precision of model weights from 32-bit or 16-bit floating point to lower precision formats like 8-bit integers (INT8), 4-bit, or even lower. This significantly reduces memory usage and improves inference speed with minimal quality loss.
+
+Key quantization features in llama.cpp include:
+1. **Multiple Quantization Levels**: Supports 8-bit, 4-bit, 3-bit, and even 2-bit quantization
+2. **GGML/GGUF Format**: Uses custom tensor formats optimized for quantized inference
+3. **Mixed Precision**: Can apply different quantization levels to different parts of the model
+4. **Hardware-Specific Optimizations**: Includes optimized code paths for various CPU architectures (AVX2, AVX-512, NEON)
+
+This approach enables running billion-parameter models on consumer hardware with limited memory, making it perfect for local deployments and edge devices.
+</Tip>
+
+
+
+### Deployment and Integration
+
+Let's move on to the deployment and integration differences between the frameworks.
+
+**TGI** excels in enterprise-level deployment with its production-ready features. It comes with built-in Kubernetes support and includes everything you need for running in production, like monitoring through Prometheus and Grafana, automatic scaling, and comprehensive safety features. The system also includes enterprise-grade logging and various protective measures like content filtering and rate limiting to keep your deployment secure and stable.
+
+**vLLM** takes a more flexible, developer-friendly approach to deployment. It's built with Python at its core and can easily replace OpenAI's API in your existing applications. The framework focuses on delivering raw performance and can be customized to fit your specific needs. It works particularly well with Ray for managing clusters, making it a great choice when you need high performance and adaptability.
+
+**llama.cpp** prioritizes simplicity and portability. Its server implementation is lightweight and can run on a wide range of hardware, from powerful servers to consumer laptops and even some high-end mobile devices. With minimal dependencies and a simple C/C++ core, it's easy to deploy in environments where installing Python frameworks would be challenging. The server provides an OpenAI-compatible API while maintaining a much smaller resource footprint than other solutions.
+
+
+
+## Getting Started
+
+Let's explore how to use these frameworks for deploying LLMs, starting with installation and basic setup.
+
+### Installation and Basic Setup
+
+{#if framework === 'tgi'}
+
+TGI is easy to install and use, with deep integration into the Hugging Face ecosystem.
+
+First, launch the TGI server using Docker:
+
+```sh
+docker run --gpus all \
+    --shm-size 1g \
+    -p 8080:80 \
+    -v ~/.cache/huggingface:/data \
+    ghcr.io/huggingface/text-generation-inference:latest \
+    --model-id HuggingFaceTB/SmolLM2-360M-Instruct
+```
+
+Then interact with it using Hugging Face's InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+# Initialize client pointing to TGI endpoint
+client = InferenceClient(
+    model="http://localhost:8080",  # URL to the TGI server
+)
+
+# Text generation
+response = client.text_generation(
+    "Tell me a story",
+    max_new_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+    details=True,
+    stop_sequences=[],
+)
+print(response.generated_text)
+
+# For chat format
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
+
+Alternatively, you can use the OpenAI client:
+
+```python
+from openai import OpenAI
+
+# Initialize client pointing to TGI endpoint
+client = OpenAI(
+    base_url="http://localhost:8080/v1",  # Make sure to include /v1
+    api_key="not-needed",  # TGI doesn't require an API key by default
+)
+
+# Chat completion
+response = client.chat.completions.create(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
+{:else if framework === 'llama.cpp'}
+
+llama.cpp is easy to install and use, requiring minimal dependencies and supporting both CPU and GPU inference.
+
+First, install and build llama.cpp:
+
+```sh
+# Clone the repository
+git clone https://github.com/ggerganov/llama.cpp
+cd llama.cpp
+
+# Build the project
+make
+
+# Download the SmolLM2-1.7B-Instruct-GGUF model
+curl -L -O https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct-GGUF/resolve/main/smollm2-1.7b-instruct.Q4_K_M.gguf
+```
+
+Then, launch the server (with OpenAI API compatibility):
+
+```sh
+# Start the server
+./server \
+    -m smollm2-1.7b-instruct.Q4_K_M.gguf \
+    --host 0.0.0.0 \
+    --port 8080 \
+    -c 4096 \
+    --n-gpu-layers 0  # Set to a higher number to use GPU
+```
+
+Interact with the server using Hugging Face's InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+# Initialize client pointing to llama.cpp server
+client = InferenceClient(
+    model="http://localhost:8080/v1",  # URL to the llama.cpp server
+    token="sk-no-key-required",  # llama.cpp server requires this placeholder
+)
+
+# Text generation
+response = client.text_generation(
+    "Tell me a story",
+    max_new_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+    details=True,
+)
+print(response.generated_text)
+
+# For chat format
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
+
+Alternatively, you can use the OpenAI client:
+
+```python
+from openai import OpenAI
+
+# Initialize client pointing to llama.cpp server
+client = OpenAI(
+    base_url="http://localhost:8080/v1",
+    api_key="sk-no-key-required",  # llama.cpp server requires this placeholder
+)
+
+# Chat completion
+response = client.chat.completions.create(
+    model="smollm2-1.7b-instruct",  # Model identifier can be anything as server only loads one model
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
 {:else}
-### 5. What is an TFAutoModel?
-
-<Question
-	choices={[
-		{
-			text: "A model that automatically trains on your data",
-			explain: "Incorrect. Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
-		},
-		{
-			text: "An object that returns the correct architecture based on the checkpoint",
-			explain: "Exactly: the <code>TFAutoModel</code> only needs to know the checkpoint from which to initialize to return the correct architecture.",
-			correct: true
-		},
-		{
-			text: "A model that automatically detects the language used for its inputs to load the correct weights",
-			explain: "Incorrect; while some checkpoints and models are capable of handling multiple languages, there are no built-in tools for automatic checkpoint selection according to language. You should head over to the <a href='https://huggingface.co/models'>Model Hub</a> to find the best checkpoint for your task!"
-		} 
-	]}
-/>
 
+vLLM is easy to install and use, with both OpenAI API compatibility and a native Python interface.
+
+First, launch the vLLM OpenAI-compatible server:
+
+```sh
+python -m vllm.entrypoints.openai.api_server \
+    --model HuggingFaceTB/SmolLM2-360M-Instruct \
+    --host 0.0.0.0 \
+    --port 8000
+```
+
+Then interact with it using Hugging Face's InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+# Initialize client pointing to vLLM endpoint
+client = InferenceClient(
+    model="http://localhost:8000/v1",  # URL to the vLLM server
+)
+
+# Text generation
+response = client.text_generation(
+    "Tell me a story",
+    max_new_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+    details=True,
+)
+print(response.generated_text)
+
+# For chat format
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
+
+Alternatively, you can use the OpenAI client:
+
+```python
+from openai import OpenAI
+
+# Initialize client pointing to vLLM endpoint
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="not-needed",  # vLLM doesn't require an API key by default
+)
+
+# Chat completion
+response = client.chat.completions.create(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
 {/if}
 
-### 6. What are the techniques to be aware of when batching sequences of different lengths together?
-
-<Question
-	choices={[
-		{
-			text: "Truncating",
-			explain: "Yes, truncation is a correct way of evening out sequences so that they fit in a rectangular shape. Is it the only one, though?",
-			correct: true
-		},
-		{
-			text: "Returning tensors",
-			explain: "While the other techniques allow you to return rectangular tensors, returning tensors isn't helpful when batching sequences together."
-		},
-		{
-			text: "Padding",
-			explain: "Yes, padding is a correct way of evening out sequences so that they fit in a rectangular shape. Is it the only one, though?",
-			correct: true
-		}, 
-		{
-			text: "Attention masking",
-			explain: "Absolutely! Attention masks are of prime importance when handling sequences of different lengths. That's not the only technique to be aware of, however.",
-			correct: true
-		} 
-	]}
-/>
-
-### 7. What is the point of applying a SoftMax function to the logits output by a sequence classification model?
-
-<Question
-	choices={[
-		{
-			text: "It softens the logits so that they're more reliable.",
-			explain: "No, the SoftMax function does not affect the reliability of results."
-		},
-		{
-			text: "It applies a lower and upper bound so that they're understandable.",
-			explain: "Correct! The resulting values are bound between 0 and 1. That's not the only reason we use a SoftMax function, though.",
-            correct: true
-		},
-		{
-			text: "The total sum of the output is then 1, resulting in a possible probabilistic interpretation.",
-			explain: "Correct! That's not the only reason we use a SoftMax function, though.",
-            correct: true
-		}
-	]}
-/>
-
-### 8. What method is most of the tokenizer API centered around?
-
-<Question
-	choices={[
-		{
-			text: "<code>encode</code>, as it can encode text into IDs and IDs into predictions",
-			explain: "Wrong! While the <code>encode</code> method does exist on tokenizers, it does not exist on models."
-		},
-		{
-			text: "Calling the tokenizer object directly.",
-			explain: "Exactly! The <code>__call__</code> method of the tokenizer is a very powerful method which can handle pretty much anything. It is also the method used to retrieve predictions from a model.",
-			correct: true
-		},
-		{
-			text: "<code>pad</code>",
-			explain: "Wrong! Padding is very useful, but it's just one part of the tokenizer API."
-		},
-		{
-			text: "<code>tokenize</code>",
-			explain: "The <code>tokenize</code> method is arguably one of the most useful methods, but it isn't the core of the tokenizer API."
-		}
-	]}
-/>
-
-### 9. What does the `result` variable contain in this code sample?
-
-```py
-from transformers import AutoTokenizer
-
-tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
-result = tokenizer.tokenize("Hello!")
-```
-
-<Question
-	choices={[
-		{
-			text: "A list of strings, each string being a token",
-			explain: "Absolutely! Convert this to IDs, and send them to a model!",
-            correct: true
-		},
-		{
-			text: "A list of IDs",
-			explain: "Incorrect; that's what the <code>__call__</code> or <code>convert_tokens_to_ids</code> method is for!"
-		},
-		{
-			text: "A string containing all of the tokens",
-			explain: "This would be suboptimal, as the goal is to split the string into multiple tokens."
-		}
-	]}
-/>
-
-{#if fw === 'pt'}
-### 10. Is there something wrong with the following code?
-
-```py
-from transformers import AutoTokenizer, AutoModel
-
-tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
-model = AutoModel.from_pretrained("gpt2")
-
-encoded = tokenizer("Hey!", return_tensors="pt")
-result = model(**encoded)
-```
-
-<Question
-	choices={[
-		{
-			text: "No, it seems correct.",
-			explain: "Unfortunately, coupling a model with a tokenizer that was trained with a different checkpoint is rarely a good idea. The model was not trained to make sense out of this tokenizer's output, so the model output (if it can even run!) will not make any sense."
-		},
-		{
-			text: "The tokenizer and model should always be from the same checkpoint.",
-			explain: "Right!",
-            correct: true
-		},
-		{
-			text: "It's good practice to pad and truncate with the tokenizer as every input is a batch.",
-			explain: "It's true that every model input needs to be a batch. However, truncating or padding this sequence wouldn't necessarily make sense as there is only one of it, and those are techniques to batch together a list of sentences."
-		}
-	]}
-/>
+### Basic Text Generation
+
+Let's look at examples of text generation with the frameworks:
+
+{#if framework === 'tgi'}
 
+First, deploy TGI with advanced parameters:
+```sh
+docker run --gpus all \
+    --shm-size 1g \
+    -p 8080:80 \
+    -v ~/.cache/huggingface:/data \
+    ghcr.io/huggingface/text-generation-inference:latest \
+    --model-id HuggingFaceTB/SmolLM2-360M-Instruct \
+    --max-total-tokens 4096 \
+    --max-input-length 3072 \
+    --max-batch-total-tokens 8192 \
+    --waiting-served-ratio 1.2
+```
+
+Use the InferenceClient for flexible text generation:
+```python
+from huggingface_hub import InferenceClient
+
+client = InferenceClient(model="http://localhost:8080")
+
+# Advanced parameters example
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,
+    max_tokens=200,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+
+# Raw text generation
+response = client.text_generation(
+    "Write a creative story about space exploration",
+    max_new_tokens=200,
+    temperature=0.8,
+    top_p=0.95,
+    repetition_penalty=1.1,
+    do_sample=True,
+    details=True,
+)
+print(response.generated_text)
+```
+
+Or use the OpenAI client:
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
+
+# Advanced parameters example
+response = client.chat.completions.create(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,  # Higher for more creativity
+)
+print(response.choices[0].message.content)
+```
+{:else if framework === 'llama.cpp'}
+
+For llama.cpp, you can set advanced parameters when launching the server:
+
+```sh
+./server \
+    -m smollm2-1.7b-instruct.Q4_K_M.gguf \
+    --host 0.0.0.0 \
+    --port 8080 \
+    -c 4096 \            # Context size
+    --threads 8 \        # CPU threads to use
+    --batch-size 512 \   # Batch size for prompt evaluation
+    --n-gpu-layers 0     # GPU layers (0 = CPU only)
+```
+
+Use the InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+client = InferenceClient(model="http://localhost:8080/v1", token="sk-no-key-required")
+
+# Advanced parameters example
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,
+    max_tokens=200,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+
+# For direct text generation
+response = client.text_generation(
+    "Write a creative story about space exploration",
+    max_new_tokens=200,
+    temperature=0.8,
+    top_p=0.95,
+    repetition_penalty=1.1,
+    details=True,
+)
+print(response.generated_text)
+```
+
+Or use the OpenAI client for generation with control over the sampling parameters:
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8080/v1", api_key="sk-no-key-required")
+
+# Advanced parameters example
+response = client.chat.completions.create(
+    model="smollm2-1.7b-instruct",
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Nucleus sampling probability
+    frequency_penalty=0.5,  # Reduce repetition of frequent tokens
+    presence_penalty=0.5,  # Reduce repetition by penalizing tokens already present
+    max_tokens=200,  # Maximum generation length
+)
+print(response.choices[0].message.content)
+```
+
+You can also use llama.cpp's native library for even more control:
+
+```python
+# Using llama-cpp-python package for direct model access
+from llama_cpp import Llama
+
+# Load the model
+llm = Llama(
+    model_path="smollm2-1.7b-instruct.Q4_K_M.gguf",
+    n_ctx=4096,  # Context window size
+    n_threads=8,  # CPU threads
+    n_gpu_layers=0,  # GPU layers (0 = CPU only)
+)
+
+# Format prompt according to the model's expected format
+prompt = """
system
+You are a creative storyteller.
+

+
user
+Write a creative story
+

+
assistant
+"""
+
+# Generate response with precise parameter control
+output = llm(
+    prompt,
+    max_tokens=200,
+    temperature=0.8,
+    top_p=0.95,
+    frequency_penalty=0.5,
+    presence_penalty=0.5,
+    stop=["
"],
+)
+
+print(output["choices"][0]["text"])
+```
+{:else}
+
+For advanced usage with vLLM, you can use the InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+client = InferenceClient(model="http://localhost:8000/v1")
+
+# Advanced parameters example
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,
+    max_tokens=200,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+
+# For direct text generation
+response = client.text_generation(
+    "Write a creative story about space exploration",
+    max_new_tokens=200,
+    temperature=0.8,
+    top_p=0.95,
+    details=True,
+)
+print(response.generated_text)
+```
+
+You can also use the OpenAI client:
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
+
+# Advanced parameters example
+response = client.chat.completions.create(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,
+    top_p=0.95,
+    max_tokens=200,
+)
+print(response.choices[0].message.content)
+```
+
+vLLM also provides a native Python interface with fine-grained control:
+
+```python
+from vllm import LLM, SamplingParams
+
+# Initialize the model with advanced parameters
+llm = LLM(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    gpu_memory_utilization=0.85,
+    max_num_batched_tokens=8192,
+    max_num_seqs=256,
+    block_size=16,
+)
+
+# Configure sampling parameters
+sampling_params = SamplingParams(
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Consider top 95% probability mass
+    max_tokens=100,  # Maximum length
+    presence_penalty=1.1,  # Reduce repetition
+    frequency_penalty=1.1,  # Reduce repetition
+    stop=["\n\n", "###"],  # Stop sequences
+)
+
+# Generate text
+prompt = "Write a creative story"
+outputs = llm.generate(prompt, sampling_params)
+print(outputs[0].outputs[0].text)
+
+# For chat-style interactions
+chat_prompt = [
+    {"role": "system", "content": "You are a creative storyteller."},
+    {"role": "user", "content": "Write a creative story"},
+]
+formatted_prompt = llm.get_chat_template()(chat_prompt)  # Uses model's chat template
+outputs = llm.generate(formatted_prompt, sampling_params)
+print(outputs[0].outputs[0].text)
+```
+{/if}
+
+## Advanced Generation Control
+
+### Token Selection and Sampling
+
+The process of generating text involves selecting the next token at each step. This selection process can be controlled through various parameters:
+
+1. **Raw Logits**: The initial output probabilities for each token
+2. **Temperature**: Controls randomness in selection (higher = more creative)
+3. **Top-p (Nucleus) Sampling**: Filters to top tokens making up X% of probability mass
+4. **Top-k Filtering**: Limits selection to k most likely tokens
+
+Here's how to configure these parameters:
+
+{#if framework === 'tgi'}
+```python
+client.generate(
+    "Write a creative story",
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Consider top 95% probability mass
+    top_k=50,  # Consider top 50 tokens
+    max_new_tokens=100,  # Maximum length
+    repetition_penalty=1.1,  # Reduce repetition
+)
+```
+{:else if framework === 'llama.cpp'}
+```python
+# Via OpenAI API compatibility
+response = client.completions.create(
+    model="smollm2-1.7b-instruct",  # Model name (can be any string for llama.cpp server)
+    prompt="Write a creative story",
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Consider top 95% probability mass
+    frequency_penalty=1.1,  # Reduce repetition
+    presence_penalty=0.1,  # Reduce repetition
+    max_tokens=100,  # Maximum length
+)
+
+# Via llama-cpp-python direct access
+output = llm(
+    "Write a creative story",
+    temperature=0.8,
+    top_p=0.95,
+    top_k=50,
+    max_tokens=100,
+    repeat_penalty=1.1,
+)
+```
+{:else}
+```python
+params = SamplingParams(
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Consider top 95% probability mass
+    top_k=50,  # Consider top 50 tokens
+    max_tokens=100,  # Maximum length
+    presence_penalty=0.1,  # Reduce repetition
+)
+llm.generate("Write a creative story", sampling_params=params)
+```
+{/if}
+
+### Controlling Repetition
+
+Both frameworks provide ways to prevent repetitive text generation:
+
+{#if framework === 'tgi'}
+```python
+client.generate(
+    "Write a varied text",
+    repetition_penalty=1.1,  # Penalize repeated tokens
+    no_repeat_ngram_size=3,  # Prevent 3-gram repetition
+)
+```
+{:else if framework === 'llama.cpp'}
+```python
+# Via OpenAI API
+response = client.completions.create(
+    model="smollm2-1.7b-instruct",
+    prompt="Write a varied text",
+    frequency_penalty=1.1,  # Penalize frequent tokens
+    presence_penalty=0.8,  # Penalize tokens already present
+)
+
+# Via direct library
+output = llm(
+    "Write a varied text",
+    repeat_penalty=1.1,  # Penalize repeated tokens
+    frequency_penalty=0.5,  # Additional frequency penalty
+    presence_penalty=0.5,  # Additional presence penalty
+)
+```
+{:else}
+```python
+params = SamplingParams(
+    presence_penalty=0.1,  # Penalize token presence
+    frequency_penalty=0.1,  # Penalize token frequency
+)
+```
+{/if}
+
+### Length Control and Stop Sequences
+
+You can control generation length and specify when to stop:
+
+{#if framework === 'tgi'}
+```python
+client.generate(
+    "Generate a short paragraph",
+    max_new_tokens=100,
+    min_new_tokens=10,
+    stop_sequences=["\n\n", "###"],
+)
+```
+{:else if framework === 'llama.cpp'}
+```python
+# Via OpenAI API
+response = client.completions.create(
+    model="smollm2-1.7b-instruct",
+    prompt="Generate a short paragraph",
+    max_tokens=100,
+    stop=["\n\n", "###"],
+)
+
+# Via direct library
+output = llm("Generate a short paragraph", max_tokens=100, stop=["\n\n", "###"])
+```
+{:else}
+```python
+params = SamplingParams(
+    max_tokens=100,
+    min_tokens=10,
+    stop=["###", "\n\n"],
+    ignore_eos=False,
+    skip_special_tokens=True,
+)
+```
+{/if}
+
+## Memory Management
+
+Both frameworks implement advanced memory management techniques for efficient inference.
+
+{#if framework === 'tgi'}
+TGI uses Flash Attention 2 and continuous batching:
+
+```sh
+# Docker deployment with memory optimization
+docker run --gpus all -p 8080:80 \
+    --shm-size 1g \
+    ghcr.io/huggingface/text-generation-inference:latest \
+    --model-id HuggingFaceTB/SmolLM2-1.7B-Instruct \
+    --max-batch-total-tokens 8192 \
+    --max-input-length 4096
+```
+{:else if framework === 'llama.cpp'}
+llama.cpp uses quantization and optimized memory layout:
+
+```sh
+# Server with memory optimizations
+./server \
+    -m smollm2-1.7b-instruct.Q4_K_M.gguf \
+    --host 0.0.0.0 \
+    --port 8080 \
+    -c 2048 \               # Context size
+    --threads 4 \           # CPU threads
+    --n-gpu-layers 32 \     # Use more GPU layers for larger models
+    --mlock \               # Lock memory to prevent swapping
+    --cont-batching         # Enable continuous batching
+```
+
+For models too large for your GPU, you can use CPU offloading:
+
+```sh
+./server \
+    -m smollm2-1.7b-instruct.Q4_K_M.gguf \
+    --n-gpu-layers 20 \     # Keep first 20 layers on GPU
+    --threads 8             # Use more CPU threads for CPU layers
+```
 {:else}
-### 10. Is there something wrong with the following code?
-
-```py
-from transformers import AutoTokenizer, TFAutoModel
-
-tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
-model = TFAutoModel.from_pretrained("gpt2")
-
-encoded = tokenizer("Hey!", return_tensors="pt")
-result = model(**encoded)
-```
-
-<Question
-	choices={[
-		{
-			text: "No, it seems correct.",
-			explain: "Unfortunately, coupling a model with a tokenizer that was trained with a different checkpoint is rarely a good idea. The model was not trained to make sense out of this tokenizer's output, so the model output (if it can even run!) will not make any sense."
-		},
-		{
-			text: "The tokenizer and model should always be from the same checkpoint.",
-			explain: "Right!",
-            correct: true
-		},
-		{
-			text: "It's good practice to pad and truncate with the tokenizer as every input is a batch.",
-			explain: "It's true that every model input needs to be a batch. However, truncating or padding this sequence wouldn't necessarily make sense as there is only one of it, and those are techniques to batch together a list of sentences."
-		}
-	]}
-/>
+vLLM uses PagedAttention for optimal memory management:
+
+```python
+from vllm.engine.arg_utils import AsyncEngineArgs
 
+engine_args = AsyncEngineArgs(
+    model="HuggingFaceTB/SmolLM2-1.7B-Instruct",
+    gpu_memory_utilization=0.85,
+    max_num_batched_tokens=8192,
+    block_size=16,
+)
+
+llm = LLM(engine_args=engine_args)
+```
 {/if}
+
+## Resources
+
+- [Text Generation Inference Documentation](https://huggingface.co/docs/text-generation-inference)
+- [TGI GitHub Repository](https://github.com/huggingface/text-generation-inference)
+- [vLLM Documentation](https://vllm.readthedocs.io/)
+- [vLLM GitHub Repository](https://github.com/vllm-project/vllm)
+- [PagedAttention Paper](https://arxiv.org/abs/2309.06180)
+- [HuggingFace Blog: TGI in Production](https://huggingface.co/blog/tgi-serving)
+- [vLLM Performance Benchmarks](https://vllm.readthedocs.io/en/latest/performance.html)
path: chapters/en/chapter2/8.mdx
user: login: sergiopaniego
body: 404
created_at: 2025-05-14T13:40:45Z
author_association: MEMBER
diff_hunk: @@ -1,310 +1,779 @@
-<FrameworkSwitchCourse {fw} />
-
-<!-- DISABLE-FRONTMATTER-SECTIONS -->
-
-# End-of-chapter quiz[[end-of-chapter-quiz]]
-
-<CourseFloatingBanner
-    chapter={2}
-    classNames="absolute z-10 right-0 top-0"
-/>
-
-### 1. What is the order of the language modeling pipeline?
-
-<Question
-	choices={[
-		{
-			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
-			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
-		},
-		{
-			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
-			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
-		},
-		{
-			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
-			explain: "Correct! The tokenizer can be used for both tokenizing and de-tokenizing.",
-            correct: true
-		}
-	]}
-/>
-
-### 2. How many dimensions does the tensor output by the base Transformer model have, and what are they?
-
-<Question
-	choices={[
-		{
-			text: "2: The sequence length and the batch size",
-			explain: "False! The tensor output by the model has a third dimension: hidden size."
-		},
-		{
-			text: "2: The sequence length and the hidden size",
-			explain: "False! All Transformer models handle batches, even with a single sequence; that would be a batch size of 1!"
-		},
-		{
-			text: "3: The sequence length, the batch size, and the hidden size",
-			explain: "Correct!",
-            correct: true
-		}
-	]}
-/>
-
-### 3. Which of the following is an example of subword tokenization?
-
-<Question
-	choices={[
-		{
-			text: "WordPiece",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-		},
-		{
-			text: "Character-based tokenization",
-			explain: "Character-based tokenization is not a type of subword tokenization."
-		},
-		{
-			text: "Splitting on whitespace and punctuation",
-			explain: "That's a word-based tokenization scheme!"
-		},
-		{
-			text: "BPE",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-        },
-		{
-			text: "Unigram",
-			explain: "Yes, that's one example of subword tokenization!",
-            correct: true
-        },
-		{
-			text: "None of the above",
-			explain: "Incorrect!"
-        }
-	]}
-/>
-
-### 4. What is a model head?
-
-<Question
-	choices={[
-		{
-			text: "A component of the base Transformer network that redirects tensors to their correct layers",
-			explain: "Incorrect! There's no such component."
-		},
-		{
-			text: "Also known as the self-attention mechanism, it adapts the representation of a token according to the other tokens of the sequence",
-			explain: "Incorrect! The self-attention layer does contain attention \"heads,\" but these are not adaptation heads."
-		},
-		{
-			text: "An additional component, usually made up of one or a few layers, to convert the transformer predictions to a task-specific output",
-			explain: "That's right. Adaptation heads, also known simply as heads, come up in different forms: language modeling heads, question answering heads, sequence classification heads... ",
-			correct: true
-		} 
-	]}
-/>
-
-{#if fw === 'pt'}
-### 5. What is an AutoModel?
-
-<Question
-	choices={[
-		{
-			text: "A model that automatically trains on your data",
-			explain: "Incorrect. Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
-		},
-		{
-			text: "An object that returns the correct architecture based on the checkpoint",
-			explain: "Exactly: the <code>AutoModel</code> only needs to know the checkpoint from which to initialize to return the correct architecture.",
-			correct: true
-		},
-		{
-			text: "A model that automatically detects the language used for its inputs to load the correct weights",
-			explain: "Incorrect; while some checkpoints and models are capable of handling multiple languages, there are no built-in tools for automatic checkpoint selection according to language. You should head over to the <a href='https://huggingface.co/models'>Model Hub</a> to find the best checkpoint for your task!"
-		} 
-	]}
-/>
+# Optimized Inference Deployment
 
+In this section, we'll explore advanced frameworks for optimizing LLM deployments: Text Generation Inference (TGI), vLLM, and llama.cpp. These applications are primarily used in production environments to serve LLMs to users. This section focuses on how to deploy these frameworks in production rather than how to use them for inference on a single machine.
+
+We'll cover how these tools maximize inference efficiency and simplify production deployments of Large Language Models.
+
+## Framework Selection Guide
+
+TGI, vLLM, and llama.cpp serve similar purposes but have distinct characteristics that make them better suited for different use cases. Let's look at the key differences between them, focusing on performance and integration.
+
+### Memory Management and Performance
+
+**TGI** is designed to be stable and predictable in production, using fixed sequence lengths to keep memory usage consistent. TGI manages memory using Flash Attention 2 and continuous batching techniques. This means it can process attention calculations very efficiently and keep the GPU busy by constantly feeding it work. The system can move parts of the model between CPU and GPU when needed, which helps handle larger models. 
+
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/tgi/flash-attn.png" alt="Flash Attention" />
+
+<Tip title="How Flash Attention Works">
+Flash Attention is a technique that optimizes the attention mechanism in transformer models by addressing memory bandwidth bottlenecks. As discussed earlier in [section 12.3](2.mdx), the attention mechanism has quadratic complexity and memory usage, making it inefficient for long sequences.
+
+The key innovation is in how it manages memory transfers between High Bandwidth Memory (HBM) and faster SRAM cache. Traditional attention repeatedly transfers data between HBM and SRAM, creating bottlenecks by leaving the GPU idle. Flash Attention loads data once into SRAM and performs all calculations there, minimizing expensive memory transfers. 
+
+While the benefits are most significant during training, Flash Attention's reduced VRAM usage and improved efficiency make it valuable for inference as well, enabling faster and more scalable LLM serving.
+</Tip>
+
+**vLLM** takes a different approach by using PagedAttention. Just like how a computer manages its memory in pages, vLLM splits the model's memory into smaller blocks. This clever system means it can handle different-sized requests more flexibly and doesn't waste memory space. It's particularly good at sharing memory between different requests and reduces memory fragmentation, which makes the whole system more efficient.
+
+<Tip title="How Paged Attention Works">
+Paged Attention is a technique that addresses another critical bottleneck in LLM inference: KV cache memory management. As discussed in [section 12.3](2.mdx), during text generation, the model stores attention keys and values (KV cache) for each generated token to reduce redundant computations. The KV cache can become enormous, especially with long sequences or multiple concurrent requests.
+
+vLLM's key innovation lies in how it manages this cache:
+
+1. **Memory Paging**: Instead of treating the KV cache as one large block, it's divided into fixed-size "pages" (similar to virtual memory in operating systems).
+2. **Non-contiguous Storage**: Pages don't need to be stored contiguously in GPU memory, allowing for more flexible memory allocation.
+3. **Page Table Management**: A page table tracks which pages belong to which sequence, enabling efficient lookup and access.
+4. **Memory Sharing**: For operations like parallel sampling, pages storing the KV cache for the prompt can be shared across multiple sequences.
+
+The PagedAttention approach can lead to up to 24x higher throughput compared to traditional methods, making it a game-changer for production LLM deployments. If you want to go really deep into how PagedAttention works, you can read the [the guide from the vLLM documentation](https://docs.vllm.ai/en/latest/design/kernel/paged_attention.html).
+</Tip>
+
+**llama.cpp** is a highly optimized C/C++ implementation originally designed for running LLaMA models on consumer hardware. It focuses on CPU efficiency with optional GPU acceleration and is ideal for resource-constrained environments. llama.cpp uses quantization techniques to reduce model size and memory requirements while maintaining good performance. It implements optimized kernels for various CPU architectures and supports basic KV cache management for efficient token generation.
+
+<Tip title="How llama.cpp Quantization Works">
+Quantization in llama.cpp reduces the precision of model weights from 32-bit or 16-bit floating point to lower precision formats like 8-bit integers (INT8), 4-bit, or even lower. This significantly reduces memory usage and improves inference speed with minimal quality loss.
+
+Key quantization features in llama.cpp include:
+1. **Multiple Quantization Levels**: Supports 8-bit, 4-bit, 3-bit, and even 2-bit quantization
+2. **GGML/GGUF Format**: Uses custom tensor formats optimized for quantized inference
+3. **Mixed Precision**: Can apply different quantization levels to different parts of the model
+4. **Hardware-Specific Optimizations**: Includes optimized code paths for various CPU architectures (AVX2, AVX-512, NEON)
+
+This approach enables running billion-parameter models on consumer hardware with limited memory, making it perfect for local deployments and edge devices.
+</Tip>
+
+
+
+### Deployment and Integration
+
+Let's move on to the deployment and integration differences between the frameworks.
+
+**TGI** excels in enterprise-level deployment with its production-ready features. It comes with built-in Kubernetes support and includes everything you need for running in production, like monitoring through Prometheus and Grafana, automatic scaling, and comprehensive safety features. The system also includes enterprise-grade logging and various protective measures like content filtering and rate limiting to keep your deployment secure and stable.
+
+**vLLM** takes a more flexible, developer-friendly approach to deployment. It's built with Python at its core and can easily replace OpenAI's API in your existing applications. The framework focuses on delivering raw performance and can be customized to fit your specific needs. It works particularly well with Ray for managing clusters, making it a great choice when you need high performance and adaptability.
+
+**llama.cpp** prioritizes simplicity and portability. Its server implementation is lightweight and can run on a wide range of hardware, from powerful servers to consumer laptops and even some high-end mobile devices. With minimal dependencies and a simple C/C++ core, it's easy to deploy in environments where installing Python frameworks would be challenging. The server provides an OpenAI-compatible API while maintaining a much smaller resource footprint than other solutions.
+
+
+
+## Getting Started
+
+Let's explore how to use these frameworks for deploying LLMs, starting with installation and basic setup.
+
+### Installation and Basic Setup
+
+{#if framework === 'tgi'}
+
+TGI is easy to install and use, with deep integration into the Hugging Face ecosystem.
+
+First, launch the TGI server using Docker:
+
+```sh
+docker run --gpus all \
+    --shm-size 1g \
+    -p 8080:80 \
+    -v ~/.cache/huggingface:/data \
+    ghcr.io/huggingface/text-generation-inference:latest \
+    --model-id HuggingFaceTB/SmolLM2-360M-Instruct
+```
+
+Then interact with it using Hugging Face's InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+# Initialize client pointing to TGI endpoint
+client = InferenceClient(
+    model="http://localhost:8080",  # URL to the TGI server
+)
+
+# Text generation
+response = client.text_generation(
+    "Tell me a story",
+    max_new_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+    details=True,
+    stop_sequences=[],
+)
+print(response.generated_text)
+
+# For chat format
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
+
+Alternatively, you can use the OpenAI client:
+
+```python
+from openai import OpenAI
+
+# Initialize client pointing to TGI endpoint
+client = OpenAI(
+    base_url="http://localhost:8080/v1",  # Make sure to include /v1
+    api_key="not-needed",  # TGI doesn't require an API key by default
+)
+
+# Chat completion
+response = client.chat.completions.create(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
+{:else if framework === 'llama.cpp'}
+
+llama.cpp is easy to install and use, requiring minimal dependencies and supporting both CPU and GPU inference.
+
+First, install and build llama.cpp:
+
+```sh
+# Clone the repository
+git clone https://github.com/ggerganov/llama.cpp
+cd llama.cpp
+
+# Build the project
+make
+
+# Download the SmolLM2-1.7B-Instruct-GGUF model
+curl -L -O https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct-GGUF/resolve/main/smollm2-1.7b-instruct.Q4_K_M.gguf
+```
+
+Then, launch the server (with OpenAI API compatibility):
+
+```sh
+# Start the server
+./server \
+    -m smollm2-1.7b-instruct.Q4_K_M.gguf \
+    --host 0.0.0.0 \
+    --port 8080 \
+    -c 4096 \
+    --n-gpu-layers 0  # Set to a higher number to use GPU
+```
+
+Interact with the server using Hugging Face's InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+# Initialize client pointing to llama.cpp server
+client = InferenceClient(
+    model="http://localhost:8080/v1",  # URL to the llama.cpp server
+    token="sk-no-key-required",  # llama.cpp server requires this placeholder
+)
+
+# Text generation
+response = client.text_generation(
+    "Tell me a story",
+    max_new_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+    details=True,
+)
+print(response.generated_text)
+
+# For chat format
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
+
+Alternatively, you can use the OpenAI client:
+
+```python
+from openai import OpenAI
+
+# Initialize client pointing to llama.cpp server
+client = OpenAI(
+    base_url="http://localhost:8080/v1",
+    api_key="sk-no-key-required",  # llama.cpp server requires this placeholder
+)
+
+# Chat completion
+response = client.chat.completions.create(
+    model="smollm2-1.7b-instruct",  # Model identifier can be anything as server only loads one model
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
 {:else}
-### 5. What is an TFAutoModel?
-
-<Question
-	choices={[
-		{
-			text: "A model that automatically trains on your data",
-			explain: "Incorrect. Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
-		},
-		{
-			text: "An object that returns the correct architecture based on the checkpoint",
-			explain: "Exactly: the <code>TFAutoModel</code> only needs to know the checkpoint from which to initialize to return the correct architecture.",
-			correct: true
-		},
-		{
-			text: "A model that automatically detects the language used for its inputs to load the correct weights",
-			explain: "Incorrect; while some checkpoints and models are capable of handling multiple languages, there are no built-in tools for automatic checkpoint selection according to language. You should head over to the <a href='https://huggingface.co/models'>Model Hub</a> to find the best checkpoint for your task!"
-		} 
-	]}
-/>
 
+vLLM is easy to install and use, with both OpenAI API compatibility and a native Python interface.
+
+First, launch the vLLM OpenAI-compatible server:
+
+```sh
+python -m vllm.entrypoints.openai.api_server \
+    --model HuggingFaceTB/SmolLM2-360M-Instruct \
+    --host 0.0.0.0 \
+    --port 8000
+```
+
+Then interact with it using Hugging Face's InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+# Initialize client pointing to vLLM endpoint
+client = InferenceClient(
+    model="http://localhost:8000/v1",  # URL to the vLLM server
+)
+
+# Text generation
+response = client.text_generation(
+    "Tell me a story",
+    max_new_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+    details=True,
+)
+print(response.generated_text)
+
+# For chat format
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
+
+Alternatively, you can use the OpenAI client:
+
+```python
+from openai import OpenAI
+
+# Initialize client pointing to vLLM endpoint
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="not-needed",  # vLLM doesn't require an API key by default
+)
+
+# Chat completion
+response = client.chat.completions.create(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Tell me a story"},
+    ],
+    max_tokens=100,
+    temperature=0.7,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+```
 {/if}
 
-### 6. What are the techniques to be aware of when batching sequences of different lengths together?
-
-<Question
-	choices={[
-		{
-			text: "Truncating",
-			explain: "Yes, truncation is a correct way of evening out sequences so that they fit in a rectangular shape. Is it the only one, though?",
-			correct: true
-		},
-		{
-			text: "Returning tensors",
-			explain: "While the other techniques allow you to return rectangular tensors, returning tensors isn't helpful when batching sequences together."
-		},
-		{
-			text: "Padding",
-			explain: "Yes, padding is a correct way of evening out sequences so that they fit in a rectangular shape. Is it the only one, though?",
-			correct: true
-		}, 
-		{
-			text: "Attention masking",
-			explain: "Absolutely! Attention masks are of prime importance when handling sequences of different lengths. That's not the only technique to be aware of, however.",
-			correct: true
-		} 
-	]}
-/>
-
-### 7. What is the point of applying a SoftMax function to the logits output by a sequence classification model?
-
-<Question
-	choices={[
-		{
-			text: "It softens the logits so that they're more reliable.",
-			explain: "No, the SoftMax function does not affect the reliability of results."
-		},
-		{
-			text: "It applies a lower and upper bound so that they're understandable.",
-			explain: "Correct! The resulting values are bound between 0 and 1. That's not the only reason we use a SoftMax function, though.",
-            correct: true
-		},
-		{
-			text: "The total sum of the output is then 1, resulting in a possible probabilistic interpretation.",
-			explain: "Correct! That's not the only reason we use a SoftMax function, though.",
-            correct: true
-		}
-	]}
-/>
-
-### 8. What method is most of the tokenizer API centered around?
-
-<Question
-	choices={[
-		{
-			text: "<code>encode</code>, as it can encode text into IDs and IDs into predictions",
-			explain: "Wrong! While the <code>encode</code> method does exist on tokenizers, it does not exist on models."
-		},
-		{
-			text: "Calling the tokenizer object directly.",
-			explain: "Exactly! The <code>__call__</code> method of the tokenizer is a very powerful method which can handle pretty much anything. It is also the method used to retrieve predictions from a model.",
-			correct: true
-		},
-		{
-			text: "<code>pad</code>",
-			explain: "Wrong! Padding is very useful, but it's just one part of the tokenizer API."
-		},
-		{
-			text: "<code>tokenize</code>",
-			explain: "The <code>tokenize</code> method is arguably one of the most useful methods, but it isn't the core of the tokenizer API."
-		}
-	]}
-/>
-
-### 9. What does the `result` variable contain in this code sample?
-
-```py
-from transformers import AutoTokenizer
-
-tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
-result = tokenizer.tokenize("Hello!")
-```
-
-<Question
-	choices={[
-		{
-			text: "A list of strings, each string being a token",
-			explain: "Absolutely! Convert this to IDs, and send them to a model!",
-            correct: true
-		},
-		{
-			text: "A list of IDs",
-			explain: "Incorrect; that's what the <code>__call__</code> or <code>convert_tokens_to_ids</code> method is for!"
-		},
-		{
-			text: "A string containing all of the tokens",
-			explain: "This would be suboptimal, as the goal is to split the string into multiple tokens."
-		}
-	]}
-/>
-
-{#if fw === 'pt'}
-### 10. Is there something wrong with the following code?
-
-```py
-from transformers import AutoTokenizer, AutoModel
-
-tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
-model = AutoModel.from_pretrained("gpt2")
-
-encoded = tokenizer("Hey!", return_tensors="pt")
-result = model(**encoded)
-```
-
-<Question
-	choices={[
-		{
-			text: "No, it seems correct.",
-			explain: "Unfortunately, coupling a model with a tokenizer that was trained with a different checkpoint is rarely a good idea. The model was not trained to make sense out of this tokenizer's output, so the model output (if it can even run!) will not make any sense."
-		},
-		{
-			text: "The tokenizer and model should always be from the same checkpoint.",
-			explain: "Right!",
-            correct: true
-		},
-		{
-			text: "It's good practice to pad and truncate with the tokenizer as every input is a batch.",
-			explain: "It's true that every model input needs to be a batch. However, truncating or padding this sequence wouldn't necessarily make sense as there is only one of it, and those are techniques to batch together a list of sentences."
-		}
-	]}
-/>
+### Basic Text Generation
+
+Let's look at examples of text generation with the frameworks:
+
+{#if framework === 'tgi'}
 
+First, deploy TGI with advanced parameters:
+```sh
+docker run --gpus all \
+    --shm-size 1g \
+    -p 8080:80 \
+    -v ~/.cache/huggingface:/data \
+    ghcr.io/huggingface/text-generation-inference:latest \
+    --model-id HuggingFaceTB/SmolLM2-360M-Instruct \
+    --max-total-tokens 4096 \
+    --max-input-length 3072 \
+    --max-batch-total-tokens 8192 \
+    --waiting-served-ratio 1.2
+```
+
+Use the InferenceClient for flexible text generation:
+```python
+from huggingface_hub import InferenceClient
+
+client = InferenceClient(model="http://localhost:8080")
+
+# Advanced parameters example
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,
+    max_tokens=200,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+
+# Raw text generation
+response = client.text_generation(
+    "Write a creative story about space exploration",
+    max_new_tokens=200,
+    temperature=0.8,
+    top_p=0.95,
+    repetition_penalty=1.1,
+    do_sample=True,
+    details=True,
+)
+print(response.generated_text)
+```
+
+Or use the OpenAI client:
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
+
+# Advanced parameters example
+response = client.chat.completions.create(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,  # Higher for more creativity
+)
+print(response.choices[0].message.content)
+```
+{:else if framework === 'llama.cpp'}
+
+For llama.cpp, you can set advanced parameters when launching the server:
+
+```sh
+./server \
+    -m smollm2-1.7b-instruct.Q4_K_M.gguf \
+    --host 0.0.0.0 \
+    --port 8080 \
+    -c 4096 \            # Context size
+    --threads 8 \        # CPU threads to use
+    --batch-size 512 \   # Batch size for prompt evaluation
+    --n-gpu-layers 0     # GPU layers (0 = CPU only)
+```
+
+Use the InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+client = InferenceClient(model="http://localhost:8080/v1", token="sk-no-key-required")
+
+# Advanced parameters example
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,
+    max_tokens=200,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+
+# For direct text generation
+response = client.text_generation(
+    "Write a creative story about space exploration",
+    max_new_tokens=200,
+    temperature=0.8,
+    top_p=0.95,
+    repetition_penalty=1.1,
+    details=True,
+)
+print(response.generated_text)
+```
+
+Or use the OpenAI client for generation with control over the sampling parameters:
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8080/v1", api_key="sk-no-key-required")
+
+# Advanced parameters example
+response = client.chat.completions.create(
+    model="smollm2-1.7b-instruct",
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Nucleus sampling probability
+    frequency_penalty=0.5,  # Reduce repetition of frequent tokens
+    presence_penalty=0.5,  # Reduce repetition by penalizing tokens already present
+    max_tokens=200,  # Maximum generation length
+)
+print(response.choices[0].message.content)
+```
+
+You can also use llama.cpp's native library for even more control:
+
+```python
+# Using llama-cpp-python package for direct model access
+from llama_cpp import Llama
+
+# Load the model
+llm = Llama(
+    model_path="smollm2-1.7b-instruct.Q4_K_M.gguf",
+    n_ctx=4096,  # Context window size
+    n_threads=8,  # CPU threads
+    n_gpu_layers=0,  # GPU layers (0 = CPU only)
+)
+
+# Format prompt according to the model's expected format
+prompt = """
system
+You are a creative storyteller.
+

+
user
+Write a creative story
+

+
assistant
+"""
+
+# Generate response with precise parameter control
+output = llm(
+    prompt,
+    max_tokens=200,
+    temperature=0.8,
+    top_p=0.95,
+    frequency_penalty=0.5,
+    presence_penalty=0.5,
+    stop=["
"],
+)
+
+print(output["choices"][0]["text"])
+```
+{:else}
+
+For advanced usage with vLLM, you can use the InferenceClient:
+
+```python
+from huggingface_hub import InferenceClient
+
+client = InferenceClient(model="http://localhost:8000/v1")
+
+# Advanced parameters example
+response = client.chat_completion(
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,
+    max_tokens=200,
+    top_p=0.95,
+)
+print(response.choices[0].message.content)
+
+# For direct text generation
+response = client.text_generation(
+    "Write a creative story about space exploration",
+    max_new_tokens=200,
+    temperature=0.8,
+    top_p=0.95,
+    details=True,
+)
+print(response.generated_text)
+```
+
+You can also use the OpenAI client:
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
+
+# Advanced parameters example
+response = client.chat.completions.create(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    messages=[
+        {"role": "system", "content": "You are a creative storyteller."},
+        {"role": "user", "content": "Write a creative story"},
+    ],
+    temperature=0.8,
+    top_p=0.95,
+    max_tokens=200,
+)
+print(response.choices[0].message.content)
+```
+
+vLLM also provides a native Python interface with fine-grained control:
+
+```python
+from vllm import LLM, SamplingParams
+
+# Initialize the model with advanced parameters
+llm = LLM(
+    model="HuggingFaceTB/SmolLM2-360M-Instruct",
+    gpu_memory_utilization=0.85,
+    max_num_batched_tokens=8192,
+    max_num_seqs=256,
+    block_size=16,
+)
+
+# Configure sampling parameters
+sampling_params = SamplingParams(
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Consider top 95% probability mass
+    max_tokens=100,  # Maximum length
+    presence_penalty=1.1,  # Reduce repetition
+    frequency_penalty=1.1,  # Reduce repetition
+    stop=["\n\n", "###"],  # Stop sequences
+)
+
+# Generate text
+prompt = "Write a creative story"
+outputs = llm.generate(prompt, sampling_params)
+print(outputs[0].outputs[0].text)
+
+# For chat-style interactions
+chat_prompt = [
+    {"role": "system", "content": "You are a creative storyteller."},
+    {"role": "user", "content": "Write a creative story"},
+]
+formatted_prompt = llm.get_chat_template()(chat_prompt)  # Uses model's chat template
+outputs = llm.generate(formatted_prompt, sampling_params)
+print(outputs[0].outputs[0].text)
+```
+{/if}
+
+## Advanced Generation Control
+
+### Token Selection and Sampling
+
+The process of generating text involves selecting the next token at each step. This selection process can be controlled through various parameters:
+
+1. **Raw Logits**: The initial output probabilities for each token
+2. **Temperature**: Controls randomness in selection (higher = more creative)
+3. **Top-p (Nucleus) Sampling**: Filters to top tokens making up X% of probability mass
+4. **Top-k Filtering**: Limits selection to k most likely tokens
+
+Here's how to configure these parameters:
+
+{#if framework === 'tgi'}
+```python
+client.generate(
+    "Write a creative story",
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Consider top 95% probability mass
+    top_k=50,  # Consider top 50 tokens
+    max_new_tokens=100,  # Maximum length
+    repetition_penalty=1.1,  # Reduce repetition
+)
+```
+{:else if framework === 'llama.cpp'}
+```python
+# Via OpenAI API compatibility
+response = client.completions.create(
+    model="smollm2-1.7b-instruct",  # Model name (can be any string for llama.cpp server)
+    prompt="Write a creative story",
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Consider top 95% probability mass
+    frequency_penalty=1.1,  # Reduce repetition
+    presence_penalty=0.1,  # Reduce repetition
+    max_tokens=100,  # Maximum length
+)
+
+# Via llama-cpp-python direct access
+output = llm(
+    "Write a creative story",
+    temperature=0.8,
+    top_p=0.95,
+    top_k=50,
+    max_tokens=100,
+    repeat_penalty=1.1,
+)
+```
+{:else}
+```python
+params = SamplingParams(
+    temperature=0.8,  # Higher for more creativity
+    top_p=0.95,  # Consider top 95% probability mass
+    top_k=50,  # Consider top 50 tokens
+    max_tokens=100,  # Maximum length
+    presence_penalty=0.1,  # Reduce repetition
+)
+llm.generate("Write a creative story", sampling_params=params)
+```
+{/if}
+
+### Controlling Repetition
+
+Both frameworks provide ways to prevent repetitive text generation:
+
+{#if framework === 'tgi'}
+```python
+client.generate(
+    "Write a varied text",
+    repetition_penalty=1.1,  # Penalize repeated tokens
+    no_repeat_ngram_size=3,  # Prevent 3-gram repetition
+)
+```
+{:else if framework === 'llama.cpp'}
+```python
+# Via OpenAI API
+response = client.completions.create(
+    model="smollm2-1.7b-instruct",
+    prompt="Write a varied text",
+    frequency_penalty=1.1,  # Penalize frequent tokens
+    presence_penalty=0.8,  # Penalize tokens already present
+)
+
+# Via direct library
+output = llm(
+    "Write a varied text",
+    repeat_penalty=1.1,  # Penalize repeated tokens
+    frequency_penalty=0.5,  # Additional frequency penalty
+    presence_penalty=0.5,  # Additional presence penalty
+)
+```
+{:else}
+```python
+params = SamplingParams(
+    presence_penalty=0.1,  # Penalize token presence
+    frequency_penalty=0.1,  # Penalize token frequency
+)
+```
+{/if}
+
+### Length Control and Stop Sequences
+
+You can control generation length and specify when to stop:
+
+{#if framework === 'tgi'}
+```python
+client.generate(
+    "Generate a short paragraph",
+    max_new_tokens=100,
+    min_new_tokens=10,
+    stop_sequences=["\n\n", "###"],
+)
+```
+{:else if framework === 'llama.cpp'}
+```python
+# Via OpenAI API
+response = client.completions.create(
+    model="smollm2-1.7b-instruct",
+    prompt="Generate a short paragraph",
+    max_tokens=100,
+    stop=["\n\n", "###"],
+)
+
+# Via direct library
+output = llm("Generate a short paragraph", max_tokens=100, stop=["\n\n", "###"])
+```
+{:else}
+```python
+params = SamplingParams(
+    max_tokens=100,
+    min_tokens=10,
+    stop=["###", "\n\n"],
+    ignore_eos=False,
+    skip_special_tokens=True,
+)
+```
+{/if}
+
+## Memory Management
+
+Both frameworks implement advanced memory management techniques for efficient inference.
+
+{#if framework === 'tgi'}
+TGI uses Flash Attention 2 and continuous batching:
+
+```sh
+# Docker deployment with memory optimization
+docker run --gpus all -p 8080:80 \
+    --shm-size 1g \
+    ghcr.io/huggingface/text-generation-inference:latest \
+    --model-id HuggingFaceTB/SmolLM2-1.7B-Instruct \
+    --max-batch-total-tokens 8192 \
+    --max-input-length 4096
+```
+{:else if framework === 'llama.cpp'}
+llama.cpp uses quantization and optimized memory layout:
+
+```sh
+# Server with memory optimizations
+./server \
+    -m smollm2-1.7b-instruct.Q4_K_M.gguf \
+    --host 0.0.0.0 \
+    --port 8080 \
+    -c 2048 \               # Context size
+    --threads 4 \           # CPU threads
+    --n-gpu-layers 32 \     # Use more GPU layers for larger models
+    --mlock \               # Lock memory to prevent swapping
+    --cont-batching         # Enable continuous batching
+```
+
+For models too large for your GPU, you can use CPU offloading:
+
+```sh
+./server \
+    -m smollm2-1.7b-instruct.Q4_K_M.gguf \
+    --n-gpu-layers 20 \     # Keep first 20 layers on GPU
+    --threads 8             # Use more CPU threads for CPU layers
+```
 {:else}
-### 10. Is there something wrong with the following code?
-
-```py
-from transformers import AutoTokenizer, TFAutoModel
-
-tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
-model = TFAutoModel.from_pretrained("gpt2")
-
-encoded = tokenizer("Hey!", return_tensors="pt")
-result = model(**encoded)
-```
-
-<Question
-	choices={[
-		{
-			text: "No, it seems correct.",
-			explain: "Unfortunately, coupling a model with a tokenizer that was trained with a different checkpoint is rarely a good idea. The model was not trained to make sense out of this tokenizer's output, so the model output (if it can even run!) will not make any sense."
-		},
-		{
-			text: "The tokenizer and model should always be from the same checkpoint.",
-			explain: "Right!",
-            correct: true
-		},
-		{
-			text: "It's good practice to pad and truncate with the tokenizer as every input is a batch.",
-			explain: "It's true that every model input needs to be a batch. However, truncating or padding this sequence wouldn't necessarily make sense as there is only one of it, and those are techniques to batch together a list of sentences."
-		}
-	]}
-/>
+vLLM uses PagedAttention for optimal memory management:
+
+```python
+from vllm.engine.arg_utils import AsyncEngineArgs
 
+engine_args = AsyncEngineArgs(
+    model="HuggingFaceTB/SmolLM2-1.7B-Instruct",
+    gpu_memory_utilization=0.85,
+    max_num_batched_tokens=8192,
+    block_size=16,
+)
+
+llm = LLM(engine_args=engine_args)
+```
 {/if}
+
+## Resources
+
+- [Text Generation Inference Documentation](https://huggingface.co/docs/text-generation-inference)
+- [TGI GitHub Repository](https://github.com/huggingface/text-generation-inference)
+- [vLLM Documentation](https://vllm.readthedocs.io/)
+- [vLLM GitHub Repository](https://github.com/vllm-project/vllm)
+- [PagedAttention Paper](https://arxiv.org/abs/2309.06180)
+- [HuggingFace Blog: TGI in Production](https://huggingface.co/blog/tgi-serving)
+- [vLLM Performance Benchmarks](https://vllm.readthedocs.io/en/latest/performance.html)
+- [llama.cpp GitHub Repository](https://github.com/ggerganov/llama.cpp)
+- [llama.cpp Server Documentation](https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md)
path: chapters/en/chapter2/8.mdx
user: login: sergiopaniego
body: 404
created_at: 2025-05-14T13:42:08Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,252 @@
+<FrameworkSwitchCourse {fw} />
+
+<!-- DISABLE-FRONTMATTER-SECTIONS -->
+
+# End-of-chapter quiz[[end-of-chapter-quiz]]
+
+<CourseFloatingBanner
+    chapter={2}
+    classNames="absolute z-10 right-0 top-0"
+/>
+
+### 1. What is the order of the language modeling pipeline?
+
+<Question
+	choices={[
+		{
+			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
+			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
+		},
+		{
+			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
+			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
+		},
+		{
+			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
+			explain: "Correct! The tokenizer can be used for both tokenizing and de-tokenizing.",
path: chapters/en/chapter2/9.mdx
user: login: sergiopaniego
body: Duplicated
```suggestion
			explain: "The tokenizer can be used for both tokenizing and de-tokenizing.",
```
created_at: 2025-05-14T13:43:32Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,252 @@
+<FrameworkSwitchCourse {fw} />
+
+<!-- DISABLE-FRONTMATTER-SECTIONS -->
+
+# End-of-chapter quiz[[end-of-chapter-quiz]]
+
+<CourseFloatingBanner
+    chapter={2}
+    classNames="absolute z-10 right-0 top-0"
+/>
+
+### 1. What is the order of the language modeling pipeline?
+
+<Question
+	choices={[
+		{
+			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
+			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
+		},
+		{
+			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
+			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
+		},
+		{
+			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
+			explain: "Correct! The tokenizer can be used for both tokenizing and de-tokenizing.",
+            correct: true
+		}
+	]}
+/>
+
+### 2. How many dimensions does the tensor output by the base Transformer model have, and what are they?
+
+<Question
+	choices={[
+		{
+			text: "2: The sequence length and the batch size",
+			explain: "False! The tensor output by the model has a third dimension: hidden size."
+		},
+		{
+			text: "2: The sequence length and the hidden size",
+			explain: "False! All Transformer models handle batches, even with a single sequence; that would be a batch size of 1!"
+		},
+		{
+			text: "3: The sequence length, the batch size, and the hidden size",
+			explain: "Correct!",
path: chapters/en/chapter2/9.mdx
user: login: sergiopaniego
body: Duplicated
```suggestion
			explain: "Nicely done!",
```
created_at: 2025-05-14T13:44:37Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,252 @@
+<FrameworkSwitchCourse {fw} />
+
+<!-- DISABLE-FRONTMATTER-SECTIONS -->
+
+# End-of-chapter quiz[[end-of-chapter-quiz]]
+
+<CourseFloatingBanner
+    chapter={2}
+    classNames="absolute z-10 right-0 top-0"
+/>
+
+### 1. What is the order of the language modeling pipeline?
+
+<Question
+	choices={[
+		{
+			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
+			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
+		},
+		{
+			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
+			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
+		},
+		{
+			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
+			explain: "Correct! The tokenizer can be used for both tokenizing and de-tokenizing.",
+            correct: true
+		}
+	]}
+/>
+
+### 2. How many dimensions does the tensor output by the base Transformer model have, and what are they?
+
+<Question
+	choices={[
+		{
+			text: "2: The sequence length and the batch size",
+			explain: "False! The tensor output by the model has a third dimension: hidden size."
+		},
+		{
+			text: "2: The sequence length and the hidden size",
+			explain: "False! All Transformer models handle batches, even with a single sequence; that would be a batch size of 1!"
+		},
+		{
+			text: "3: The sequence length, the batch size, and the hidden size",
+			explain: "Correct!",
+            correct: true
+		}
+	]}
+/>
+
+### 3. Which of the following is an example of subword tokenization?
+
+<Question
+	choices={[
+		{
+			text: "WordPiece",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+		},
+		{
+			text: "Character-based tokenization",
+			explain: "Character-based tokenization is not a type of subword tokenization."
+		},
+		{
+			text: "Splitting on whitespace and punctuation",
+			explain: "That's a word-based tokenization scheme!"
+		},
+		{
+			text: "BPE",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+        },
+		{
+			text: "Unigram",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+        },
+		{
+			text: "None of the above",
+			explain: "Incorrect!"
path: chapters/en/chapter2/9.mdx
user: login: sergiopaniego
body: ```suggestion
			explain: "Wrong!"
```
created_at: 2025-05-14T13:44:58Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,252 @@
+<FrameworkSwitchCourse {fw} />
+
+<!-- DISABLE-FRONTMATTER-SECTIONS -->
+
+# End-of-chapter quiz[[end-of-chapter-quiz]]
+
+<CourseFloatingBanner
+    chapter={2}
+    classNames="absolute z-10 right-0 top-0"
+/>
+
+### 1. What is the order of the language modeling pipeline?
+
+<Question
+	choices={[
+		{
+			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
+			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
+		},
+		{
+			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
+			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
+		},
+		{
+			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
+			explain: "Correct! The tokenizer can be used for both tokenizing and de-tokenizing.",
+            correct: true
+		}
+	]}
+/>
+
+### 2. How many dimensions does the tensor output by the base Transformer model have, and what are they?
+
+<Question
+	choices={[
+		{
+			text: "2: The sequence length and the batch size",
+			explain: "False! The tensor output by the model has a third dimension: hidden size."
+		},
+		{
+			text: "2: The sequence length and the hidden size",
+			explain: "False! All Transformer models handle batches, even with a single sequence; that would be a batch size of 1!"
+		},
+		{
+			text: "3: The sequence length, the batch size, and the hidden size",
+			explain: "Correct!",
+            correct: true
+		}
+	]}
+/>
+
+### 3. Which of the following is an example of subword tokenization?
+
+<Question
+	choices={[
+		{
+			text: "WordPiece",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+		},
+		{
+			text: "Character-based tokenization",
+			explain: "Character-based tokenization is not a type of subword tokenization."
+		},
+		{
+			text: "Splitting on whitespace and punctuation",
+			explain: "That's a word-based tokenization scheme!"
+		},
+		{
+			text: "BPE",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+        },
+		{
+			text: "Unigram",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+        },
+		{
+			text: "None of the above",
+			explain: "Incorrect!"
+        }
+	]}
+/>
+
+### 4. What is a model head?
+
+<Question
+	choices={[
+		{
+			text: "A component of the base Transformer network that redirects tensors to their correct layers",
+			explain: "Incorrect! There's no such component."
path: chapters/en/chapter2/9.mdx
user: login: sergiopaniego
body: Duplicated
```suggestion
			explain: "There's no such component."
```
created_at: 2025-05-14T13:45:24Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,252 @@
+<FrameworkSwitchCourse {fw} />
+
+<!-- DISABLE-FRONTMATTER-SECTIONS -->
+
+# End-of-chapter quiz[[end-of-chapter-quiz]]
+
+<CourseFloatingBanner
+    chapter={2}
+    classNames="absolute z-10 right-0 top-0"
+/>
+
+### 1. What is the order of the language modeling pipeline?
+
+<Question
+	choices={[
+		{
+			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
+			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
+		},
+		{
+			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
+			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
+		},
+		{
+			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
+			explain: "Correct! The tokenizer can be used for both tokenizing and de-tokenizing.",
+            correct: true
+		}
+	]}
+/>
+
+### 2. How many dimensions does the tensor output by the base Transformer model have, and what are they?
+
+<Question
+	choices={[
+		{
+			text: "2: The sequence length and the batch size",
+			explain: "False! The tensor output by the model has a third dimension: hidden size."
+		},
+		{
+			text: "2: The sequence length and the hidden size",
+			explain: "False! All Transformer models handle batches, even with a single sequence; that would be a batch size of 1!"
+		},
+		{
+			text: "3: The sequence length, the batch size, and the hidden size",
+			explain: "Correct!",
+            correct: true
+		}
+	]}
+/>
+
+### 3. Which of the following is an example of subword tokenization?
+
+<Question
+	choices={[
+		{
+			text: "WordPiece",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+		},
+		{
+			text: "Character-based tokenization",
+			explain: "Character-based tokenization is not a type of subword tokenization."
+		},
+		{
+			text: "Splitting on whitespace and punctuation",
+			explain: "That's a word-based tokenization scheme!"
+		},
+		{
+			text: "BPE",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+        },
+		{
+			text: "Unigram",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+        },
+		{
+			text: "None of the above",
+			explain: "Incorrect!"
+        }
+	]}
+/>
+
+### 4. What is a model head?
+
+<Question
+	choices={[
+		{
+			text: "A component of the base Transformer network that redirects tensors to their correct layers",
+			explain: "Incorrect! There's no such component."
+		},
+		{
+			text: "Also known as the self-attention mechanism, it adapts the representation of a token according to the other tokens of the sequence",
+			explain: "Incorrect! The self-attention layer does contain attention \"heads,\" but these are not adaptation heads."
path: chapters/en/chapter2/9.mdx
user: login: sergiopaniego
body: ```suggestion
			explain: "The self-attention layer does contain attention \"heads,\" but these are not adaptation heads."
```
created_at: 2025-05-14T13:45:32Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,252 @@
+<FrameworkSwitchCourse {fw} />
+
+<!-- DISABLE-FRONTMATTER-SECTIONS -->
+
+# End-of-chapter quiz[[end-of-chapter-quiz]]
+
+<CourseFloatingBanner
+    chapter={2}
+    classNames="absolute z-10 right-0 top-0"
+/>
+
+### 1. What is the order of the language modeling pipeline?
+
+<Question
+	choices={[
+		{
+			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
+			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
+		},
+		{
+			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
+			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
+		},
+		{
+			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
+			explain: "Correct! The tokenizer can be used for both tokenizing and de-tokenizing.",
+            correct: true
+		}
+	]}
+/>
+
+### 2. How many dimensions does the tensor output by the base Transformer model have, and what are they?
+
+<Question
+	choices={[
+		{
+			text: "2: The sequence length and the batch size",
+			explain: "False! The tensor output by the model has a third dimension: hidden size."
+		},
+		{
+			text: "2: The sequence length and the hidden size",
+			explain: "False! All Transformer models handle batches, even with a single sequence; that would be a batch size of 1!"
+		},
+		{
+			text: "3: The sequence length, the batch size, and the hidden size",
+			explain: "Correct!",
+            correct: true
+		}
+	]}
+/>
+
+### 3. Which of the following is an example of subword tokenization?
+
+<Question
+	choices={[
+		{
+			text: "WordPiece",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+		},
+		{
+			text: "Character-based tokenization",
+			explain: "Character-based tokenization is not a type of subword tokenization."
+		},
+		{
+			text: "Splitting on whitespace and punctuation",
+			explain: "That's a word-based tokenization scheme!"
+		},
+		{
+			text: "BPE",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+        },
+		{
+			text: "Unigram",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+        },
+		{
+			text: "None of the above",
+			explain: "Incorrect!"
+        }
+	]}
+/>
+
+### 4. What is a model head?
+
+<Question
+	choices={[
+		{
+			text: "A component of the base Transformer network that redirects tensors to their correct layers",
+			explain: "Incorrect! There's no such component."
+		},
+		{
+			text: "Also known as the self-attention mechanism, it adapts the representation of a token according to the other tokens of the sequence",
+			explain: "Incorrect! The self-attention layer does contain attention \"heads,\" but these are not adaptation heads."
+		},
+		{
+			text: "An additional component, usually made up of one or a few layers, to convert the transformer predictions to a task-specific output",
+			explain: "That's right. Adaptation heads, also known simply as heads, come up in different forms: language modeling heads, question answering heads, sequence classification heads... ",
+			correct: true
+		} 
+	]}
+/>
+
+### 5. What is an AutoModel?
+
+<Question
+	choices={[
+		{
+			text: "A model that automatically trains on your data",
+			explain: "Incorrect. Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
path: chapters/en/chapter2/9.mdx
user: login: sergiopaniego
body: Duplicated
```suggestion
			explain: "Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
```
created_at: 2025-05-14T13:45:57Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,252 @@
+<FrameworkSwitchCourse {fw} />
+
+<!-- DISABLE-FRONTMATTER-SECTIONS -->
+
+# End-of-chapter quiz[[end-of-chapter-quiz]]
+
+<CourseFloatingBanner
+    chapter={2}
+    classNames="absolute z-10 right-0 top-0"
+/>
+
+### 1. What is the order of the language modeling pipeline?
+
+<Question
+	choices={[
+		{
+			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
+			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
+		},
+		{
+			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
+			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
+		},
+		{
+			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
+			explain: "Correct! The tokenizer can be used for both tokenizing and de-tokenizing.",
+            correct: true
+		}
+	]}
+/>
+
+### 2. How many dimensions does the tensor output by the base Transformer model have, and what are they?
+
+<Question
+	choices={[
+		{
+			text: "2: The sequence length and the batch size",
+			explain: "False! The tensor output by the model has a third dimension: hidden size."
+		},
+		{
+			text: "2: The sequence length and the hidden size",
+			explain: "False! All Transformer models handle batches, even with a single sequence; that would be a batch size of 1!"
+		},
+		{
+			text: "3: The sequence length, the batch size, and the hidden size",
+			explain: "Correct!",
+            correct: true
+		}
+	]}
+/>
+
+### 3. Which of the following is an example of subword tokenization?
+
+<Question
+	choices={[
+		{
+			text: "WordPiece",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+		},
+		{
+			text: "Character-based tokenization",
+			explain: "Character-based tokenization is not a type of subword tokenization."
+		},
+		{
+			text: "Splitting on whitespace and punctuation",
+			explain: "That's a word-based tokenization scheme!"
+		},
+		{
+			text: "BPE",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+        },
+		{
+			text: "Unigram",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+        },
+		{
+			text: "None of the above",
+			explain: "Incorrect!"
+        }
+	]}
+/>
+
+### 4. What is a model head?
+
+<Question
+	choices={[
+		{
+			text: "A component of the base Transformer network that redirects tensors to their correct layers",
+			explain: "Incorrect! There's no such component."
+		},
+		{
+			text: "Also known as the self-attention mechanism, it adapts the representation of a token according to the other tokens of the sequence",
+			explain: "Incorrect! The self-attention layer does contain attention \"heads,\" but these are not adaptation heads."
+		},
+		{
+			text: "An additional component, usually made up of one or a few layers, to convert the transformer predictions to a task-specific output",
+			explain: "That's right. Adaptation heads, also known simply as heads, come up in different forms: language modeling heads, question answering heads, sequence classification heads... ",
+			correct: true
+		} 
+	]}
+/>
+
+### 5. What is an AutoModel?
+
+<Question
+	choices={[
+		{
+			text: "A model that automatically trains on your data",
+			explain: "Incorrect. Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
+		},
+		{
+			text: "An object that returns the correct architecture based on the checkpoint",
+			explain: "Exactly: the <code>AutoModel</code> only needs to know the checkpoint from which to initialize to return the correct architecture.",
+			correct: true
+		},
+		{
+			text: "A model that automatically detects the language used for its inputs to load the correct weights",
+			explain: "Incorrect; while some checkpoints and models are capable of handling multiple languages, there are no built-in tools for automatic checkpoint selection according to language. You should head over to the <a href='https://huggingface.co/models'>Model Hub</a> to find the best checkpoint for your task!"
path: chapters/en/chapter2/9.mdx
user: login: sergiopaniego
body: ```suggestion
			explain: "While some checkpoints and models are capable of handling multiple languages, there are no built-in tools for automatic checkpoint selection according to language. You should head over to the <a href='https://huggingface.co/models'>Model Hub</a> to find the best checkpoint for your task!"
```
created_at: 2025-05-14T13:46:12Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,252 @@
+<FrameworkSwitchCourse {fw} />
+
+<!-- DISABLE-FRONTMATTER-SECTIONS -->
+
+# End-of-chapter quiz[[end-of-chapter-quiz]]
+
+<CourseFloatingBanner
+    chapter={2}
+    classNames="absolute z-10 right-0 top-0"
+/>
+
+### 1. What is the order of the language modeling pipeline?
+
+<Question
+	choices={[
+		{
+			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
+			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
+		},
+		{
+			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
+			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
+		},
+		{
+			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
+			explain: "Correct! The tokenizer can be used for both tokenizing and de-tokenizing.",
+            correct: true
+		}
+	]}
+/>
+
+### 2. How many dimensions does the tensor output by the base Transformer model have, and what are they?
+
+<Question
+	choices={[
+		{
+			text: "2: The sequence length and the batch size",
+			explain: "False! The tensor output by the model has a third dimension: hidden size."
+		},
+		{
+			text: "2: The sequence length and the hidden size",
+			explain: "False! All Transformer models handle batches, even with a single sequence; that would be a batch size of 1!"
+		},
+		{
+			text: "3: The sequence length, the batch size, and the hidden size",
+			explain: "Correct!",
+            correct: true
+		}
+	]}
+/>
+
+### 3. Which of the following is an example of subword tokenization?
+
+<Question
+	choices={[
+		{
+			text: "WordPiece",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+		},
+		{
+			text: "Character-based tokenization",
+			explain: "Character-based tokenization is not a type of subword tokenization."
+		},
+		{
+			text: "Splitting on whitespace and punctuation",
+			explain: "That's a word-based tokenization scheme!"
+		},
+		{
+			text: "BPE",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+        },
+		{
+			text: "Unigram",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+        },
+		{
+			text: "None of the above",
+			explain: "Incorrect!"
+        }
+	]}
+/>
+
+### 4. What is a model head?
+
+<Question
+	choices={[
+		{
+			text: "A component of the base Transformer network that redirects tensors to their correct layers",
+			explain: "Incorrect! There's no such component."
+		},
+		{
+			text: "Also known as the self-attention mechanism, it adapts the representation of a token according to the other tokens of the sequence",
+			explain: "Incorrect! The self-attention layer does contain attention \"heads,\" but these are not adaptation heads."
+		},
+		{
+			text: "An additional component, usually made up of one or a few layers, to convert the transformer predictions to a task-specific output",
+			explain: "That's right. Adaptation heads, also known simply as heads, come up in different forms: language modeling heads, question answering heads, sequence classification heads... ",
+			correct: true
+		} 
+	]}
+/>
+
+### 5. What is an AutoModel?
+
+<Question
+	choices={[
+		{
+			text: "A model that automatically trains on your data",
+			explain: "Incorrect. Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
+		},
+		{
+			text: "An object that returns the correct architecture based on the checkpoint",
+			explain: "Exactly: the <code>AutoModel</code> only needs to know the checkpoint from which to initialize to return the correct architecture.",
+			correct: true
+		},
+		{
+			text: "A model that automatically detects the language used for its inputs to load the correct weights",
+			explain: "Incorrect; while some checkpoints and models are capable of handling multiple languages, there are no built-in tools for automatic checkpoint selection according to language. You should head over to the <a href='https://huggingface.co/models'>Model Hub</a> to find the best checkpoint for your task!"
+		} 
+	]}
+/>
+
+### 6. What are the techniques to be aware of when batching sequences of different lengths together?
+
+<Question
+	choices={[
+		{
+			text: "Truncating",
+			explain: "Yes, truncation is a correct way of evening out sequences so that they fit in a rectangular shape. Is it the only one, though?",
+			correct: true
+		},
+		{
+			text: "Returning tensors",
+			explain: "While the other techniques allow you to return rectangular tensors, returning tensors isn't helpful when batching sequences together."
+		},
+		{
+			text: "Padding",
+			explain: "Yes, padding is a correct way of evening out sequences so that they fit in a rectangular shape. Is it the only one, though?",
+			correct: true
+		}, 
+		{
+			text: "Attention masking",
+			explain: "Absolutely! Attention masks are of prime importance when handling sequences of different lengths. That's not the only technique to be aware of, however.",
+			correct: true
+		} 
+	]}
+/>
+
+### 7. What is the point of applying a SoftMax function to the logits output by a sequence classification model?
+
+<Question
+	choices={[
+		{
+			text: "It softens the logits so that they're more reliable.",
+			explain: "No, the SoftMax function does not affect the reliability of results."
+		},
+		{
+			text: "It applies a lower and upper bound so that they're understandable.",
+			explain: "Correct! The resulting values are bound between 0 and 1. That's not the only reason we use a SoftMax function, though.",
path: chapters/en/chapter2/9.mdx
user: login: sergiopaniego
body: ```suggestion
			explain: "The resulting values are bound between 0 and 1. That's not the only reason we use a SoftMax function, though.",
```
created_at: 2025-05-14T13:46:22Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,252 @@
+<FrameworkSwitchCourse {fw} />
+
+<!-- DISABLE-FRONTMATTER-SECTIONS -->
+
+# End-of-chapter quiz[[end-of-chapter-quiz]]
+
+<CourseFloatingBanner
+    chapter={2}
+    classNames="absolute z-10 right-0 top-0"
+/>
+
+### 1. What is the order of the language modeling pipeline?
+
+<Question
+	choices={[
+		{
+			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
+			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
+		},
+		{
+			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
+			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
+		},
+		{
+			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
+			explain: "Correct! The tokenizer can be used for both tokenizing and de-tokenizing.",
+            correct: true
+		}
+	]}
+/>
+
+### 2. How many dimensions does the tensor output by the base Transformer model have, and what are they?
+
+<Question
+	choices={[
+		{
+			text: "2: The sequence length and the batch size",
+			explain: "False! The tensor output by the model has a third dimension: hidden size."
+		},
+		{
+			text: "2: The sequence length and the hidden size",
+			explain: "False! All Transformer models handle batches, even with a single sequence; that would be a batch size of 1!"
+		},
+		{
+			text: "3: The sequence length, the batch size, and the hidden size",
+			explain: "Correct!",
+            correct: true
+		}
+	]}
+/>
+
+### 3. Which of the following is an example of subword tokenization?
+
+<Question
+	choices={[
+		{
+			text: "WordPiece",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+		},
+		{
+			text: "Character-based tokenization",
+			explain: "Character-based tokenization is not a type of subword tokenization."
+		},
+		{
+			text: "Splitting on whitespace and punctuation",
+			explain: "That's a word-based tokenization scheme!"
+		},
+		{
+			text: "BPE",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+        },
+		{
+			text: "Unigram",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+        },
+		{
+			text: "None of the above",
+			explain: "Incorrect!"
+        }
+	]}
+/>
+
+### 4. What is a model head?
+
+<Question
+	choices={[
+		{
+			text: "A component of the base Transformer network that redirects tensors to their correct layers",
+			explain: "Incorrect! There's no such component."
+		},
+		{
+			text: "Also known as the self-attention mechanism, it adapts the representation of a token according to the other tokens of the sequence",
+			explain: "Incorrect! The self-attention layer does contain attention \"heads,\" but these are not adaptation heads."
+		},
+		{
+			text: "An additional component, usually made up of one or a few layers, to convert the transformer predictions to a task-specific output",
+			explain: "That's right. Adaptation heads, also known simply as heads, come up in different forms: language modeling heads, question answering heads, sequence classification heads... ",
+			correct: true
+		} 
+	]}
+/>
+
+### 5. What is an AutoModel?
+
+<Question
+	choices={[
+		{
+			text: "A model that automatically trains on your data",
+			explain: "Incorrect. Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
+		},
+		{
+			text: "An object that returns the correct architecture based on the checkpoint",
+			explain: "Exactly: the <code>AutoModel</code> only needs to know the checkpoint from which to initialize to return the correct architecture.",
+			correct: true
+		},
+		{
+			text: "A model that automatically detects the language used for its inputs to load the correct weights",
+			explain: "Incorrect; while some checkpoints and models are capable of handling multiple languages, there are no built-in tools for automatic checkpoint selection according to language. You should head over to the <a href='https://huggingface.co/models'>Model Hub</a> to find the best checkpoint for your task!"
+		} 
+	]}
+/>
+
+### 6. What are the techniques to be aware of when batching sequences of different lengths together?
+
+<Question
+	choices={[
+		{
+			text: "Truncating",
+			explain: "Yes, truncation is a correct way of evening out sequences so that they fit in a rectangular shape. Is it the only one, though?",
+			correct: true
+		},
+		{
+			text: "Returning tensors",
+			explain: "While the other techniques allow you to return rectangular tensors, returning tensors isn't helpful when batching sequences together."
+		},
+		{
+			text: "Padding",
+			explain: "Yes, padding is a correct way of evening out sequences so that they fit in a rectangular shape. Is it the only one, though?",
+			correct: true
+		}, 
+		{
+			text: "Attention masking",
+			explain: "Absolutely! Attention masks are of prime importance when handling sequences of different lengths. That's not the only technique to be aware of, however.",
+			correct: true
+		} 
+	]}
+/>
+
+### 7. What is the point of applying a SoftMax function to the logits output by a sequence classification model?
+
+<Question
+	choices={[
+		{
+			text: "It softens the logits so that they're more reliable.",
+			explain: "No, the SoftMax function does not affect the reliability of results."
+		},
+		{
+			text: "It applies a lower and upper bound so that they're understandable.",
+			explain: "Correct! The resulting values are bound between 0 and 1. That's not the only reason we use a SoftMax function, though.",
+            correct: true
+		},
+		{
+			text: "The total sum of the output is then 1, resulting in a possible probabilistic interpretation.",
+			explain: "Correct! That's not the only reason we use a SoftMax function, though.",
path: chapters/en/chapter2/9.mdx
user: login: sergiopaniego
body: ```suggestion
			explain: "That's not the only reason we use a SoftMax function, though.",
```
created_at: 2025-05-14T13:46:35Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,252 @@
+<FrameworkSwitchCourse {fw} />
+
+<!-- DISABLE-FRONTMATTER-SECTIONS -->
+
+# End-of-chapter quiz[[end-of-chapter-quiz]]
+
+<CourseFloatingBanner
+    chapter={2}
+    classNames="absolute z-10 right-0 top-0"
+/>
+
+### 1. What is the order of the language modeling pipeline?
+
+<Question
+	choices={[
+		{
+			text: "First, the model, which handles text and returns raw predictions. The tokenizer then makes sense of these predictions and converts them back to text when needed.",
+			explain: "The model cannot understand text! The tokenizer must first tokenize the text and convert it to IDs so that it is understandable by the model."
+		},
+		{
+			text: "First, the tokenizer, which handles text and returns IDs. The model handles these IDs and outputs a prediction, which can be some text.",
+			explain: "The model's prediction cannot be text straight away. The tokenizer has to be used in order to convert the prediction back to text!"
+		},
+		{
+			text: "The tokenizer handles text and returns IDs. The model handles these IDs and outputs a prediction. The tokenizer can then be used once again to convert these predictions back to some text.",
+			explain: "Correct! The tokenizer can be used for both tokenizing and de-tokenizing.",
+            correct: true
+		}
+	]}
+/>
+
+### 2. How many dimensions does the tensor output by the base Transformer model have, and what are they?
+
+<Question
+	choices={[
+		{
+			text: "2: The sequence length and the batch size",
+			explain: "False! The tensor output by the model has a third dimension: hidden size."
+		},
+		{
+			text: "2: The sequence length and the hidden size",
+			explain: "False! All Transformer models handle batches, even with a single sequence; that would be a batch size of 1!"
+		},
+		{
+			text: "3: The sequence length, the batch size, and the hidden size",
+			explain: "Correct!",
+            correct: true
+		}
+	]}
+/>
+
+### 3. Which of the following is an example of subword tokenization?
+
+<Question
+	choices={[
+		{
+			text: "WordPiece",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+		},
+		{
+			text: "Character-based tokenization",
+			explain: "Character-based tokenization is not a type of subword tokenization."
+		},
+		{
+			text: "Splitting on whitespace and punctuation",
+			explain: "That's a word-based tokenization scheme!"
+		},
+		{
+			text: "BPE",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+        },
+		{
+			text: "Unigram",
+			explain: "Yes, that's one example of subword tokenization!",
+            correct: true
+        },
+		{
+			text: "None of the above",
+			explain: "Incorrect!"
+        }
+	]}
+/>
+
+### 4. What is a model head?
+
+<Question
+	choices={[
+		{
+			text: "A component of the base Transformer network that redirects tensors to their correct layers",
+			explain: "Incorrect! There's no such component."
+		},
+		{
+			text: "Also known as the self-attention mechanism, it adapts the representation of a token according to the other tokens of the sequence",
+			explain: "Incorrect! The self-attention layer does contain attention \"heads,\" but these are not adaptation heads."
+		},
+		{
+			text: "An additional component, usually made up of one or a few layers, to convert the transformer predictions to a task-specific output",
+			explain: "That's right. Adaptation heads, also known simply as heads, come up in different forms: language modeling heads, question answering heads, sequence classification heads... ",
+			correct: true
+		} 
+	]}
+/>
+
+### 5. What is an AutoModel?
+
+<Question
+	choices={[
+		{
+			text: "A model that automatically trains on your data",
+			explain: "Incorrect. Are you mistaking this with our <a href='https://huggingface.co/autotrain'>AutoTrain</a> product?"
+		},
+		{
+			text: "An object that returns the correct architecture based on the checkpoint",
+			explain: "Exactly: the <code>AutoModel</code> only needs to know the checkpoint from which to initialize to return the correct architecture.",
+			correct: true
+		},
+		{
+			text: "A model that automatically detects the language used for its inputs to load the correct weights",
+			explain: "Incorrect; while some checkpoints and models are capable of handling multiple languages, there are no built-in tools for automatic checkpoint selection according to language. You should head over to the <a href='https://huggingface.co/models'>Model Hub</a> to find the best checkpoint for your task!"
+		} 
+	]}
+/>
+
+### 6. What are the techniques to be aware of when batching sequences of different lengths together?
+
+<Question
+	choices={[
+		{
+			text: "Truncating",
+			explain: "Yes, truncation is a correct way of evening out sequences so that they fit in a rectangular shape. Is it the only one, though?",
+			correct: true
+		},
+		{
+			text: "Returning tensors",
+			explain: "While the other techniques allow you to return rectangular tensors, returning tensors isn't helpful when batching sequences together."
+		},
+		{
+			text: "Padding",
+			explain: "Yes, padding is a correct way of evening out sequences so that they fit in a rectangular shape. Is it the only one, though?",
+			correct: true
+		}, 
+		{
+			text: "Attention masking",
+			explain: "Absolutely! Attention masks are of prime importance when handling sequences of different lengths. That's not the only technique to be aware of, however.",
+			correct: true
+		} 
+	]}
+/>
+
+### 7. What is the point of applying a SoftMax function to the logits output by a sequence classification model?
+
+<Question
+	choices={[
+		{
+			text: "It softens the logits so that they're more reliable.",
+			explain: "No, the SoftMax function does not affect the reliability of results."
+		},
+		{
+			text: "It applies a lower and upper bound so that they're understandable.",
+			explain: "Correct! The resulting values are bound between 0 and 1. That's not the only reason we use a SoftMax function, though.",
+            correct: true
+		},
+		{
+			text: "The total sum of the output is then 1, resulting in a possible probabilistic interpretation.",
+			explain: "Correct! That's not the only reason we use a SoftMax function, though.",
+            correct: true
+		}
+	]}
+/>
+
+### 8. What method is most of the tokenizer API centered around?
+
+<Question
+	choices={[
+		{
+			text: "<code>encode</code>, as it can encode text into IDs and IDs into predictions",
+			explain: "Wrong! While the <code>encode</code> method does exist on tokenizers, it does not exist on models."
+		},
+		{
+			text: "Calling the tokenizer object directly.",
+			explain: "Exactly! The <code>__call__</code> method of the tokenizer is a very powerful method which can handle pretty much anything. It is also the method used to retrieve predictions from a model.",
+			correct: true
+		},
+		{
+			text: "<code>pad</code>",
+			explain: "Wrong! Padding is very useful, but it's just one part of the tokenizer API."
+		},
+		{
+			text: "<code>tokenize</code>",
+			explain: "The <code>tokenize</code> method is arguably one of the most useful methods, but it isn't the core of the tokenizer API."
+		}
+	]}
+/>
+
+### 9. What does the `result` variable contain in this code sample?
+
+```py
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
+result = tokenizer.tokenize("Hello!")
+```
+
+<Question
+	choices={[
+		{
+			text: "A list of strings, each string being a token",
+			explain: "Absolutely! Convert this to IDs, and send them to a model!",
+            correct: true
+		},
+		{
+			text: "A list of IDs",
+			explain: "Incorrect; that's what the <code>__call__</code> or <code>convert_tokens_to_ids</code> method is for!"
path: chapters/en/chapter2/9.mdx
user: login: sergiopaniego
body: ```suggestion
			explain: "That's what the <code>__call__</code> or <code>convert_tokens_to_ids</code> method is for!"
```
created_at: 2025-05-14T13:47:02Z
author_association: MEMBER
diff_hunk: @@ -46,6 +46,8 @@
   - local: chapter2/7
     title: Basic usage completed!
   - local: chapter2/8
+    title: Inference with transformers and beyond
path: chapters/en/_toctree.yml
user: login: sergiopaniego
body: Same as in the `.mdx`
```suggestion
    title: Optimized Inference Deployment
```
created_at: 2025-05-14T13:48:23Z
author_association: MEMBER

PR comments:
user: login: sergiopaniego
created_at: 2025-05-14T13:57:19Z
body: Super well written and informative, as always @burtenshaw. Great improvement over the previous iteration! Main issue is the failing build process. Rest are just nits 😄 
author_association: MEMBER
user: login: burtenshaw
created_at: 2025-05-14T14:04:07Z
body: > Super well written and informative, as always @burtenshaw. Great improvement over the previous iteration! Main issue is the failing build process. Rest are just nits 😄

Thanks @sergiopaniego . Working on the framework options now.
author_association: COLLABORATOR
user: login: HuggingFaceDocBuilderDev
created_at: 2025-05-14T14:15:59Z
body: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/course/pr_911). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
author_association: NONE