End-of-chapter quiz

Let’s test what you learned in this chapter!

1. When should you train a new tokenizer?

2. What is the advantage of using a generator of lists of texts compared to a list of lists of texts when using `train_new_from_iterator()`?

3. What are the advantages of using a “fast” tokenizer?

4. How does the `token-classification` pipeline handle entities that span over several tokens?

5. How does the `question-answering` pipeline handle long contexts?

6. What is normalization?

7. What is pre-tokenization for a subword tokenizer?

8. Select the sentences that apply to the BPE model of tokenization.

BPE is a subword tokenization algorithm that starts with a small vocabulary and learns merge rules. BPE is a subword tokenization algorithm that starts with a big vocabulary and progressively removes tokens from it. BPE tokenizers learn merge rules by merging the pair of tokens that is the most frequent. A BPE tokenizer learns a merge rule by merging the pair of tokens that maximizes a score that privileges frequent pairs with less frequent individual parts. BPE tokenizes words into subwords by splitting them into characters and then applying the merge rules. BPE tokenizes words into subwords by finding the longest subword starting from the beginning that is in the vocabulary, then repeating the process for the rest of the text.

9. Select the sentences that apply to the WordPiece model of tokenization.

WordPiece is a subword tokenization algorithm that starts with a small vocabulary and learns merge rules. WordPiece is a subword tokenization algorithm that starts with a big vocabulary and progressively removes tokens from it. WordPiece tokenizers learn merge rules by merging the pair of tokens that is the most frequent. A WordPiece tokenizer learns a merge rule by merging the pair of tokens that maximizes a score that privileges frequent pairs with less frequent individual parts. WordPiece tokenizes words into subwords by finding the most likely segmentation into tokens, according to the model. WordPiece tokenizes words into subwords by finding the longest subword starting from the beginning that is in the vocabulary, then repeating the process for the rest of the text.

Course

End-of-chapter quiz

1. When should you train a new tokenizer?

2. What is the advantage of using a generator of lists of texts compared to a list of lists of texts when using `train_new_from_iterator()`?

3. What are the advantages of using a “fast” tokenizer?

4. How does the `token-classification` pipeline handle entities that span over several tokens?

5. How does the `question-answering` pipeline handle long contexts?

6. What is normalization?

7. What is pre-tokenization for a subword tokenizer?

8. Select the sentences that apply to the BPE model of tokenization.

9. Select the sentences that apply to the WordPiece model of tokenization.

10. Select the sentences that apply to the Unigram model of tokenization.

Course

End-of-chapter quiz

1. When should you train a new tokenizer?

2. What is the advantage of using a generator of lists of texts compared to a list of lists of texts when using train_new_from_iterator()?

3. What are the advantages of using a “fast” tokenizer?

4. How does the token-classification pipeline handle entities that span over several tokens?

5. How does the question-answering pipeline handle long contexts?

6. What is normalization?

7. What is pre-tokenization for a subword tokenizer?

8. Select the sentences that apply to the BPE model of tokenization.

9. Select the sentences that apply to the WordPiece model of tokenization.

10. Select the sentences that apply to the Unigram model of tokenization.

2. What is the advantage of using a generator of lists of texts compared to a list of lists of texts when using `train_new_from_iterator()`?

4. How does the `token-classification` pipeline handle entities that span over several tokens?

5. How does the `question-answering` pipeline handle long contexts?