What is so special about special tokens?

username: ShaoMinLiu-Holmusk
cooked: <p>I am not sure if anyone can help to answer this here but I cannot seems to be able to find an answer from anywhere:<br>
what exactly is the difference between “token” and a “special token”?</p>
<p>I understand the following:</p>
<ul>
<li>what is a typical token</li>
<li>what is a typical special token: MASK, UNK, SEP, etc</li>
<li>when do you add a token (when you want to expand your vocab)</li>
</ul>
<p>What I don’t understand is, under what kind of capacity will you want to create a new special token, any examples what we need it for and when we want to create a special token other than those default special tokens? If an example uses a special token, why can’t a normal token achieve the same objective?</p>
<pre><code class="lang-auto">tokenizer.add_tokens(['[EOT]'], special_tokens=True)
</code></pre>
<p>And I also dont quite understand the following description in the source documentation.<br>
what difference does it do to our model if we set add_special_tokens to False?</p>
<pre><code class="lang-auto">add_special_tokens (bool, optional, defaults to True) — Whether or not to encode the sequences with the special tokens relative to their model.
</code></pre>

username: basingse
cooked: <p>As per the docs <a href="https://huggingface.co/docs/transformers/main/en/internal/tokenization_utils#transformers.SpecialTokensMixin.add_tokens" rel="noopener nofollow ugc">here</a></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8bf77c8dd3cd2a567e09ff83e6ec073eb62c0afa.png" alt="image" data-base62-sha1="jYcByUHc7EcjTYu29W9dbRhsKD0" width="605" height="92"></p>
<p>It means if you have added <code>token = EOT</code> then suppose your string contains <code>eot i am here</code> then <code>eot</code> will be treated differently if its special token.</p>
<p>For example, this is how it tooks<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cff247cf73d64802921906e74d0eb0ea020c9fb.png" data-download-href="https://discuss.pytorch.org/uploads/default/6cff247cf73d64802921906e74d0eb0ea020c9fb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6cff247cf73d64802921906e74d0eb0ea020c9fb_2_690x366.png" alt="image" data-base62-sha1="fyecQktdnkqAWZgTHeMY4IIjvwn" width="690" height="366" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6cff247cf73d64802921906e74d0eb0ea020c9fb_2_690x366.png, https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cff247cf73d64802921906e74d0eb0ea020c9fb.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cff247cf73d64802921906e74d0eb0ea020c9fb.png 2x" data-dominant-color="F4F4F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">895×475 41.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>and also about first question, I dont have much idea as I never used <code>add_tokens</code> functionality (the work I do is mostly on NLP and here the default special tokens are enough)</p>

username: ShaoMinLiu-Holmusk
cooked: <p>interesting, in the above example, the word <code>eot</code> got splitted into two tokens despite EOT being a special token, does it mean that special tokens are case sensitive?</p>

username: basingse
cooked: <p>yeah. For special tokens there will no processing (like lowercase etc) so <code>eot</code> will be splitted but <code>EOT</code> will not be splitted.</p>