BERT for multiple sentences

username: sandeep1
cooked: <p>I know that [CLS] means the start of a sentence and [SEP] makes BERT know the second sentence has begun.</p>
<p>However, I have a question.</p>
<p>If I have 3 sentences, which are s1 and s2 and s3, and our fine-tuning task is the same.</p>
<p>In one way, I add special tokens and the input looks like [CLS]+s1+[SEP] + s2 + [SEP]+ s3 [SEP].</p>
<pre><code class="lang-auto">from transformers import BertTokenizer

model_name='allenai/scibert_scivocab_uncased'
model_name = 'bert-base-uncased'

tokenizer= AutoTokenizer.from_pretrained(model_name)

</code></pre>
<p>tokenizer([[“hello i am going to”,“Rome”]])</p>
<pre><code class="lang-auto">Outputs correct ie:
</code></pre>
<pre><code class="lang-auto">{'input_ids': [102, 121, 1972, 705, 8303, 147, 758, 26241, 188, 9319, 24616, 422, 6474, 129, 9724, 650, 3978, 190, 9319, 24616, 13903, 372, 1942, 19226, 24648, 4931, 137, 3594, 17345, 190, 6666, 422, 334, 12605, 30113, 475, 506, 3437, 131, 633, 10945, 5458, 263, 17094, 205, 103]}
</code></pre>
<p>But, while tokenizing three sentences like this :-<br>
<code>tokenizer([["hello i am going","to","Rome"]])</code><br>
It give the following error :-<br>
<code>TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]</code></p>
<p>How do we input such a sequences to the BERT and it is justified to do so?</p>
<p>Thanks for any help!</p>

username: Matias_Vasquez
cooked: <p>Hi,</p>
<p>I tried to reproduce the error, but could not do it. With your example, I get the following (no error)</p>
<pre><code class="lang-python"># Input
tokenizer(["hello i am going","to","Rome"])

# Output
# {'input_ids': [[101, 7592, 1045, 2572, 2183, 102], [101, 2000, 102], [101, 4199, 102]], 'token_type_ids': [[0, 0, 0, 0, 0, 0], [0, 0, 0], [0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1], [1, 1, 1], [1, 1, 1]]}
</code></pre>

username: sandeep1
cooked: <p>Sorry, I missed one bracket. Edited the question.<br>
What i want is 1 Dimensional ids of multiple sentences provided by BERT</p>

username: Matias_Vasquez
cooked: <p>tl:dr;</p>
<p>your approach is correct. Don´t think you need to change it.</p>
<aside class="quote no-group" data-username="sandeep1" data-post="1" data-topic="150004">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sandeep1/48/12454_2.png" class="avatar"> sandeep1:</div>
<blockquote>
<p>In one way, I add special tokens and the input looks like [CLS]+s1+[SEP] + s2 + [SEP]+ s3 [SEP].</p>
</blockquote>
</aside>
<details>
<summary>
Long answer</summary>
<p>If you wanted each sentence to be a batch, then you would do it as in the first post (before editing).</p>
<p><a href="https://huggingface.co/docs/transformers/preprocessing" rel="noopener nofollow ugc">Here</a> is more information about batches and preprocessing for the tokenizer.</p>
<pre data-code-wrap="python"><code class="lang-python">tokenizer(["hello i am going","to","Rome"])
</code></pre>
<p>However, as you mentioned before, you do <strong>NOT</strong> want this, but rather 1 Dimensional ids of multiple sentences.</p>
<p>If you go to the <a href="https://huggingface.co/docs/transformers/main/en/model_doc/bert#transformers.BertTokenizer" rel="noopener nofollow ugc">BertTokenizer</a> and  to the <a href="https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.PreTrainedTokenizer" rel="noopener nofollow ugc">PreTrainedTokenizer</a> documentation and look for the <code>__call__</code> method you will see which parameters it accepts.</p>
<p>The most relevant for us right now are <code>text</code> and <code>text_pair</code>. Both accept the same types described below.</p>
<h3><a name="text-text_pair-1" class="anchor" href="#text-text_pair-1"></a><code>text</code> + <code>text_pair</code></h3>
<p>Both sentences are merged into a 1D vector, separated with the <code>[SEP] = 102</code> token.</p>
<pre data-code-wrap="python"><code class="lang-python">tokenizer(text="hello i am going, to", text_pair="Rome")
# {'input_ids': [101, 7592, 1045, 2572, 2183, 1010, 2000, 102, 4199, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 1, 1], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
</code></pre>
<h1><a name="parameters-accepted-in-text-text_pair-2" class="anchor" href="#parameters-accepted-in-text-text_pair-2"></a>Parameters accepted in text / text_pair</h1>
<h3><a name="str-3" class="anchor" href="#str-3"></a>str</h3>
<p>This is the most simple case. You only input one <code>str</code>. This can be a word or a sentence.</p>
<pre data-code-wrap="python"><code class="lang-python">tokenizer("hello i am going to Rome")
#{'input_ids': [101, 7592, 1045, 2572, 2183, 2000, 4199, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]}
</code></pre>
<h3><a name="liststr-4" class="anchor" href="#liststr-4"></a>List[str]</h3>
<p>Here we define a batch of sentences to be tokenized.</p>
<pre data-code-wrap="python"><code class="lang-python">tokenizer(["hello i am going","to","Rome"])
# {'input_ids': [[101, 7592, 1045, 2572, 2183, 102], [101, 2000, 102], [101, 4199, 102]], 'token_type_ids': [[0, 0, 0, 0, 0, 0], [0, 0, 0], [0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1], [1, 1, 1], [1, 1, 1]]}
</code></pre>
<h3><a name="listliststr-5" class="anchor" href="#listliststr-5"></a>List[List[str]]</h3>
<p>Now this is the case you are trying to do. However, even though it is not explicit anywhere (at least I did not find it), this list of lists is <strong>not</strong> for sequences of any length. As far as I understood, the outer list refers to the batches. However, the inner list is for a <strong>pair</strong> of sentences that will be treated like <code>text</code> - <code>text_pair</code>. If you enter more than 2 sentences in a batch, you will get an error.</p>
<p>Here is an example for two batches adding the special token manually.</p>
<pre data-code-wrap="python"><code class="lang-python">encoded_input = tokenizer(text=[["hello i am going [SEP] to [SEP] Rome", "other sentence"], ["next", "batch"]])

print("input:   ", encoded_input)
print("batch 1: ", tokenizer.decode(encoded_input["input_ids"][0]))
print("batch 2: ", tokenizer.decode(encoded_input["input_ids"][1]))
</code></pre>
<pre data-code-wrap="python"><code class="lang-python"># Output:
# input:    {'input_ids': [[101, 7592, 1045, 2572, 2183, 102, 2000, 102, 4199, 102, 2060, 6251, 102], [101, 2279, 102, 14108, 102]], 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1], [0, 0, 0, 1, 1]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1]]}
# batch 1:  [CLS] hello i am going [SEP] to [SEP] rome [SEP] other sentence [SEP]
# batch 2:  [CLS] next [SEP] batch [SEP]
</code></pre>
<p>As you can see, the manually added special tokens are treated the same when encoding and decoding the sentence. The problem is that <code>token_type_ids</code> can <strong>only</strong> be <code>0</code> or <code>1</code>, there are no more <code>ids</code> for a third sentence.</p>
<p>You can use a visualization tool like <a href="https://github.com/jessevig/bertviz" rel="noopener nofollow ugc"><code>BertViz</code></a> to see how your sentences behave and if it is your desired behavior.</p>
<ul>
<li>No ```[`SEP]`` token</li>
</ul>
<p><code>going</code> attends to <code>to</code><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0bf066e7ea5bc66f54794dba5b9cc44f0d2a1e7e.png" alt="image" data-base62-sha1="1HCjhC1nTsE5hKrMJ3PnUIfo01g" width="323" height="166"></p>
<ul>
<li>With <code>[`SEP]`` token </code>going<code>attends to</code>[SEP]<code>, almost does not notice </code>to```<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/7/0705870c45e9680b9047f9be410f7a4e4d257c51.png" alt="image" data-base62-sha1="107bt2q9sbAMNexXbzRTHEo5oqd" width="317" height="217"></li>
</ul>
</details>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>