How to make BertTokenizer return GPU tensors instead of CPU tensors?

username: Jack_N
cooked: <p>I am wondering how I can make the BERT tokenizer return tensors on the GPU rather than the CPU. I am following the sample code found here: <a href="https://huggingface.co/docs/transformers/main/en/model_doc/bert#transformers.BertForMaskedLM" class="inline-onebox" rel="noopener nofollow ugc">BERT</a>. The code is below.</p>
<p>My question is about the 5th line of code, specifically how I can make the tokenizer return a cuda tensor instead of having to add the line of code <code>inputs = inputs.to("cuda")</code>.</p>
<pre><code class="lang-auto">from transformers import BertTokenizer, BertForPreTraining

import torch

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

model = BertForPreTraining.from_pretrained("bert-base-uncased")

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")

outputs = model(**inputs)

prediction_logits = outputs.prediction_logits

seq_relationship_logits = outputs.seq_relationship_logits
</code></pre>

username: thecho7
cooked: <p>I found a class <code>BatchEncoding</code> which has a function <code>to</code> to allocate the result tensor to certain device.<br>
It is inherited to BaseTokenizer but I think it is way better to cast <code>input.to(device)</code> instead of doing something to create <code>BertTokenizer</code> instance.</p>
<p>These are the references,</p><aside class="onebox githubblob" data-onebox-src="https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py#L1452">
  <header class="source">

      <a href="https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py#L1452" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py#L1452" target="_blank" rel="noopener nofollow ugc">huggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py#L1452</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="1442" style="counter-reset: li-counter 1441 ;">
          <li>            A special token representing a masked token (used by masked-language modeling pretraining objectives, like</li>
          <li>            BERT). Will be associated to `self.mask_token` and `self.mask_token_id`.</li>
          <li>        additional_special_tokens (tuple or list of `str` or `tokenizers.AddedToken`, *optional*):</li>
          <li>            A tuple or a list of additional special tokens. Add them here to ensure they won't be split by the</li>
          <li>            tokenization process. Will be associated to `self.additional_special_tokens` and</li>
          <li>            `self.additional_special_tokens_ids`.</li>
          <li>"""</li>
          <li>
          </li>
<li>
          </li>
<li>@add_end_docstrings(INIT_TOKENIZER_DOCSTRING)</li>
          <li class="selected">class PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin):</li>
          <li>    """</li>
          <li>    Base class for [`PreTrainedTokenizer`] and [`PreTrainedTokenizerFast`].</li>
          <li>
          </li>
<li>    Handles shared (mostly boiler plate) methods for those two classes.</li>
          <li>    """</li>
          <li>
          </li>
<li>    vocab_files_names: Dict[str, str] = {}</li>
          <li>    pretrained_vocab_files_map: Dict[str, Dict[str, str]] = {}</li>
          <li>    pretrained_init_configuration: Dict[str, Dict[str, Any]] = {}</li>
          <li>    max_model_input_sizes: Dict[str, Optional[int]] = {}</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox githubblob" data-onebox-src="https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/tokenization_bert.py#L137">
  <header class="source">

      <a href="https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/tokenization_bert.py#L137" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/tokenization_bert.py#L137" target="_blank" rel="noopener nofollow ugc">huggingface/transformers/blob/main/src/transformers/models/bert/tokenization_bert.py#L137</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="127" style="counter-reset: li-counter 126 ;">
          <li>
          </li>
<li>def whitespace_tokenize(text):</li>
          <li>    """Runs basic whitespace cleaning and splitting on a piece of text."""</li>
          <li>    text = text.strip()</li>
          <li>    if not text:</li>
          <li>        return []</li>
          <li>    tokens = text.split()</li>
          <li>    return tokens</li>
          <li>
          </li>
<li>
          </li>
<li class="selected">class BertTokenizer(PreTrainedTokenizer):</li>
          <li>    r"""</li>
          <li>    Construct a BERT tokenizer. Based on WordPiece.</li>
          <li>
          </li>
<li>    This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to</li>
          <li>    this superclass for more information regarding those methods.</li>
          <li>
          </li>
<li>    Args:</li>
          <li>        vocab_file (`str`):</li>
          <li>            File containing the vocabulary.</li>
          <li>        do_lower_case (`bool`, *optional*, defaults to `True`):</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>