Is it a correct way to build_vocab()? (torchtext)

username: al314
cooked: <p>Hi all, sorry for basic question.<br>
Is it a correct way to build_vocab()?</p>
<pre><code class="lang-auto">import torch
from torchtext.data import Dataset, Example, Field
from torchtext.data import Iterator, BucketIterator

TEXT  = Field(sequential=True, tokenize=lambda x: x.split(), lower=True)
LABEL = Field(sequential=False, use_vocab=False)

data = [("The mountain is hight", "A"), ("Work is quite interesting", "B")]
fs = [('text', LABEL), ('category', TEXT)]
examples = list(map(lambda x: Example.fromlist(list(x), fields=fs), data))
dt = Dataset(examples, fields=fs)
TEXT.build_vocab(dt, vectors="glove.6B.100d")

print(len(TEXT.vocab))

for el in data:
    tokens = el[0].split()
    print(tokens)
    for t in tokens:
        print(TEXT.vocab.stoi[t])
</code></pre>
<p>Asking because in my working code (above is a test) all indices are set to 0, as also in this example.</p>

username: zhangguanheng66
cooked: <p>It generates vocab based only on the second entry (a.k.a. <code>category</code>). Therefore, any tokens in <code>text</code> are <code>unk</code>, whose id is <code>0</code>. IMO</p>
<p>You could write something similar like <a href="https://github.com/pytorch/text/blob/a5880a3da7928dd7dd529507eec943a307204de7/torchtext/vocab.py#L547" rel="nofollow noopener">build_vocab_from_iterator</a>. The constructor of the <code>Vocab</code> class needs a counter of tokens. Take a look at the <code>Vocab</code> class <a href="https://github.com/pytorch/text/blob/master/torchtext/vocab.py" rel="nofollow noopener">here</a></p>

username: al314
cooked: <p>Thanks!</p>
<p>Now I fixed my code snippet into</p>
<pre><code class="lang-auto">import torch
from torchtext.data import Dataset, Example, Field
from torchtext.data import Iterator, BucketIterator

TEXT  = Field(sequential=True, tokenize=lambda x: x.split(), 
                       lower=True, use_vocab=True)
LABEL = Field(sequential=False, use_vocab=False)

data = [("shop street mountain is hight", "a"), 
         ("work is interesting", "b")]

FIELDS = [('text', TEXT), ('category', LABEL)]

examples = list(map(lambda x: Example.fromlist(list(x), fields=FIELDS), 
                                 data))

dt = Dataset(examples, fields=FIELDS)

TEXT.build_vocab(dt, vectors="glove.6B.100d")
LABEL.build_vocab(dt, vectors="glove.6B.100d")

print(TEXT.vocab.stoi["is"])

data_iter = Iterator(dt, batch_size=4, sort_key=lambda x: len(x))
</code></pre>
<p>But now I have the next question )</p>
<p>How do I transform text data in <code>dt</code> or <code>data_iter</code> into numerical format suitable to be fed in into the model?</p>
<p>Now I have iterator over ‘dt’, but it contains text field as a text, not as numerical torch tensors.</p>
<p>As I understand TEXT field now contains mappings to tensors, but I need to use dt or data_iter as input to the model.</p>
<p>Update: reformulated into the question: <a href="https://discuss.pytorch.org/t/creating-input-for-the-model-from-the-raw-text/68219" class="inline-onebox">Creating input for the model from the raw text</a></p>

username: zhangguanheng66
cooked: <aside class="quote no-group" data-username="al314" data-post="3" data-topic="68094">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/al314/48/20046_2.png" class="avatar"> al314:</div>
<blockquote>
<p>batch_size</p>
</blockquote>
</aside>
<p>You mean <code>data_iter</code> is still text now? At some point, I remember, it numericalizes text into tensor. <code>data_iter</code> should be ready to use to train model.</p>
<p>If not, you can use <code>TEXT.vocab.stoi()</code> to numericalize tokens as a list and convert to tensors <code>torch.Tensor([tok_ids])</code>.</p>