Padding text data from torchtext.datasets for Recurrence network

username: alabijesujoba
cooked: <p>I am working with the torchtext.datasets datasets. I need to do classification using a recurrence network.</p>
<pre><code class="lang-auto">NGRAMS = 1
BATCH_SIZE = 20
train_dataset, test_dataset = text_classification.DATASETS['AG_NEWS'](
    root='./.data', ngrams=NGRAMS, vocab=None)
</code></pre>
<p>and the dataLoader</p>
<pre><code class="lang-auto">data = DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True,collate_fn=batching)
</code></pre>
<p>create batch function</p>
<pre><code class="lang-auto">def batching(batch):
    #print(batch.shape)
    label = torch.tensor([entry[0] for entry in batch])
    text = [entry[1] for entry in batch]
    textLength = torch.tensor([entry.shape for entry in text])
    max_text_length = torch.max(textLength)
    newtext = [torch.cat((torch.zeros([max_text_length-textLength[i]], dtype=torch.long), text[i]),0) for i in range(len(text))]
    new_textLength = torch.tensor([entry.shape for entry in newtext])
    text2=torch.stack(newtext)  #the padded tensor
    
    return text2, label
</code></pre>
<p>What I have in the function above is to ensure all the sentences in the batch are of equal lengths by padding them by 0(to the left).</p>
<ol>
<li>Am I doing the right thing?</li>
<li>Do I need to add 0 to the vocabulary? How I add 0 to the vocabulary?  Since the vocabulary is created automatically?</li>
<li>Do I add start and end of sentence token? How do I add to the vocabulary?</li>
<li>Lastly, do I need an offset?</li>
</ol>

username: vdw
cooked: <ol>
<li>
<p>I personally don’t work with <code>torchtext</code> but do all the preprocessing myself; it’s not complicated and I can tweak it to my liking. I assume there are a bunch of tutorials out there.</p>
</li>
<li>
<p>Strictly speaking, I don’t think that 0 (representing padding <code>&lt;PAD&gt;</code>) has to be in the vocabulary (you probably also need an index to represent unknown <code>&lt;UNK&gt;</code>) – although, it would probably the cleaner solution. However</p>
<ul>
<li>you have to make sure the 0 does not already reference a real word in your existing vocabulary</li>
<li>your word embedding layer has to be large enough to cover all words in your vocabulary and the padding (and <code>&lt;UNK&gt;</code>, <code>&lt;SOS&gt;</code>, <code>&lt;EOS&gt;</code>, if needed)</li>
</ul>
</li>
<li>
<p>Since you do only classification, I don’t think you need <code>&lt;SOS&gt;</code> and/or <code>&lt;EOS&gt;</code>. Those are important particularly for the decoder in sequence-to-sequence models.</p>
</li>
<li>
<p>No idea what you mean by offset <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
</li>
</ol>
<p>If it’s useful, here’s <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/utils/data/text/vocabulary.py" rel="nofollow noopener">the code to my <code>Vocabulary</code> class</a> I use. Depending what extra indexes I need, I can create an initial one like:</p>
<pre><code>vocabulary = Vocabulary(default_indexes={0: '&lt;pad&gt;', 1: '&lt;unk&gt;', 2: '&lt;sos&gt;', 3: '&lt;eos&gt;'})
</code></pre>
<p>or</p>
<pre><code>vocabulary = Vocabulary(default_indexes={0: '&lt;pad&gt;', 1: '&lt;unk&gt;'})
</code></pre>
<p>if I just have a classifier and don’t need <code>&lt;sos&gt;</code> and <code>&lt;eos&gt;</code>. The first word I then add to the vocabulary then has index 4 or 2, respectively, and so on. Do you mean this with offset?</p>
<p>Anyway, I would assume <code>torchtext</code> can handle this smoothly as well. You might want to look for a sequence-to-sequence tutorial that uses <code>torchtext</code>, since there <code>&lt;sos&gt;</code> and <code>&lt;eos&gt;</code> are needed.</p>

username: zhangguanheng66
cooked: <aside class="quote no-group" data-username="alabijesujoba" data-post="1" data-topic="66787">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alabijesujoba/48/18493_2.png" class="avatar"> alabijesujoba:</div>
<blockquote>
<p>aLoader(train_data,</p>
</blockquote>
</aside>
<p>Thanks <a class="mention" href="/u/vdw">@vdw</a> for the nice answer. I think <code>offsets</code> is required by <code>EmbeddingBag</code> where sequences are concatenated as a long sequence and <code>offsets</code> is used to separate individual sequences. Since you choose to pad sequences, it’s not really necessary to have <code>offset</code> here.</p>
<p>If your vocab comes with <code>&lt;PAD&gt;</code> token, you could get the pad id by</p>
<pre><code class="lang-auto">pad_id = train_dataset.get_vocab()['&lt;pad&gt;']
</code></pre>
<p>Sometime, you may even want to pad sequences with similar lengths. There is an issue post on torchtext to explain padding <a href="https://github.com/pytorch/text/issues/664" rel="noopener nofollow ugc">link</a>.</p>