Torchtext and token <eos>, <pad>

username: Tomas_fang
cooked: <p>When I did machine translation task(using the Transformer model), I find a question:</p>
<p>As I want the model to predict the  token but not have it be an input into the model, I simply slice the <code>&lt;eos&gt;</code> token off the end of the sequence. Thus:</p>
<p>trg = [sos, x_1, x_2, x_3, eos]<br>
trg[:-1] = [sos, x_1, x_2, x_3]</p>
<p>But when I use torchtext generate my datasets, I find the last elements of sentence is the <code>&lt;pad&gt;</code> token, such as:</p>
<p>trg = [sos, x_1, x_2, x_3, eos, pad, pad, pad]<br>
trg[:-1] = [sos, x_1, x_2, x_3, eos, pad, pad]</p>
<p>so i can’t slice the <code>&lt;eos&gt;</code> token, how can I solve this question?</p>

username: zhangguanheng66
cooked: <p>can you open an issue on torchtest github with a code snippet. The translation dataset has not been re-written and there may be some bugs there. But we can try to have some people take a look at it.</p>

username: Tomas_fang
cooked: <p>Ok, I will do it. And could you tell me how to solve this problem?</p>

username: zhangguanheng66
cooked: <p>Please post the issue along with some code snippets so we could help you figure out a solution.</p>

username: Tomas_fang
cooked: <p>This is my code:</p>
<pre><code class="lang-python">spacy_de = spacy.load('de')
spacy_en = spacy.load('en')
def tokenize_de(text):
    """
    Tokenizes German text from a string into a list of strings (tokens) and reverses it
    """
    return [tok.text for tok in spacy_de.tokenizer(text)][::-1]

def tokenize_en(text):
    """
    Tokenizes English text from a string into a list of strings (tokens)
    """
    return [tok.text for tok in spacy_en.tokenizer(text)]
SRC = Field(tokenize=tokenize_de,
            init_token=init_token,
            eos_token=eos_token,
            lower=True,
            batch_first=True)
TRG = Field(tokenize=tokenize_en,
            init_token=init_token,
            eos_token=eos_token,
            lower=True,
            batch_first=True)

train_data, valid_data = IWSLT.splits(exts=('.de', '.en'),
                                                 fields=(SRC, TRG),
                                                 test=None,
                                                 filter_pred=lambda x: len(vars(x)['src']) &lt;= max_seq_len and
                                                 len(vars(x)['trg']) &lt;= max_seq_len)
SRC.build_vocab(train_data, min_freq=min_freq)
TRG.build_vocab(train_data, min_freq=min_freq)


train_iter, valid_iter = BucketIterator.splits(
    (train_data, valid_data),
    batch_size=batch_size,
    device=device)
</code></pre>

username: Tomas_fang
cooked: <p>When I print the data in train_iter, the output as follow:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6f87d8e3f3cd33cfd4b8d28e2e2a75b2b8fbb4e.png" data-download-href="https://discuss.pytorch.org/uploads/default/c6f87d8e3f3cd33cfd4b8d28e2e2a75b2b8fbb4e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c6f87d8e3f3cd33cfd4b8d28e2e2a75b2b8fbb4e_2_690x386.png" alt="image" data-base62-sha1="soaVvfZNd0sf0zYQLyjZ582Cz0i" width="690" height="386" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c6f87d8e3f3cd33cfd4b8d28e2e2a75b2b8fbb4e_2_690x386.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c6f87d8e3f3cd33cfd4b8d28e2e2a75b2b8fbb4e_2_1035x579.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6f87d8e3f3cd33cfd4b8d28e2e2a75b2b8fbb4e.png 2x" data-dominant-color="38404F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1052×590 35 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As i say at the beginning, I think the last element of src and trg should be <code>&lt;eos&gt;</code> token, which is 3, but in fact the last element of src and trg is 1, that is <code>&lt;pad&gt;</code> token.</p>

username: jimmykobe
cooked: <p>May I ask have you resolved this issue?</p>