Torchtext and <unk>, <pad>

username: iclementine
cooked: <p>I have a question regarding when doing nlp tasks such as tagging and parsing. When I define a Field for the input sentences, I define a sequential Field, and torchtext considerably handles &lt;unk&gt; for me, and &lt;pad&gt;, &lt;eos&gt;, &lt;sos&gt; if necessary.</p>
<p>The question is when I define a Field for the outputs or targets, like tagging, I do not want a &lt;unk&gt; to be in the vocabulary in that Field because I don’t want a classifier to output &lt;unk&gt;. Although we can build a Field without &lt;pad&gt;, &lt;eos&gt;, or &lt;sos&gt;, but &lt;unk&gt; is always included. Is there a way to define a Field without  in its vocab?</p>
<p>I have been doing it in two ways:</p>
<ol>
<li>
<p>Just let it be there, be confident that the model will not prefer a &lt;unk&gt; output because the training data does not even contain &lt;unk&gt; as output.</p>
</li>
<li>
<p>build a softmax layer whose output_size is smaller that the output_field’s vocab size, add some code like <code>pred = softmax_output.max(dim=-1) + 1</code> or <code>gold_output = gold_output - 1</code>.</p>
</li>
</ol>
<p>I don’t feel it is a neat way to do that. So, does any one have ideas what is the best practice for that?</p>
<p>Thank you!</p>

username: 11177
cooked: <p>For example, You can define Field TEXT for your input sequence and Field TAG for your output.<br>
When you build your vocabulay, you can do the following:<br>
TEXT.build_vocab(train_dataset)<br>
In this way, the words in your output won’t be included in you vocabulary. That’s the mechanism in torchtext. Powerful! Right?</p>

username: iclementine
cooked: <p>Yes, but when you are building a C-class classifier, the &lt;unk&gt; will make it more staightforward to build a (C+1)-class classifier. How do you handle that?</p>

username: 11177
cooked: <p>When you defining your Field for target, you shouldn’t treat it as a sequential Field, you just treat it as a discret label, even though it is in a sequential form. In a way, you shouldn’y build vocabulary for your target, because you are performing a classification problem, not a sequence generation problem.</p>

username: iclementine
cooked: <p>But in tasks like POS-tagging, the output field is sequential. I would provide</p>
<pre><code class="lang-auto">sequences: LongTensor(sequence_length, batch_size)
tags: LongTensor(sequence_length, batch_size)
</code></pre>
<p>to my model, if the target_field is not sequential, may be <code>a sequence at one forward</code> and batch inferencing would be a bit inconvenient. Or do you conventionally implement a <code>a step at one forward</code> model?</p>

username: 11177
cooked: <p>I am Chinese, my English is poor, i can’t seek a way to express my meaning. Sorry about that!<img src="https://discuss.pytorch.org/images/emoji/apple/sweat.png?v=5" title=":sweat:" class="emoji" alt=":sweat:"></p>

username: iclementine
cooked: <p>thank you! may be some pseudocode would help.</p>
<pre><code class="lang-auto">defaultdict(&lt;function torchtext.vocab._default_unk_index&gt;,
            {'#': 40,
             '$': 21,
             "''": 26,
             '(': 35,
             ')': 34,
             ',': 8,
             '.': 11,
             ':': 27,
             '&lt;pad&gt;': 1,
             '&lt;unk&gt;': 0, # that's it! th &lt;unk&gt;
             'CC': 13,
             'CD': 9,
             'DT': 5,
             'EX': 37,
             'FW': 43,
             'IN': 4,
             'JJ': 7,
             'JJR': 29,
             'JJS': 32,
             'LS': 41,
             'MD': 23,
             'NN': 2,
             'NNP': 3,
             'NNPS': 44,
             'NNS': 6,
             'PDT': 45,
             'POS': 20,
             'PRP': 19,
             'PRP$': 24,
             'RB': 12,
             'RBR': 31,
             'RBS': 38,
             'RP': 30,
             'TO': 15,
             'UH': 42,
             'VB': 14,
             'VBD': 10,
             'VBG': 18,
             'VBN': 16,
             'VBP': 22,
             'VBZ': 17,
             'WDT': 28,
             'WP': 33,
             'WP$': 39,
             'WRB': 36,
             '``': 25})
</code></pre>
<p>May be a non-sequential Field would help. I would try to figure that out.</p>

username: brobaidek
cooked: <p>I think you are looking for:<br>
LABEL = data.Field(sequential=False, unk_token=None)</p>
<p>Note: I did have to update PyTorch and torchtext for this to work</p>