Out-of-vocabulary KeyError on vocab.stoi in TorchText

proxyproxyproxy_prox: <p>I created an NLP model on dataset <code>a</code> with torchtext. Now I tried to generate predictions for new cases but <code>vocab.stoi</code> is throwing a <code>KeyError</code>.</p>
<pre><code class="lang-auto"># Original TEXT field
# TEXT = Field(
#     sequential=True, 
#     tokenize=tokenizer, 
#     lower=True,
#     use_vocab = True,
#     #fix_length = 200,
#     pad_first = True
# ) 
TEXT = dill.load(open("field.dill", "rb"))

unl_datafields = [('Column1', None), ('Text', TEXT), ('Column2', None)]

unl = TabularDataset(
    path='./unl.csv', 
    format='csv',
    csv_reader_params={ 'delimiter': ';' },
    skip_header=True,
    fields=unl_datafields
)

unl_iter = Iterator(
    unl,
    batch_size=64,
    shuffle=False,
    sort=False,
    sort_within_batch=False
)

next(iter(unl_iter))
</code></pre>
<p>The code above throws the following error:</p>
<pre><code class="lang-auto"> 334         if self.use_vocab:
    335             if self.sequential:
--&gt; 336                 arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]
    337             else:
    338                 arr = [self.vocab.stoi[x] for x in arr]

KeyError: ' '
</code></pre>
<p>It seems like ’ ’ (a single space) cannot be found in <code>vocab.stoi</code>, which is suprising since the tokenizer splits on spaces. Moreover, I would like this to work with more words that are out of vocabulary. Can anyone give me some pointers on how to proceed?</p>

vdw: <p>I’m not sure if I understand you correctly, but a the whitespaces shouldn’t be part of your vocabulary. Whitespaces only separate words/tokens, and are not words/tokens themselves. So the error is to be expected.</p>
<p>The question is rather why you are trying to look up whitespaces in the first place. That shouldn’t happen. Usually you have something like:</p>
<pre><code>sentence = 'This is a normal sentence, nothing special.'
tokens = tokenize(sentence, separator=' ')
# tokens = ['This', 'is', 'a', 'normal', 'sentence, ',', 'nothin', 'special', '.']
</code></pre>
<p>Is there any chance that you have sentence with more than one whitespaces in a row, and your tokenizer gets thrown off by that, for example:</p>
<pre><code>sentence = 'This sentence has a gap     with five whitespaces.'
</code></pre>
<p>Maybe in this case <code>tokens</code> then contains whitespaces as elements. Anyway, there should be no situation where you have to look up whitespaces.</p>

zhangguanheng66: <p>I think even for the continuous whitespace case, the tokenizer should handle it properly (a.k.a. remove multiple whitespace all together).</p>

vdw: <p>I agree, proper tokenizers will most likely do that properly. I was just making a crude guess why there might be lookup for a whitespace in the first place.</p>
<p>I certainly had similar issues since I had to implement my own tokenizer for better handling social media text where people forget whitespaces, use emojis or emoticons as sentence separators, etc. Granted the more common problem I had was empty stokens instead of tokens with one or more whitespaces, like in:</p>
<pre><code>'1 2 3     '.split(' ')
# ['1', '2', '', '', '', '', '3']</code></pre>

zhangguanheng66: <p>If you see the basic English normalization in torchtext <a href="https://github.com/pytorch/text/blob/2dd003150b452ea061964be9843776ed960c0cb2/torchtext/data/utils.py#L46" rel="nofollow noopener">link</a> - also same as fastText, the multiple space is replaced with single space first before tokenizer.</p>
<p>I guess you could remove the emoji as well.</p>