Can't Iterate Validation set using BucketIterator

username: j-mehul
cooked: <p>Hello Pytorch Experts,</p>
<p>I am facing an issue while using BucketIterator. Below is a code snippet to divide the data into the train and validation set. I am able to iterate the training data but getting the error in the validation dataset. I have tried different seed to check if there is any issue in the data but couldn’t resolve.</p>
<pre><code class="lang-auto">train_iterator, valid_iterator = data.BucketIterator.splits(
    (train_data, valid_data), 
    batch_size = BATCH_SIZE,
    sort_key = lambda x: len(x.text),
    sort_within_batch=True,
    device = device)

for batch in valid_iterator :
        text, text_lengths = batch.text  
        print(text)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/2/42cadb6c86628f2576b77f6d1fba93c2b3e30cb7.png" data-download-href="https://discuss.pytorch.org/uploads/default/42cadb6c86628f2576b77f6d1fba93c2b3e30cb7" title="0"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/2/42cadb6c86628f2576b77f6d1fba93c2b3e30cb7.png" alt="0" data-base62-sha1="9wS8d4WlbcFp7EuCXVu611ahCon" width="506" height="500" data-dominant-color="F1F4F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">0</span><span class="informations">575×568 31.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Based on the error message it seems the validation dataset cannot load a specific sample due to a <code>KeyError</code>.<br>
You could check the index of the failing sample via:</p>
<pre><code class="lang-python">for idx, batch in enumerate(valid_iterator):
    print(idx)
    text, text_lengths = batch.text
</code></pre>
<p>and based on this index check why this sample is failing to load.</p>

username: j-mehul
cooked: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a>, Thankyou for responding.<br>
I have tried using your snippet.<br>
I don’t think its a data issue as I have changes seeds and  split_ratio.<br>
Always, the error is in the validation set.<br>
If there had been an issue in data then I guess train_iterator would also throw an error but train_iterator runs perfectly well.</p>
<p>Full Code Snippet for your reference:</p>
<pre><code class="lang-auto">TEXT = data.Field(tokenize='spacy',batch_first=True,include_lengths=True)
LABEL = data.LabelField(dtype = torch.float,batch_first=True)

fields = [('question_text',TEXT),('target', LABEL)]
training_data=data.TabularDataset(path = 'quora.csv',format = 'csv',fields = fields,skip_header = True)

SEED = 20
train_data, valid_data = training_data.split(split_ratio=0.7, random_state = random.seed(SEED))

TEXT.build_vocab(train_data,min_freq=3,vectors = "glove.6B.100d")  
LABEL.build_vocab(train_data)

#set batch size
BATCH_SIZE = 64

#Load an iterator
train_iterator, valid_iterator = data.BucketIterator.splits(
    (train_data, valid_data), 
    batch_size = BATCH_SIZE,
    sort_key = lambda x: len(x.question_text),
    sort_within_batch=True,sort=False)

for idx, batch in enumerate(valid_iterator):
    print(idx)
    text, text_lengths = batch.question_text
</code></pre>

username: Abhilash_Srivastava
cooked: <aside class="quote no-group" data-username="j-mehul" data-post="3" data-topic="98643">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j-mehul/48/29573_2.png" class="avatar"> j-mehul:</div>
<blockquote>
<pre><code class="lang-auto">TEXT.build_vocab(train_data,min_freq=3,vectors = "glove.6B.100d")  
LABEL.build_vocab(train_data)
</code></pre>
</blockquote>
</aside>
<p>Can you try building the vocab using the train_data and valid_data combined.</p>

username: Harsh_Sharma
cooked: <p>This One actually worked.<br>
It seems like while using <code>torchtext</code> data split for different <code>build_vocabs</code>, the valid vocab doesn’t seem to have an index located, even after using Glove Embeddings.<br>
I was getting various key errors and then realized this could be one of the issues.</p>