BucketIterator shows different behavior

Minseok: <p>torchtext.data.iterator.BucketIterator</p>
<p>I am writing some sentiment analysis code using torchtext bucketiterator and surprised by the behavior of how we make dataset</p>
<p>for example if we have</p>
<pre><code class="lang-auto">from torchtext.data import TabularDataset


TEXT = data.Field(tokenize = 'spacy', include_lengths = True, preprocessing= lambda x: preprocessor(x), lower=True)
LABEL = data.LabelField(dtype = torch.long)
INDEX = data.RawField()
INDEX.is_target = False

train_data = TabularDataset('./data/train.tsv', 
                            format='tsv',
                            skip_header=True,
                            fields=[('PhraseId', None), ('SentenceId', None), ('Phrase', TEXT), ('Sentiment', LABEL)])
test_data = TabularDataset('./data/test.tsv', 
                            format='tsv',
                            skip_header=True,
                            fields=[('PhraseId', INDEX), ('SentenceId', None), ('Phrase', TEXT)])
TEXT.build_vocab(train_data,
                #max_size = MAX_VOCAB_SIZE,
                vectors = 'glove.6B.100d',
                unk_init = torch.Tensor.normal_)

LABEL.build_vocab(train_data)
</code></pre>
<p>Above is kinda thing you wouldn’t be interested, but magic happens below:</p>
<pre><code class="lang-auto">BATCH_SIZE= 64

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

test_iterator = data.BucketIterator.splits(
    test_data,
    sort = True,
    sort_within_batch=True,
    sort_key = lambda x: len(x.Phrase),
    batch_size = BATCH_SIZE,
    device = device)
</code></pre>
<p>if we have data like above, below  code shows</p>
<pre><code class="lang-auto">vars(test_iterator[107].dataset)
</code></pre>
<p>Out[47]:</p>
<p>{‘Phrase’: [‘movie’, ‘becomes’, ‘heady’, ‘experience’], ‘PhraseId’: ‘156168’}</p>
<p>But below shows</p>
<pre><code class="lang-auto">train_iterator, test_iterator = data.BucketIterator.splits(
    (train_data, test_data),
    sort = True,
    sort_within_batch=True,
    sort_key = lambda x: len(x.Phrase),
    batch_size = BATCH_SIZE,
    device = device)

vars(test_iterator[107].dataset)
</code></pre>
<p>it throws an error that (TypeError: ‘BucketIterator’ object does not support indexing)</p>
<p>Don’t know why but the only difference there is whether you construct iterator data using train_data and test_data together or just using test_data. Just parenthesis for test data like</p>
<pre><code class="lang-auto"> test_iterator = data.BucketIterator.splits(
    (test_data),
...
</code></pre>
<p>does not show any difference compared to using only test_data(i.e., without parenthesis)</p>

mfa: <p>You should use <code>BucketIterator.splits()</code> when you actually have a split data. If you want to create <code>BucketIterator</code> only for one split e.g. test or train, use <code>BucketIterator</code> only. That means your above case where you only pass <code>test_data</code> should be changed to:</p>
<pre><code class="lang-auto">test_iterator = data.BucketIterator(
    test_data,
    sort = True,
    sort_within_batch=True,
    sort_key = lambda x: len(x.Phrase),
    batch_size = BATCH_SIZE,
    device = device)
</code></pre>