Problem with stratified sampling on dataframe and then convert to torchtext

username: Colospring_Liu
cooked: <p>Hi guys, I am very new to pytorch and torchtext. I am trying to do a stratified sampling before convert the training and test sets to torchtext datasets. Here are my codes:</p>
<pre><code class="lang-auto"># stratified sampling

TEXT = data.Field(tokenize = tokenizer, lower=True, include_lengths = True)
LABEL = data.LabelField(dtype = torch.long)
fields = {'reviewText': ('t', TEXT), 'sentiment': ('l', LABEL)}

label = text.sentiment
train_data, valid_data, y_train, y_val = train_test_split(text, label, test_size=0.33, stratify=label)
train_data = train_data.reset_index(drop=True)
train_data.head()

train_dataset = data.Dataset(train_data, fields)
valid_dataset = data.Dataset(valid_data, fields)

#### Vocabulary ###

MAX_VOCAB_SIZE = 25000

#GloVe vocabulary
TEXT.build_vocab(train_dataset, 
                 max_size = MAX_VOCAB_SIZE, 
                 vectors = "glove.6B.100d", 
                 unk_init = torch.Tensor.normal_)
LABEL.build_vocab(train_dataset)

# Check if it worked
print(f'Number of training examples: {len(train_dataset)}')
print(f'Number of testing examples: {len(valid_dataset)}')
print(vars(train_dataset[0]))
</code></pre>
<p>Then I got error at the last line when printing the train dataset:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a518e1c8f57eb9aba174e207bb1403da46ebfae9.png" data-download-href="https://discuss.pytorch.org/uploads/default/a518e1c8f57eb9aba174e207bb1403da46ebfae9" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a518e1c8f57eb9aba174e207bb1403da46ebfae9_2_690x492.png" alt="Capture" data-base62-sha1="nyw5HFh777DZG8Z5aFZAqcOtUIF" width="690" height="492" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a518e1c8f57eb9aba174e207bb1403da46ebfae9_2_690x492.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a518e1c8f57eb9aba174e207bb1403da46ebfae9_2_1035x738.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/5/a518e1c8f57eb9aba174e207bb1403da46ebfae9.png 2x" data-dominant-color="F4F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1052×751 69.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/6/76e56eb7a16801da346362952b580303db018de5.png" data-download-href="https://discuss.pytorch.org/uploads/default/76e56eb7a16801da346362952b580303db018de5" title="Capture1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76e56eb7a16801da346362952b580303db018de5_2_690x360.png" alt="Capture1" data-base62-sha1="gXNTHh4jNSNsLWhih9xhqrg9QMt" width="690" height="360" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76e56eb7a16801da346362952b580303db018de5_2_690x360.png, https://discuss.pytorch.org/uploads/default/original/3X/7/6/76e56eb7a16801da346362952b580303db018de5.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/6/76e56eb7a16801da346362952b580303db018de5.png 2x" data-dominant-color="F2F6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture1</span><span class="informations">1030×538 55.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Could anyone help? Many thanks!</p>

username: ptrblck
cooked: <p>Based on the error message it seems that the <code>torchtext.data.Dataset</code> is using a <code>pandas.DataFrame</code> internally and tries to index it directly with an integer, which creates the error (if I’m not mistaken, <code>.iloc</code> should be used instead). You could try to pass the data as a tensor to the dataset instead.</p>

username: seanbenhur
cooked: <p>is there a way to use weighted sampling or any other methods to handle a class imbalance in torchtext, I have a <a href="https://discuss.pytorch.org/t/sampler-missing-in-torchtext/83572">problem similar to this</a></p>