BERT ImbalancedDatasetSampler ValueError:Cannot set a frame with no defined index and a value that cannot be converted to a Series

Jeeg_Zhang: <p>Using BERT for multi class classification.<br>
Dataset is imbalanced.</p>
<p>The train_dataset is tokenized, so every element is 3d and has id, mask and label.</p>
<p>When applying RandomSampler, SequentialSampler it’s fine.</p>
<p>Replace the sampler to ImbalancedDatasetSampler</p>
<p>batch_size=3<br>
dataloader_train_o = DataLoader(<br>
dataset_train,<br>
sampler=ImbalancedDatasetSampler(dataset_train),<br>
batch_size=batch_size,<br>
# **kwargs<br>
)</p>
<p>results in ValueError:</p>
<hr>
<p>ValueError                                Traceback (most recent call last)<br>
File D:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\pandas\core\frame.py:3892, in DataFrame._ensure_valid_index(self, value)<br>
3891 try:<br>
 → 3892     value = Series(value)<br>
3893 except (ValueError, NotImplementedError, TypeError) as err:</p>
<p>File D:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\pandas\core\series.py:451, in Series.<strong>init</strong>(self, data, index, dtype, name, copy, fastpath)<br>
450 else:<br>
 → 451     data = sanitize_array(data, index, dtype, copy)<br>
453     manager = get_option(“mode.data_manager”)</p>
<p>File D:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\pandas\core\construction.py:601, in sanitize_array(data, index, dtype, copy, raise_cast_failure, allow_2d)<br>
599             subarr = maybe_infer_to_datetimelike(subarr)<br>
 → 601 subarr = _sanitize_ndim(subarr, data, dtype, index, allow_2d=allow_2d)<br>
603 if isinstance(subarr, np.ndarray):<br>
604     # at this point we should have dtype be None or subarr.dtype == dtype</p>
<p>File D:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\pandas\core\construction.py:652, in _sanitize_ndim(result, data, dtype, index, allow_2d)<br>
651         return result<br>
 → 652     raise ValueError(“Data must be 1-dimensional”)<br>
653 if is_object_dtype(dtype) and isinstance(dtype, ExtensionDtype):<br>
654     # i.e. PandasDtype(“O”)</p>
<p>ValueError: Data must be 1-dimensional</p>
<p>The above exception was the direct cause of the following exception:</p>
<p>ValueError                                Traceback (most recent call last)<br>
Input In [49], in &lt;cell line: 5&gt;()<br>
2 from torchsampler import ImbalancedDatasetSampler<br>
4 batch_size=3<br>
5 dataloader_train_o = DataLoader(<br>
6     dataset_train,<br>
----&gt; 7     sampler=ImbalancedDatasetSampler(dataset_train),<br>
8     batch_size=batch_size,<br>
9     # **kwargs<br>
10 )<br>
12 dataloader_validation_o = DataLoader(<br>
13     dataset_val,<br>
14     sampler=SequentialSampler(dataset_val),<br>
15     batch_size=batch_size,<br>
16     # **kwargs<br>
17 )</p>
<p>File D:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\torchsampler\imbalanced.py:37, in ImbalancedDatasetSampler.<strong>init</strong>(self, dataset, labels, indices, num_samples, callback_get_label)<br>
35 # distribution of classes in the dataset<br>
36 df = pd.DataFrame()<br>
—&gt; 37 df[“label”] = self._get_labels(dataset) if labels is None else labels<br>
38 df.index = self.indices<br>
39 df = df.sort_index()</p>
<p>File D:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\pandas\core\frame.py:3655, in DataFrame.<strong>setitem</strong>(self, key, value)<br>
3652     self._setitem_array([key], value)<br>
3653 else:<br>
3654     # set column<br>
 → 3655     self._set_item(key, value)</p>
<p>File D:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\pandas\core\frame.py:3832, in DataFrame._set_item(self, key, value)<br>
3822 def _set_item(self, key, value) → None:<br>
3823     “”"<br>
3824     Add series to DataFrame in specified column.<br>
3825<br>
(…)<br>
3830     ensure homogeneity.<br>
3831     “”"<br>
 → 3832     value = self._sanitize_column(value)<br>
3834     if (<br>
3835         key in self.columns<br>
3836         and value.ndim == 1<br>
3837         and not is_extension_array_dtype(value)<br>
3838     ):<br>
3839         # broadcast across multiple columns if necessary<br>
3840         if not self.columns.is_unique or isinstance(self.columns, MultiIndex):</p>
<p>File D:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\pandas\core\frame.py:4528, in DataFrame._sanitize_column(self, value)<br>
4515 def _sanitize_column(self, value) → ArrayLike:<br>
4516     “”"<br>
4517     Ensures new columns (which go into the BlockManager as new blocks) are<br>
4518     always copied and converted into an array.<br>
(…)<br>
4526     numpy.ndarray or ExtensionArray<br>
4527     “”"<br>
 → 4528     self._ensure_valid_index(value)<br>
4530     # We should never get here with DataFrame value<br>
4531     if isinstance(value, Series):</p>
<p>File D:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\pandas\core\frame.py:3894, in DataFrame._ensure_valid_index(self, value)<br>
3892         value = Series(value)<br>
3893     except (ValueError, NotImplementedError, TypeError) as err:<br>
 → 3894         raise ValueError(<br>
3895             "Cannot set a frame with no defined index "<br>
3896             “and a value that cannot be converted to a Series”<br>
3897         ) from err<br>
3899 # GH31368 preserve name of index<br>
3900 index_copy = value.index.copy()</p>
<p>ValueError: Cannot set a frame with no defined index and a value that cannot be converted to a Series</p>