Repository: pytorch/vision

Issue:
STL-10 Testing Protocol
Class STL10 does not support recommended testing protocol. 
See STL-10 official page: https://cs.stanford.edu/~acoates/stl10/


Discussion:
This is important!
Hi @gaa-cifasis sorry for the delayed response. I'm not too familiar with this dataset, can you elaborate a bit on what is wrong?

As far as I can tell, the `STL10` Dataset is [loading the test set correctly](https://github.com/pytorch/vision/blob/master/torchvision/datasets/stl10.py#L40). As for the testing protocol, I would assume that this would be handled and by the code consuming the data and not the dataset itself?
I think this issue is related with the loading of the training set. According to the website:

> The file fold_indices.txt contains the (zero-based) indices of the examples to be used for each training fold

but STL10 just loads all the folds for training. Therefore, the protocol is not followed and the results cannot be easily compared with the other papers.
For this you could use the subsetrandomsampler.

```
folds = np.loadtxt('fold_indices.txt')

fold = folds[use_fold]
fold = torch.from_numpy(fold)

dataloader = torch.utils.data.DataLoader(dataset, batch_size=batchSize, sampler=SubsetRandomSampler(fold), num_workers=2,pin_memory=False)
```
That should do it
While we can use the subset approach from @omair-kg , I believe we should add the folds as as option for STL-10.

Resolving patch:
diff --git a/torchvision/datasets/stl10.py b/torchvision/datasets/stl10.py
--- a/torchvision/datasets/stl10.py
+++ b/torchvision/datasets/stl10.py
@@ -14,6 +14,9 @@ class STL10(CIFAR10):
             ``stl10_binary`` exists.
         split (string): One of {'train', 'test', 'unlabeled', 'train+unlabeled'}.
             Accordingly dataset is selected.
+        folds (int, optional): One of {0-9} or None.
+            For training, loads one of the 10 pre-defined folds of 1k samples for the
+             standard evaluation procedure. If no value is passed, loads the 5k samples.
         transform (callable, optional): A function/transform that  takes in an PIL image
             and returns a transformed version. E.g, ``transforms.RandomCrop``
         target_transform (callable, optional): A function/transform that takes in the
@@ -28,6 +31,7 @@ class STL10(CIFAR10):
     filename = "stl10_binary.tar.gz"
     tgz_md5 = '91f7769df0f17e558f3565bffb0c7dfb'
     class_names_file = 'class_names.txt'
+    folds_list_file = 'fold_indices.txt'
     train_list = [
         ['train_X.bin', '918c2871b30a85fa023e0c44e0bee87f'],
         ['train_y.bin', '5a34089d4802c674881badbb80307741'],
@@ -40,7 +44,7 @@ class STL10(CIFAR10):
     ]
     splits = ('train', 'train+unlabeled', 'unlabeled', 'test')
 
-    def __init__(self, root, split='train',
+    def __init__(self, root, split='train', folds=None,
                  transform=None, target_transform=None, download=False):
         if split not in self.splits:
             raise ValueError('Split "{}" not found. Valid splits are: {}'.format(
@@ -50,6 +54,7 @@ def __init__(self, root, split='train',
         self.transform = transform
         self.target_transform = target_transform
         self.split = split  # train/test/unlabeled set
+        self.folds = folds  # one of the 10 pre-defined folds or the full dataset
 
         if download:
             self.download()
@@ -63,9 +68,12 @@ def __init__(self, root, split='train',
         if self.split == 'train':
             self.data, self.labels = self.__loadfile(
                 self.train_list[0][0], self.train_list[1][0])
+            self.__load_folds(folds)
+
         elif self.split == 'train+unlabeled':
             self.data, self.labels = self.__loadfile(
                 self.train_list[0][0], self.train_list[1][0])
+            self.__load_folds(folds)
             unlabeled_data, _ = self.__loadfile(self.train_list[2][0])
             self.data = np.concatenate((self.data, unlabeled_data))
             self.labels = np.concatenate(
@@ -131,3 +139,16 @@ def __loadfile(self, data_file, labels_file=None):
 
     def extra_repr(self):
         return "Split: {split}".format(**self.__dict__)
+
+    def __load_folds(self, folds):
+        # loads one of the folds if specified
+        if isinstance(folds, int):
+            if folds >= 0 and folds < 10:
+                path_to_folds = os.path.join(
+                    self.root, self.base_folder, self.folds_list_file)
+                with open(path_to_folds, 'r') as f:
+                    str_idx = f.read().splitlines()[folds]
+                    list_idx = np.fromstring(str_idx, dtype=np.uint8, sep=' ')
+                    self.data, self.labels = self.data[list_idx, :, :, :], self.labels[list_idx]
+            else:
+                raise ValueError('Folds "{}" not found. Valid splits are: 0-9.'.format(folds))


Test patch: